WO2013153661A1 - Three-dimensional measuring device, three-dimensional measuring system, control method, program, and storage medium - Google Patents

Abstract

In the present invention, a three-dimensional measuring device is provided with an image acquisition means, a removal means, and a luminance correction means, and the device conducts three-dimensional measuring of a measurement object onto which a light pattern is projected via a transmission device. The image acquisition means acquires an image of the measurement object captured via the transmission device. The removal means removes from the image of the measurement object the light pattern projected and displayed by the transmission device. The luminance correction means detects the display region of the measurement object onto which the light pattern was projected and corrects the luminance of that display region.

Description

Three-dimensional measuring apparatus, three-dimensional measuring system, control method, program, and storage medium

The present invention relates to a three-dimensional measurement technique.

Conventionally, a technique for performing three-dimensional measurement of an object as a subject based on an image taken by a camera is known. For example, in Patent Document 1 and Non-Patent Document 1, a plurality of light patterns are projected on a measurement object, and the three-dimensional position of the measurement object is measured based on an image obtained by photographing the measurement object on which these light patterns are projected. Technology is disclosed. Further, in Patent Document 2, when a person is photographed through a screen while projecting and displaying an image on a screen in a video conference system or the like, a process of subtracting a noise signal generated based on the characteristics of the screen, A technique for improving the image quality of a screen is disclosed.

JP 2008-170282 A JP 11-355742 A

In a three-dimensional measurement system that projects a light pattern onto a measurement object and shoots the measurement object via a transmission device, a part of the light pattern to be projected is projected and displayed on the transmission device. The problem that the accuracy of the original measurement is reduced may occur depending on the characteristics of the transmission device. Further, when the process of removing the light pattern projected and displayed on the transmission device is performed, there is a problem that the luminance of the light pattern projected on the measurement object is lowered.

The present invention has been made to solve the above-described problems, and preferably performs measurement even when a light pattern is projected onto a measurement object and a measurement object is photographed via a transmission device. The main object is to provide a three-dimensional measuring apparatus capable of three-dimensional measurement of an object.

The invention according to claim 1 is a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device, and an image of the measurement object photographed through the transmission device An image acquisition means for acquiring the light pattern, a removal means for removing the light pattern projected and displayed on the transmission device from the image of the measurement object, and the measurement object irradiated with the light pattern in the image of the measurement object Brightness correction means for detecting the display area and correcting the brightness of the display area.

The invention according to claim 10 is a control method executed by a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected through a transmission device, and is photographed through the transmission device. In the image acquisition step of acquiring the image of the measurement object, the removal step of removing the light pattern projected and displayed on the transmission device from the image of the measurement object, and in the image of the measurement object, the light pattern is irradiated And a luminance correction step of detecting a display area of the measured object and correcting the luminance of the display area.

The invention according to claim 11 is a program executed by a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device, and is taken by the transmission device. An image acquisition unit that acquires an image of the measurement object, a removal unit that removes the light pattern projected and displayed on the transmission device from the image of the measurement object, and the light pattern is irradiated on the image of the measurement object. In addition, the three-dimensional measurement apparatus is caused to function as a luminance correction unit that detects a display area of the measurement object and corrects the luminance of the display area.

It is a schematic block diagram of the three-dimensional measurement system which concerns on a present Example. The functional block of a three-dimensional measuring device is shown. It is an image which a projection pattern process part refers and produces | generates. It is an image which a difference process part refers and produces | generates. It is an image which a brightness correction process part refers and produces | generates. It is an image which a brightness correction process part refers and produces | generates.

In one preferred embodiment of the present invention, a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device, the measurement object photographed via the transmission device Image acquisition means for acquiring the image of the measurement object, removal means for removing the light pattern projected and displayed on the transmission device from the image of the measurement object, and the light pattern irradiated on the image of the measurement object. Brightness correction means for detecting the display area of the measurement object and correcting the brightness of the display area.

The three-dimensional measurement apparatus includes an image acquisition unit, a removal unit, and a luminance correction unit, and performs a three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device. The image acquisition unit acquires an image of the measurement object photographed through the transmission device. The removing unit removes the light pattern projected and displayed on the transmission device from the image of the measurement object. Thereby, a removal means suppresses that a transmissive device is detected accidentally. The brightness correction unit detects the display area of the measurement object irradiated with the light pattern and corrects the brightness of the display area. In this way, when the three-dimensional measurement apparatus removes the light pattern projected on the transmission device, the display area of the measurement object irradiated with the light pattern becomes dark. It can suppress that measurement accuracy falls.

In one aspect of the three-dimensional measurement apparatus, in the image of the measurement object, an overlapping portion of the area where the light pattern is removed and the display area of the measurement object are overlapped with the measurement object irradiated with the light pattern. Detect as display area. Thereby, the three-dimensional measuring apparatus can preferably recognize the display area of the measurement object irradiated with the light pattern.

In another aspect of the three-dimensional measurement apparatus, the luminance correction unit corrects the luminance of the display area of the measurement object irradiated with the light pattern based on the luminance before processing by the removing unit. By doing in this way, the three-dimensional measuring device increases the difference between the luminance of the display area of the measurement object irradiated with the light pattern and the luminance of the display area of the measurement object not irradiated with the light pattern. Original measurement can be performed with high accuracy.

In another aspect of the three-dimensional measurement apparatus, the brightness correction unit may determine the brightness of the display area of the measurement object irradiated with the light pattern and the brightness of the display area of the measurement object not irradiated with the light pattern. Correct based on brightness. Even in this mode, the three-dimensional measurement apparatus increases the difference between the luminance of the display area of the measurement object irradiated with the light pattern and the luminance of the display area of the measurement object not irradiated with the light pattern, and performs three-dimensional measurement. Can be performed with high accuracy.

In another aspect of the three-dimensional measurement apparatus, the removing unit includes a projection pattern processing unit that generates an image indicating the light pattern projected and displayed on the transmission device, and an image of the measurement object. Difference processing means for performing difference processing on the image generated by the projection pattern processing means. Here, “difference processing” refers to processing for removing a component of a predetermined image from a target image, for example, processing for subtracting each pixel value of the latter image from each pixel value of the former image. Thereby, the removal means can preferably remove the light pattern projected and displayed on the transmission device from the image of the measurement object.

In another aspect of the three-dimensional measurement apparatus, the projection pattern processing unit may be configured so that the measurement object does not exist with respect to an image of the transmission device on which the light pattern is projected in a state where the measurement object does not exist. An image indicating the light pattern is generated by performing a difference process on the image of the transmissive device on which the light pattern is not projected. Thus, the removing unit can generate an image showing the light pattern projected and displayed on the transmission device, and can remove the light pattern projected and displayed on the transmission device from the image of the measurement object.

In another aspect of the three-dimensional measurement apparatus, the difference processing unit may be configured such that the luminance of the light pattern projected and displayed on the transmission device in an image captured in a state where the measurement object does not exist, When it is detected that the brightness of the light pattern projected and displayed on the transmissive device in the image is smaller than the brightness value of the light pattern generated by the projection pattern processing means, the difference process is performed. In general, when the optical characteristics of the background of the measurement object when shooting without the measurement object and the optical characteristics of the background of the measurement object when shooting with the measurement object are different, the projected image is displayed on the transmission device. The brightness of the generated light pattern may be different. Even in such a case, the three-dimensional measurement apparatus can suitably remove the light pattern projected and displayed on the transmission device from the image of the measurement object.

In another aspect of the three-dimensional measurement apparatus, the removal unit further removes the light pattern projected and displayed on a background object of the measurement object from the image of the measurement object, and the luminance correction unit includes the brightness correction unit, The display area of the measurement object irradiated with the light pattern is detected in the image of the measurement object in the area to be processed by the removing unit, and the luminance of the display area is corrected. According to this aspect, the three-dimensional measurement apparatus can correct the luminance of the display area of the measurement object irradiated with the light pattern even when the light pattern is irradiated on the background object of the measurement object.

In another preferred embodiment of the present invention, a three-dimensional measurement system includes a transmission device, the three-dimensional measurement apparatus described above, and a projector that emits the light pattern to the measurement object via the transmission device. And a camera that transmits an image obtained by photographing the measurement object via the transmission device to the three-dimensional measurement apparatus. According to this three-dimensional measurement system, the three-dimensional measurement apparatus performs measurement because the display area of the measurement object irradiated with the light pattern becomes dark when removing the light pattern projected on the transmission device. It can suppress that the measurement accuracy of an object falls.

In still another embodiment of the present invention, there is provided a control method executed by a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device, the image being shot through the transmission device. In the image acquisition step of acquiring the image of the measurement object, the removal step of removing the light pattern projected and displayed on the transmission device from the image of the measurement object, and the image of the measurement object, the light pattern is A luminance correction step of detecting a display area of the irradiated measurement object and correcting the luminance of the display area. By executing this control method, the three-dimensional measurement apparatus performs measurement because the display area of the measurement object irradiated with the light pattern becomes dark when removing the light pattern projected on the transmission device. It can suppress that the measurement accuracy of an object falls.

In still another embodiment of the present invention, there is provided a program executed by a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected through a transmission device, and is photographed through the transmission device. Image acquisition means for acquiring an image of the measurement object, removal means for removing the light pattern projected and displayed on the transmission device from the image of the measurement object, and irradiation of the light pattern in the image of the measurement object The display area of the measured object thus detected is detected, and the three-dimensional measurement device is caused to function as a luminance correction means for correcting the luminance of the display area. By executing this program, the 3D measurement device removes the light pattern projected on the transmission device, resulting in the display area of the measurement object irradiated with the light pattern becoming dark. It is possible to suppress a decrease in measurement accuracy. Preferably, the program is stored in a storage medium.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[Configuration of 3D measurement system]
FIG. 1 is a schematic configuration diagram of a three-dimensional measurement system according to the present embodiment. As shown in FIG. 1, the three-dimensional measurement system includes a transmission device 1, a projector 2, a camera 3, a three-dimensional measurement device 4, and a measurement object 5.

The transmission device 1 is a transparent device such as a transparent screen, a transparent display, or transparent glass. The transmission device 1 may be a device that can switch between a transmission state in which light is transmitted and a shielding state in which light is blocked. In this case, the transmissive device 1 is in a transmissive state when the measurement object 5 is measured.

The projector 2 irradiates the measurement object 5 with a plurality of light patterns for three-dimensional measurement. The light pattern is, for example, striped, and may be various patterns used in a pattern projection method for three-dimensional measurement. For example, when a three-dimensional position is measured by a spatial encoding method, the projector 2 projects a striped light pattern in which, for example, the light / dark width is changed by two times in order.

The camera 3 is electrically connected to the three-dimensional measuring device 4 and transmits the captured image (also referred to as “captured image Im”) to the three-dimensional measuring device 4. At the time of three-dimensional measurement, the camera 3 photographs the measurement object 5 irradiated with the light pattern by the projector 2 via the transmission device 1 in the transmission state.

The three-dimensional measuring apparatus 4 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like (not shown), and the tertiary of the measuring object 5 based on the captured image Im acquired from the camera 3. Calculate the original coordinates. Even when the transmissive device 1 is in the transmissive state, the light pattern emitted from the projector 2 may appear on the transmissive device 1. Therefore, the three-dimensional measurement apparatus 4 removes the light pattern projected and displayed on the transmission device 1 from the captured image Im, and acquires the three-dimensional coordinates of the measurement object 5 with high accuracy. The CPU of the three-dimensional measuring device 4 functions as “image acquisition means”, “removal means”, and “luminance correction means” in the present invention.

FIG. 2 shows functional blocks of the three-dimensional measuring apparatus 4. As shown in FIG. 2, the three-dimensional measurement apparatus 4 includes a projection pattern processing unit 11, a difference processing unit 12, a luminance correction processing unit 13, and a three-dimensional coordinate calculation processing unit 14.

The projection pattern processing unit 11 generates an image (also referred to as “projection pattern image Ima”) indicating the light pattern projected and displayed on the transmission device 1 before the measurement object 5 is measured. At this time, the projection pattern processing unit 11 projects the light pattern in the absence of the measurement object 5 on the captured image Im projected on the transmission device 1 in which the light pattern is in the transmission state in the absence of the measurement object 5. Difference processing of the captured image Im that has not been performed is performed. Hereinafter, the generation time of the captured image Im to be processed by the projection pattern processing unit 11 is also referred to as “reference time”.

The difference processing unit 12 generates an image obtained by removing the light pattern component from the captured image Im at the time of measurement (also referred to as “pattern removed image Imb”). At this time, the difference processing unit 12 performs a difference process of the projection pattern image Ima obtained by the projection pattern processing unit 11 for the captured image Im at the time of measurement on which the measurement object 5 is displayed.

The luminance correction processing unit 13 corrects the luminance of the display region (also referred to as “target region Rtag”) of the measurement object 5 irradiated with the light pattern in the pattern removal image Imb obtained by the difference processing unit 12. Do. As will be described later, when each pixel value of the projection pattern image Ima is subtracted from each pixel value of the captured image Im at the time of measurement, there arises a problem that the luminance of the target region Rtag decreases. In consideration of the above, the luminance correction processing unit 13 performs a process of increasing the luminance of the target region Rtag in the pattern removal image Imb.

The three-dimensional coordinate calculation processing unit 14 measures the three-dimensional position of the measurement object 5 based on the pattern removal image Imb (also referred to as “luminance correction image Imc”) that has been subjected to luminance correction by the luminance correction processing unit 13. For example, when the spatial coding method is used, a striped light pattern whose brightness width is changed by a factor of two is projected from the projector 2 in order, and when n pattern lights are prepared, the measurement region is projected by pattern light. It is divided into 2 n powers based on the direction. In this case, the three-dimensional coordinate calculation processing unit 14 uses the light pattern projection direction and the pixel position on the brightness correction image Imc for the n-bit spatial code value uniquely given corresponding to the presence or absence of each pattern light. From the relationship with the determined measurement direction, distance information is obtained for all pixels based on the principle of triangulation.

[Pattern removal processing]
Next, a process (pattern removal process) for removing the light pattern projected and displayed on the transmissive device 1 from the captured image Im generated at the time of measurement will be described. The pattern removal process is executed by the projection pattern processing unit 11 and the difference processing unit 12.

First, specific processing of the projection pattern processing unit 11 will be described with reference to FIG. FIG. 3A shows an example of a captured image Im at the reference time on which the light pattern is projected (also referred to as “reference time projected image Imx”), and FIG. 3B shows that the light pattern is not projected. An example of the captured image Im at the reference time (also referred to as “reference non-projected image Imy”) is shown, and FIG. 3C shows an example of the projection pattern image Ima. In FIG. 3A and the like, the projection range 20 of the projector 2 is indicated by a broken line, and the display area (also referred to as “projection pattern area Rp”) of the light pattern projected on the transmissive device 1 is indicated by an alternate long and short dash line.

The projection pattern processing unit 11 targets the reference-time projection image Imx (see FIG. 3A), and the reference-time non-projection image Imy (see FIG. 3B) in which the light pattern is not projected and displayed on the transmission device 1. ) Difference processing. Specifically, the projection pattern processing unit 11 subtracts the RGB pixel value indicated by each pixel of the reference non-projection image Imy from the RGB pixel value of each pixel of the reference projection image Imx. Thereby, the projection pattern processing unit 11 generates a projection pattern image Ima (see FIG. 3C) that is an image from which only the projection pattern region Rp is extracted.

Next, specific processing of the difference processing unit 12 will be described with reference to FIG. 4A shows the captured image Im at the time of measurement, FIG. 4B shows the projection pattern image Ima generated by the projection pattern processing unit 11, and FIG. 4C shows the pattern removal image Imb. Show.

As shown in FIG. 4A, the captured image Im at the time of measurement displays a measurement object 5 and a projection pattern region Rp indicating a light pattern projected on the transmission device 1. Further, the measurement object 5 is irradiated with the light pattern transmitted through the transmissive device 1 in the target region Rtag that is an overlapping region of the measurement object 5 and the projection pattern region Rp.

And the difference process part 12 performs the difference process of the projection pattern image Ima shown in FIG.4 (b) for the picked-up image Im at the time of the measurement shown to Fig.4 (a), and is made into FIG.4 (c). The pattern removal image Imb shown is generated. Specifically, the difference processing unit 12 generates the pattern removal image Imb by subtracting the RGB component indicated by each pixel of the projection pattern image Ima from the RGB component of each pixel of the captured image Im. As a result, in the pattern removal image Imb, the light pattern is projected and displayed only on the target region Rtag, and the light pattern projected on the transmission device 1 is removed.

Thus, the difference processing unit 12 can remove the light pattern projected and displayed on the transmission device 1 and generate the pattern removal image Imb in which only the measurement object 5 is irradiated with the light pattern. As a result, the three-dimensional measurement apparatus 4 can measure without erroneously measuring the transmissive device 1 even when the measurement object 5 is away from the transmissive device 1 as compared with the case where the pattern removal process is not performed. It is possible to perform three-dimensional measurement of the object 5.

[Brightness correction processing]
Next, the process performed by the brightness correction processing unit 13 will be specifically described. In the pattern removal image Imb, as a result of performing the difference process using the projection pattern image Ima, the luminance of the target region Rtag is lowered. Accordingly, the luminance correction processing unit 13 corrects the luminance of the target region Rtag.

First, a method for specifying the target region Rtag will be described with reference to FIGS.

FIG. 5A shows the pattern removal image Imb, FIG. 5B shows the reference non-projection image Imy, and FIG. 5C removes the display other than the measurement object 5 from the pattern removal image Imb. The image “Imb2” is shown.

First, the luminance correction processing unit 13 performs a difference process on the reference non-projection image Imy shown in FIG. 5B for the pattern removal image Imb shown in FIG. Thereby, the brightness correction processing unit 13 generates an image Imb2 (see FIG. 5C) obtained by removing the display other than the measurement object 5 such as the transmission device 1 from the pattern removal image Imb.

Next, the luminance correction processing unit 13 generates an image “Imb3” obtained by extracting only the target region Rtag from the image Imb2. This process will be specifically described with reference to FIG. 6A shows the image Imb2, FIG. 6B shows the projection pattern image Ima, and FIG. 6C shows the image Imb3.

As shown in FIGS. 6A to 6C, the brightness correction processing unit 13 projects the projection pattern area of the projection pattern image Ima (see FIG. 6B) in the image Imb2 (see FIG. 6A). An image Imb3 (see FIG. 6C) in which pixel values other than the pixels overlapping with Rp are set to “0” is generated. Then, the luminance correction processing unit 13 specifies the target region Rtag by binarizing the image Imb3.

Next, a method for generating the brightness correction image Imc from the pattern removal image Imb based on the target region Rtag specified by the image Imb3 will be described.

The brightness correction processing unit 13 generates a brightness correction image Imc in which the brightness of the target region Rtag of the pattern removal image Imb is increased by a predetermined value or a predetermined rate. Here, the luminance correction processing unit 13 sets the predetermined value and the predetermined ratio so that the luminance of the target region Rtag of the luminance correction image Imc is equal to or higher than the luminance of the target region Rtag of the captured image Im at the time of measurement. Set. Thereby, the brightness correction processing unit 13 improves the calculation accuracy of the three-dimensional coordinate position of the measurement object 5.

This will be supplementarily described. In the general calculation of the three-dimensional coordinate position, the three-dimensional coordinate calculation processing unit 14 determines whether or not the light pattern is projected on each pixel of the image in which each light pattern is projected and displayed on the measurement object 5. The determination is made with reference to the luminance of the pixel. In this case, the three-dimensional coordinate calculation processing unit 14 sets a threshold value (also referred to as “threshold value Cth”) for determining whether or not the light pattern is projected, and determines whether or not the light pattern is projected for each pixel. When the luminance of a predetermined pixel when the light pattern is projected is “A” and the luminance of the pixel when the light pattern is not projected is “B”, the stripes satisfying the following expression (1) An area corresponding to the light pattern is obtained (“Cth” is a threshold value).
AB> Cth (1)
Here, generally, the higher the threshold value Cth, the more the erroneous determination of the region corresponding to the striped light pattern is suppressed. On the other hand, when the luminance difference based on the presence or absence of projection of the light pattern is small, if the threshold Cth is increased, it becomes impossible to determine the region corresponding to the striped light pattern. Therefore, the three-dimensional coordinate calculation processing unit 14 can set the threshold value Cth higher as the luminance difference based on the presence or absence of the projection of the light pattern is larger, and suppresses the erroneous determination described above, thereby generating a striped light pattern. The corresponding area can be determined.

In consideration of the above, the luminance correction processing unit 13 sets the luminance of the target region Rtag of the luminance correction image Imc so as to be equal to or higher than the luminance at the time of actual light pattern projection. Thereby, the brightness correction processing unit 13 can improve the calculation accuracy of the three-dimensional coordinates of the measurement object 5.

Note that, as another method for increasing the luminance of the target region Rtag of the pattern removal image Imb, it is conceivable to reduce the luminance of the projection pattern image Ima in advance when calculating the pattern removal image Imb. However, in this case, the light pattern projected and displayed on the transmissive device 1 cannot be completely removed. Therefore, in this case, there is a high probability that the three-dimensional position of the transmission device 1 is erroneously calculated. Compared with the case where three-dimensional measurement is performed using the brightness correction image Imc generated based on the present embodiment, the measurement accuracy is higher. Decreases.

[Robustness of brightness correction processing]
The brightness correction processing unit 13 preferably executes the above-described processing, so that the transmission device 1 is preferably used regardless of the brightness of the shooting environment of the camera 3, the distance between the projector 2 and the measurement object 5, and the type of the measurement object 5. It is possible to generate the brightness correction image Imc on which the light pattern of the measurement object 5 is projected while eliminating the influence of the above. This will be supplementarily described.

(1) Change in brightness Generally, as the shooting environment of the camera 3 becomes brighter, it becomes difficult to determine whether or not a light pattern is irradiated on the measurement object 5 based on the threshold value Cth, and the accuracy of three-dimensional measurement decreases. To do.

Even in this case, in the present embodiment, the luminance correction processing unit 13 increases the luminance of the target region Rtag based on the luminance correction processing described above. Thereby, the luminance correction processing unit 13 can set the threshold value Cth to a larger value as compared with the case where the luminance correction processing is not performed, and can improve the accuracy of the determination processing based on the threshold value Cth. Therefore, the three-dimensional measurement apparatus 4 can suppress a decrease in the accuracy of the three-dimensional measurement even when the shooting environment of the camera 3 is bright.

(2) Change in distance Generally, as the distance between the projector 2 and the measurement object 5 becomes longer, the luminance of the light pattern irradiated to the measurement object 5 becomes smaller. It becomes difficult to determine whether or not it is irradiated, and the accuracy of three-dimensional measurement is reduced.

Even in this case, in the present embodiment, the luminance correction processing unit 13 increases the luminance difference based on the presence or absence of the projection of the light pattern by increasing the luminance of the target region Rtag based on the luminance correction processing described above. Thereby, the luminance correction processing unit 13 can set the threshold value Cth to a larger value as compared with the case where the luminance correction processing is not performed, and can improve the accuracy of the determination processing based on the threshold value Cth. Therefore, the three-dimensional measuring device 4 can suppress a decrease in the accuracy of the three-dimensional measurement even when the distance between the projector 2 and the measurement object 5 is long.

(3) Change of measurement object Generally, the luminance in the irradiated portion of the light pattern may differ depending on whether the optical characteristic of the measurement object 5 is a diffusion type or a reflection type. In this case, when the threshold value Cth common to all the measurement objects 5 is used, the determination accuracy of whether or not the light pattern is irradiated on the measurement object 5 based on the threshold value Cth decreases, and the accuracy of the three-dimensional measurement decreases.

Even in this case, in the present embodiment, the luminance correction processing unit 13 increases the luminance difference based on the presence or absence of the projection of the light pattern by increasing the luminance of the target region Rtag based on the luminance correction processing described above. Thereby, the luminance correction processing unit 13 can set the threshold value Cth to a larger value as compared with the case where the luminance correction processing is not performed, and can improve the accuracy of the determination processing based on the threshold value Cth. Therefore, the three-dimensional measurement apparatus 4 can suppress a decrease in the accuracy of the three-dimensional measurement even when the three-dimensional measurement of the measurement object 5 having different optical characteristics is performed.

In addition, when the background object of the measurement object 5 is changed between the reference time and the measurement time, the three-dimensional measurement device 4 needs to perform processing according to the case according to the difference in the optical characteristics of the background object. This process is described in detail in the next section.

[Processing based on background changes]
In general, the brightness of the projection pattern region Rp varies depending on the optical characteristics of the object that is the background of the measurement object 5. For example, the brightness of the projection pattern region Rp is smaller for a reflective background (for example, a whiteboard) having optical characteristics than for a diffuse background (for example, cloth, white paper).

Taking the above into consideration, the three-dimensional measurement apparatus 4 changes the process to be executed based on the magnitude relationship between the luminance of the light pattern of the transmissive device 1 at the reference time and the luminance of the light pattern of the transmissive device 1 at the time of measurement. In the following, description will be made on a case-by-case basis for each of the above-described luminance magnitude relationships.

Note that the three-dimensional measurement device 4 uses, for example, a reference when the average luminance of the projection pattern region Rp in the projection pattern image Ima is larger than the average luminance of the captured image Im at the time of measurement corresponding to the projection pattern region Rp. It is determined that the brightness of the light pattern of the transmissive device 1 at the time is larger than the brightness of the light pattern of the transmissive device 1 at the time of measurement. Similarly, the three-dimensional measurement apparatus 4 determines that the reference brightness is obtained when the average brightness of the projection pattern area Rp in the projection pattern image Ima is smaller than the average brightness of the captured image Im at the time of measurement corresponding to the projection pattern area Rp. It is determined that the luminance of the light pattern of the transmissive device 1 is lower than the luminance of the light pattern of the transmissive device 1 at the time of measurement.

(1) When the luminance of the light pattern of the transmissive device 1 at the time of reference is the same as the luminance of the light pattern of the transmissive device 1 at the time of measurement At this time, the projection pattern image Ima is calculated from each pixel value of the captured image Im at the time of measurement. When the pattern removal image Imb is generated by performing the subtraction process on each of the pixel values, the projection pattern region Rp does not remain in the pattern removal image Imb as in FIG. Therefore, in this case, the three-dimensional measurement apparatus 4 executes the pattern removal process and the brightness correction process described above.

(2) When the luminance of the light pattern of the transmissive device 1 at the reference time is larger than the luminance of the light pattern of the transmissive device 1 at the time of measurement, the projection pattern image Ima is similarly calculated from each pixel value of the captured image Im at the time of measurement. When the pattern removal image Imb is generated by subtracting each of the pixel values, the projection pattern region Rp does not remain in the pattern removal image Imb as in FIG. 4C. Therefore, in this case, the three-dimensional measurement apparatus 4 executes the pattern removal process and the brightness correction process described above.

(3) When the luminance of the light pattern of the transmissive device 1 at the reference time is smaller than the luminance of the light pattern of the transmissive device 1 at the time of measurement At this time, each pixel of the projection pattern image Ima from each pixel value of the captured image Im at the time of measurement When the pattern removal image Imb is generated by subtracting the value, the projection pattern region Rp remains in the pattern removal image Imb. Therefore, in this case, for example, the difference processing unit 12 performs correction to increase the luminance of each pixel in the projection pattern region Rp of the projection pattern image Ima by a predetermined value or a predetermined rate. And the difference process part 12 produces | generates the pattern removal image Imb by performing the difference process of the projection pattern image Ima after the said brightness correction with respect to the picked-up image Im at the time of measurement. The predetermined value and the predetermined rate described above are set to values at which the luminance of each pixel of the projection pattern region Rp of the projection pattern image Ima is equal to or higher than the luminance of each pixel of the projection pattern region Rp of the captured image Im at the time of measurement. The

Thereby, the difference processing unit 12 can suitably generate the pattern removal image Imb from which the projection pattern region Rp has been removed. Then, the luminance correction processing unit 13 executes luminance correction processing based on the pattern removal image Imb generated by the difference processing unit 12.

[Modification]
Hereinafter, modified examples suitable for the above-described embodiments will be described. The following modifications may be applied in any combination to the above-described embodiments.

(Modification 1)
When correcting the luminance of the target region Rtag, the luminance correction processing unit 13 may correct the luminance of the target region Rtag based on the luminance of the pixels of the measurement object 5 other than the target region Rtag of the captured image Im at the time of measurement. Specifically, the luminance correction processing unit 13 identifies a pixel having a higher luminance among the pixels of the measurement object 5 other than the target region Rtag of the captured image Im at the time of measurement, and the luminance of the target region Rtag is higher than the luminance. Set to be higher. Also by this, the luminance correction processing unit 13 can execute the determination process based on the threshold value Cth with high accuracy.

(Modification 2)
The three-dimensional measurement device 4 generates a pattern removal image Imb from the captured image Im at the time of measurement by pattern removal processing, and then recognizes the target region Rtag in the luminance correction processing, and sets the luminance of the target region Rtag of the pattern removal image Imb. Raised. However, the processing procedure to which the present invention is applicable is not limited to this.

Instead, the three-dimensional measuring device 4 first recognizes the target area Rtag, and then lowers the luminance of the target area Rtag of the projection pattern image Ima. Then, in the pattern removal process, the three-dimensional measurement apparatus 4 generates a pattern removal image Imb by performing a difference process on the projection pattern image Ima with the brightness reduced, for the captured image Im at the time of measurement. In this case, the luminance of the target region Rtag of the pattern removal image Imb has already been corrected to a high value as much as the luminance of the target region Rtag of the projection pattern image Ima is lowered. Therefore, in this case, the three-dimensional coordinate calculation processing unit 14 performs a three-dimensional measurement process based on the above-described pattern removal image Imb.

In any case, the three-dimensional measurement device 4 recognizes the target region Rtag that is the projection region of the light pattern of the measurement object 5, thereby increasing the luminance of the target region Rtag and improving the three-dimensional measurement accuracy. Can be made.

(Modification 3)
When a background object exists behind the measurement object 5 and the background object is brightly photographed by irradiation of the light pattern, the reference projection image Imx and the projection pattern image Ima are irradiated on the transmission device 1. In addition to the light pattern, the light pattern irradiated on the background is displayed.

In this case, when the projection pattern processing unit 11 performs the difference process of the reference non-projection image Imy on the reference projection image Imx to generate the projection pattern image Ima, the projection pattern image Ima includes the projection pattern region Rp. The portion of the background object that has been illuminated by the irradiation of the light pattern is also displayed. Therefore, in this case, when the difference processing unit 12 performs the difference processing of the pattern removal image Imb on the captured image Im at the time of measurement, the background object is irradiated with the light pattern in addition to the projection pattern region Rp in the pattern removal image Imb. As a result, there is a problem that the brightness of the illuminated portion is also lowered.

Even in this case, the luminance correction processing unit 13 determines the overlapping portion of the pattern removal image Imb generated by the difference processing unit 12 and the projection pattern image Ima as the target region Rtag, as in the above-described embodiment, In the removed image Imb, the luminance of the target region Rtag is increased. Thereby, the luminance correction processing unit 13 can increase the luminance of the overlapping portion of the measurement object 5 and the portion illuminated by the light pattern irradiation in addition to the overlapping portion of the measurement object 5 and the projection pattern region Rp. it can.

As described above, the three-dimensional measurement device 4 appropriately adjusts the luminance of the portion of the measurement object 5 irradiated with the light pattern even when the background object is photographed brightly by the irradiation of the light pattern. Can do.

(Modification 4)
In the above-described embodiment, the projection pattern processing unit 11 acquires the reference-time projection image Imx from the camera 3 for each light pattern, and generates and stores the projection pattern image Ima for each light pattern. Instead, the projection pattern processing unit 11 uses a projection pattern image Ima (““ reference pattern image ”) on which a predetermined light pattern is displayed to display another projection pattern image Ima (“ May also be generated and stored.

In this case, the projection pattern processing unit 11 recognizes the position of the projection pattern region Rp to be displayed on each of the other pattern images by storing in advance information on the shape and arrangement of each light pattern in the projection range 20. Then, the projection pattern processing unit 11 sets each pixel value of the projection pattern region Rp to be displayed on each of the other pattern images based on the pixel value of the projection pattern region Rp displayed on the reference pattern image.

In the above-described embodiment, the projection pattern processing unit 11 generates the projection pattern image Ima based on the reference-time projection image Imx and the reference-time non-projection image Imy previously captured when the measurement object 5 is not in the imaging range. . That is, the reference time and the measurement time are set at different times. Instead, the projection pattern processing unit 11 may generate the projection pattern image Ima based on the captured image Im at the time of measurement.

In this case, for example, the projection pattern processing unit 11 recognizes, from the captured image Im at the time of measurement, a part of the projection pattern region Rp that does not overlap the measurement object 5, for example, a stripe of the light pattern formed at the left end or the right end. . And the projection pattern process part 11 produces | generates the projection pattern image Ima which set the pixel value of the other stripe of the light pattern containing the part which overlapped with the measurement object 5 with reference to the pixel value of the said stripe. In this case, the projection pattern processing unit 11 stores in advance information on the shape and arrangement of each light pattern in the projection range 20, and recognizes the positions and ranges of other stripes of the light pattern based on the information. This eliminates the need for the camera 3 to capture the reference-time projection image Imx and the reference-time non-projection image Imy in advance in order to generate the projection pattern image Ima before the measurement object 5 is measured.

(Modification 5)
In the above-described embodiment, the three-dimensional measurement apparatus 4 performs a determination process based on the threshold value Cth based on the luminance determined by the RGB components. Instead, the three-dimensional measurement device 4 may perform a determination process using the threshold value Cth based on at least one value of RGB.

Further, the projector 2 may be a projector that irradiates an infrared light pattern, and the camera 3 may be an infrared camera. In this case, the three-dimensional measurement apparatus 4 performs the same processing as in the above-described embodiment based on the captured image Im transmitted from the infrared camera.

(Modification 6)
In the three-dimensional measurement system shown in FIG. 1, one transmission device 1 is installed between the projector 2, the camera 3, and the measurement object 5. Instead, the projector 2, the camera 3, and the measurement object A plurality of transmission devices 1 may be installed between the first and second transmission devices 1.

(Modification 7)
The projector 2 may emit light for displaying information on the transmissive device 1 (also referred to as “display light”) and light constituting a light pattern (also referred to as “pattern light”). . In this case, the transmissive device 1 is, for example, a display that can be switched between a transmissive state and a shielding state, and is in a shielding state when display light is emitted from the projector 2, and is transmissive when pattern light is emitted from the projector 2. It becomes a state. According to this configuration, the three-dimensional measurement system appropriately controls the display rate and the like so that a human can visually recognize the image by the display light and the measurement object 5 such as a hand. Three-dimensional measurement is possible.

In this case, the projector 2 may be realized by two projectors: a projector that emits display light and a projector that emits pattern light.

The present invention can be suitably applied to non-contact input devices, display devices, and other devices that perform three-dimensional measurement.

DESCRIPTION OF SYMBOLS 1 Transmission device 2 Projector 3 Camera 4 Three-dimensional measuring device 5 Measurement object

Claims (12)

1. A three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device,
   Image acquisition means for acquiring an image of the measurement object photographed via the transmission device;
   Removing means for removing the light pattern projected and displayed on the transmission device from the image of the measurement object;
   Brightness correction means for detecting a display area of the measurement object irradiated with the light pattern in the image of the measurement object and correcting the brightness of the display area;
   A three-dimensional measuring apparatus comprising:
2. The brightness correction unit detects, in the image of the measurement object, an overlapping portion of the area where the light pattern is removed and the display area of the measurement object as a display area of the measurement object irradiated with the light pattern. The three-dimensional measuring apparatus according to claim 1, wherein:
3. 3. The three-dimensional image according to claim 1, wherein the luminance correction unit corrects the luminance of the display area of the measurement object irradiated with the light pattern based on luminance before processing by the removing unit. Measuring device.
4. The brightness correction means corrects the brightness of the display area of the measurement object irradiated with the light pattern based on the brightness of the display area of the measurement object not irradiated with the light pattern. The three-dimensional measuring apparatus according to any one of claims 1 to 2.
5. The removing means includes
   Projection pattern processing means for generating an image showing the light pattern projected and displayed on the transmission device;
   Difference processing means for performing difference processing of the image generated by the projection pattern processing means for the measurement object image;
   5. The three-dimensional measuring apparatus according to claim 1, wherein
6. The projection pattern processing means targets the image of the transmission device on which the light pattern is projected in a state where the measurement object does not exist, and the transmission device in which the light pattern is not projected in a state where the measurement object does not exist The three-dimensional measurement apparatus according to claim 5, wherein an image indicating the light pattern is generated by performing the difference processing of the image.
7. The difference processing means includes
   The brightness of the light pattern projected and displayed on the transmission device in an image photographed in the absence of the measurement object is smaller than the brightness of the light pattern projected and displayed on the transmission device in the image of the measurement object. If we detect that
   The three-dimensional measurement apparatus according to claim 6, wherein the difference process is performed by increasing a pixel value of an image generated by the projection pattern processing unit.
8. The removing means further removes the light pattern projected and displayed on the background object of the measurement object from the image of the measurement object,
   The brightness correction unit detects a display area of the measurement object irradiated with the light pattern in the image of the measurement object, and corrects the brightness of the display area, among the areas to be processed by the removal unit. The three-dimensional measuring apparatus according to claim 1, wherein
9. A transmission device;
   A three-dimensional measuring apparatus according to any one of claims 1 to 8,
   A projector that emits the light pattern to the measurement object via the transmission device;
   A camera that transmits an image obtained by capturing the measurement object via the transmission device to the three-dimensional measurement apparatus;
   A three-dimensional measurement system characterized by comprising:
10. A control method executed by a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device,
    An image acquisition step of acquiring an image of the measurement object photographed via the transmission device;
    A removal step of removing the light pattern projected and displayed on the transmission device from the image of the measurement object;
    A luminance correction step of detecting a display area of the measurement object irradiated with the light pattern in the image of the measurement object and correcting the luminance of the display area;
    A control method characterized by comprising:
11. A program executed by a three-dimensional measurement apparatus that performs three-dimensional measurement of a measurement object onto which a light pattern is projected via a transmission device,
    Image acquisition means for acquiring an image of the measurement object photographed via the transmission device;
    Removing means for removing the light pattern projected and displayed on the transmission device from the image of the measurement object;
    A program for detecting a display area of the measurement object irradiated with the light pattern in the image of the measurement object and causing the three-dimensional measurement apparatus to function as a luminance correction unit that corrects the luminance of the display area. .
12. A storage medium storing the program according to claim 11.

Images