JP6571948B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to a technique for obtaining a correspondence relationship between a camera and a projector.

  Projecting pattern light from the projector onto the object to be measured and observing the reflected light by the imaging device, the correspondence between the projected coordinate and the imaging coordinate is obtained, and the 3D measurement device that obtains the 3D coordinate by the principle of triangulation Widely known. In the three-dimensional measurement method of projecting pattern light and observing reflected light, when a black object or metal object having a low diffuse reflectance is a measurement target object, the reflected light is small and the observation brightness of the pattern light is lowered. Therefore, it is greatly affected by noise such as shot noise, and the correspondence between the projection coordinates and the imaging coordinates cannot be obtained with high accuracy, so that the three-dimensional measurement accuracy is deteriorated.

  As one method for dealing with this problem, Patent Document 1 increases the observation luminance and reduces the influence of noise by extending the exposure time of a specific pattern. In Non-Patent Document 2, the spread of the light output from the projector is narrowed, the illuminance of the pattern light is increased, and the observation luminance necessary for highly accurate measurement is obtained.

JP 2014-115264 A

Gupta, Mohitten Agrawal, Amitten Veeraraghavan, Ashok; Narashimhan, Srinivasa G, Structured Light In Sunlight, ICCV, 2013 Gupta, Mohit; Agrawal, Amit; Veeraraghavan, Ashok; Narasimhan, Srinivasa G, Structured Light 3D Scanning Under Global Illumination, CV

  However, the method of Patent Document 1 has a problem that it takes a measurement time to increase the exposure time. . Further, the method of Non-Patent Document 1 has a problem that the measurement range is narrowed because the range in which the pattern light is projected is narrowed.

  The present invention has been made in view of the above problems, and an object of the present invention is to obtain a correspondence between a projection coordinate and an imaging coordinate with high accuracy even when the observation luminance of a target object onto which a pattern is projected is small and affected by noise. And

In order to solve the above-described problem, the information processing apparatus according to the present invention acquires , for example, a plurality of images obtained by capturing a target object in which a plurality of types of patterns having a bright part and a dark part are projected in time series by a projection unit. An image acquisition unit; an applying unit that applies a label corresponding to the luminance of the pixel to pixels in the plurality of images; and a bright portion and a dark portion in one of the plurality of projected patterns And a changing unit that changes the assigned label based on a constraint that the boundary does not occur in another pattern, and a projection plane of the projection unit based on the changed label. And a correspondence relationship deriving unit for deriving a correspondence relationship between the position of the pattern and the position of the pattern in the acquired image.

  According to the present invention, even when the observation brightness of the target object onto which the pattern is projected is small and affected by noise, the correspondence between the projection coordinates and the imaging coordinates can be obtained with high accuracy.

1 is a block diagram illustrating a configuration example of an information processing apparatus according to a first embodiment. The figure which shows the pattern of a gray code as an example of a space division | segmentation pattern. The figure which shows the code | symbol of the pattern edge position used as the light-dark boundary in the pattern 1. FIG. The figure which shows the code | symbol of the pattern edge position used as the light-dark boundary in the pattern 2. FIG. The figure explaining the process by the edge detection means of 1st Embodiment. The measurement processing flowchart by the information processing apparatus of 1st Embodiment. The figure which shows the pattern of MMSW (Maximum Min-Stripe-Width) as an example of a space division | segmentation pattern. The figure which shows a comparison of a part of pattern with the narrowest stripe width of Gray code and MMSW. The figure which shows the pattern of MMSW in a continuous pixel. The figure which shows the result of having binarized the captured image. The figure explaining the process by the information processing apparatus of 2nd Embodiment. The figure explaining the process by the information processing apparatus of 2nd Embodiment. The figure explaining the process by the information processing apparatus of 2nd Embodiment. The flowchart explaining the measurement process by the three-dimensional measuring apparatus of 2nd Embodiment. The block diagram which shows the structural example of the three-dimensional measuring apparatus of 3rd Embodiment. The figure explaining the process by the information processing apparatus of 3rd Embodiment. The figure explaining the process by the information processing apparatus of 3rd Embodiment. The processing flowchart by the information processor of a 3rd embodiment. The figure which shows an example of the projection pattern which consists of a multi-valued luminance level. 10 is a flowchart of processing performed by an information processing apparatus according to a fourth embodiment. The block diagram which shows the structural example of the information processing apparatus of 5th Embodiment. 10 is a processing flowchart of an information processing apparatus according to a fifth embodiment. The figure which shows the example of the hardware constitutions of the information processing apparatus of this invention.

  Hereinafter, an information processing apparatus according to an embodiment of the present invention will be described in detail.

  Prior to describing the embodiments according to the present invention, a hardware configuration in which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 23 is a hardware configuration diagram of the information device according to the present embodiment. In the figure, a CPU 2310 comprehensively controls devices connected via a bus 2300. The CPU 2310 reads and executes processing steps and programs stored in a read-only memory (ROM) 2320. In addition to the operating system (OS), each processing program, device driver, and the like according to this embodiment are stored in the ROM 2320, temporarily stored in a random access memory (RAM) 2330, and appropriately executed by the CPU 2310. The input I / F 2340 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, or the like). The output I / F 2350 is output as an output signal in a format that can be processed by the display device to an external device (display device).

  Each of these functional units is realized by the CPU 2310 developing a program stored in the ROM 2320 in the RAM 2330 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 2310, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

(First embodiment)
The information processing apparatus according to the first embodiment corrects and determines a measurement target region identification error caused by the influence of noise based on the structural features of the projected gray code pattern group, thereby achieving a highly accurate tertiary The original coordinates can be calculated.

[Device configuration]
The block diagram of FIG. 1 shows a configuration example of a three-dimensional measurement apparatus 1000 including the information processing apparatus 1100 of the present embodiment. The projection device 1 projects the space division pattern light onto the measurement target object. The pattern is reflected by the surface of the measurement target object and imaged by the imaging device 2. The image captured by the imaging device 2 is sent to the information processing device 1100, and the information processing device 1100 calculates the three-dimensional coordinates of the target object. Further, the information processing apparatus 1100 controls the operations of the projection apparatus 1 and the imaging apparatus 2.

  The information processing apparatus 1100 includes a projection control unit 101, an image input unit 102, a label applying unit 103, a label changing unit 104, an edge detecting unit 105, a three-dimensional coordinate calculating unit 106, and an output unit 107.

  The projection control unit 101 holds a gray code pattern image including a bright part and a dark part shown in FIG. 2, and controls the projection apparatus 1 so that patterns are sequentially formed from a low frequency pattern to a high frequency pattern of the gray code pattern. Simultaneously with the projection, a control signal is output to the image input unit 102. However, the pattern to be projected is not limited to the gray code, and may be a space division pattern group having the same characteristics as those of the gray code pattern group described later.

  The image input unit 102 controls the imaging device 2 in accordance with the timing of the control signal input from the projection control unit 101 to receive an image, and receives a captured image on which a pattern is projected (image acquisition). Further, the captured image is sequentially output to the label attaching unit 103. However, pattern projection and imaging do not have to be performed in order, and the order may be changed. By projecting the light wavelengths of each pattern separately and using an imaging device corresponding to each wavelength, all patterns May be simultaneously projected and imaged.

  The label assigning unit 103 assigns a binary (white, black) label according to the luminance value of the captured image on which the pattern is projected. The binarization may be performed using a method similar to that of normal spatial coding. For example, when the luminance histogram of the captured image is divided by a certain threshold value, the binarization may be performed by a threshold value that maximizes the interclass variance. The label assigning unit 103 outputs the obtained binarization result to the label changing unit 104.

  The label changing unit 104 uses the feature of the gray code pattern to change the label in order to correct the labeling error of the binarization result input from the label attaching unit 103. The label changing unit 302 outputs the obtained binarization result to the edge detecting unit 105.

  The edge detection unit 105 detects a subpixel edge from the binarization result input from the label change unit 104. The detected edge position of subpixel accuracy is output to the label changing unit 104. When all the patterns have been processed, the binarization result of each pattern and the subpixel edge information are integrated, spatial encoding is performed, and the correspondence between the projected coordinates and the image coordinates is obtained (correspondence derivation). The obtained correspondence between the projected coordinates and the image coordinates is output to the three-dimensional coordinate calculation unit 106.

  The three-dimensional coordinate calculation unit 106 uses a correspondence relationship between the projection coordinates (coordinates on the projection plane, that is, image coordinates of the projector) input from the edge detection unit 105 and the image coordinates to correspond to the three-dimensional edge corresponding to the subpixel edge. Deriving measurement points (three-dimensional coordinate derivation). The three-dimensional coordinates are calculated (derived) from the correspondence between the projection coordinates and the image coordinates and the calibration data of the projection apparatus 1 and the imaging apparatus 2 obtained in advance.

  The output unit 107 outputs the obtained three-dimensional coordinates to a monitor, another computer, a server device, an auxiliary storage device, various recording media, or the like.

  Each of these functional units is realized by the CPU 2310 developing a program stored in the ROM 2320 in the RAM 2330 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 2310, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

[Measurement processing]
The measurement process by the information processing apparatus 1100 of the first embodiment will be described with reference to the flowchart of FIG.

(Step S11)
In step S11, when the three-dimensional measurement apparatus 1000 is activated, an initialization process is executed. The initialization process includes a process of starting up the projection apparatus 1 and the imaging apparatus 2, a process of reading calibration data and projection patterns of the projection apparatus 1 and the imaging apparatus 2, and the like.

(Step S12)
In step S <b> 12, the projection control unit 101 causes the projection apparatus 1 to project patterns in the order of the gray code pattern low-frequency pattern to high-frequency pattern, and sends a control signal to the image input unit 102. Upon receiving the control signal, the image input unit 102 causes the imaging device 2 to capture an image on which the pattern is projected, and sends the captured image to the label applying unit 103.

(Step S13)
In step S13, the label attaching unit 103 that has received the captured image provides a binary label to the captured image on which the pattern is projected.

(Step S14)
In step S <b> 14, the label changing unit 104 changes the label based on the feature of the gray code so as to maintain the pattern group constraint using the edge positions detected so far. As shown in FIG. 2, the gray code has a feature that the hamming distance between adjacent codes is 1 (for example, 00100 and 00110). Therefore, adjacent projection pixels that form edges of a certain pattern (hereinafter referred to as pattern edge positions). Is the same brightness in the other patterns. The Gray code projects one edge for each identified region. Therefore, when processing is performed in order from the low-frequency pattern to the high-frequency pattern, when the edge is detected, the correspondence between the imaging pixel forming the edge (hereinafter referred to as the edge position) and the projection coordinates is obtained, and the subsequent positions of the edge position are determined. The label in the pattern can be determined.

  For example, at the pattern edge position indicated by the frame set in the pattern 1 of FIG. 3, the light and darkness of the pattern 1 is different, the light of the pattern 2 is bright, and the light is dark after the pattern 3. Furthermore, at the two pattern edge positions shown by the frame in FIG. 4, light and dark are different in pattern 2, light is determined in pattern 3, and dark is determined after pattern 4. As described above, the pattern edge positions in the gray code are determined such that if the brightness of a certain pattern is different, the next pattern is bright, and the subsequent pattern is dark. Therefore, using this feature (referring to), when decoding a higher frequency pattern, the label is changed with the label already determined at the edge position detected with the lower frequency pattern. Do. The lower the frequency pattern, the thicker the light and dark stripes, the higher the contrast and the easier the labeling, so the edge position is also likely. Therefore, a more probable label can be given preferentially by processing in order from the low frequency pattern. The edge position is obtained from the subsequent edge detection unit 105. In this way, a more plausible binarization result in which the restrictions on the pattern group (pattern time-series characteristics) are maintained is obtained.

(Step S15)
In step S15, the edge detection unit 105 detects an edge position and a subpixel edge based on the binarization result. The processing by the edge detection unit 105 is shown in FIG. First, the binarization result is scanned in the horizontal direction of the image (if the edge is in the horizontal direction, the vertical direction of the image), and adjacent pixels at the boundary where the white label and the black label are interchanged are extracted and set as edge positions. Then, a subpixel edge whose edge positions are smoothly connected is obtained by performing function fitting or the like. Note that the edge position need not be limited to adjacent pixels, and may be widened by a plurality of pixels. Also, the method for obtaining the subpixel edge need not be limited to the function fitting. For example, in addition to the normal pattern of FIG. 2, a bright / dark reversal pattern may be projected and imaged, and the intersection position formed when the observed luminance of the normal pattern and the reversal pattern is plotted may be used as the subpixel edge. The edge detection unit 105 stores the binarization result of the pattern thus obtained and the subpixel edge, and outputs the edge position to the label change unit 104.

(Step S16)
In step S16, the edge detection unit 105 determines whether projection, imaging, and processing of all pattern images have been completed. If completed, the process proceeds to step S17, and if not completed, the process returns to step S12.

(Step S17)
In step S <b> 17, the three-dimensional coordinate calculation unit 106 calculates three-dimensional coordinates from the correspondence between the projected coordinates and the image coordinates, and sends them to the output unit 107. Then, the output unit 107 outputs the three-dimensional coordinate data, and the process ends.

  Note that the processing in step S12 is not necessarily included in the repetition. After all patterns are projected and imaged in step S12, the processing from step S13 to step S15 may be repeated, or parallel processing may be performed. Moreover, it is not necessary to perform the process of step S13 and step S14 in the order shown to a figure, and you may replace an order suitably.

  With the configuration described above, using the geometric features of the projection pattern, correcting the spatial code sign error reduces the effect of noise generated on the captured image and calculates the three-dimensional coordinates with high accuracy. can do.

(Second Embodiment)
In the second embodiment, a maximum min stripe-width (hereinafter referred to as MMSW) pattern that is more suitable for measurement of an object that is less reflective than the gray code is projected, and spatial encoding is performed. Furthermore, high-precision three-dimensional coordinates are calculated by correcting and determining an identification error of the measurement target region that has occurred under the influence of noise based on the characteristics of the MMSW pattern group.

[Device configuration]
The information processing apparatus 2100 according to the second embodiment of the present invention will be described below.

  The information processing apparatus 2100 includes a projection control unit 201, an image input unit 202, a label assignment unit 203, a label change unit 204, an edge detection unit 205, a three-dimensional coordinate calculation unit 206, and an output unit 207. Since the block diagram of the present embodiment has the same configuration as that of the first embodiment (FIG. 1), the block diagram is omitted.

  The projection control unit 201 holds the pattern image of the MMSW shown in FIG. The MMSW pattern is a pattern designed so that the narrowest stripe is as thick as possible. Therefore, as shown in FIG. 8, MMSW is a code composed of a stripe whose width of the thinnest stripe is thicker than that of a gray code, and therefore has a feature that the contrast is high at the most disadvantageous and is not easily affected by noise. Therefore, it is suitable for the measurement of lower reflection objects. The projector 1 is controlled to project the MMSW pattern, and at the same time, a control signal is output to the image input unit 2. However, the pattern to be projected need not be limited to the MMSW, and may be a space division pattern group having the characteristics of the MMSW pattern group described in the description of the label changing unit 204 described later. Moreover, the projection and imaging of the patterns are in no particular order, and by projecting the light wavelengths of each pattern separately and using an imaging device corresponding to each wavelength, all patterns may be projected and imaged simultaneously.

  The label changing unit 204 corrects the labeling error of the binarization result input from the label adding unit 203 using the feature of the MMSW pattern. The MMSW has a feature that the hamming distance of the pattern edge position is 1 like the gray code. Therefore, after the binarization of all patterns is completed, if the label is changed so that the hamming distance of the edge position becomes 1, a binarization result in which the pattern group constraint is maintained can be obtained. The obtained binarization result is output to the edge detection unit 205.

  Other operations are the same as those in the first embodiment, and thus are omitted.

[Measurement processing]
The measurement process by the three-dimensional measurement apparatus 2000 of the second embodiment will be described with reference to the flowchart of FIG.

(Step S22)
In step S <b> 22, the projection control unit 201 causes the projection apparatus 1 to project MMSW pattern light and sends a control signal to the image input unit 202. Upon receiving the control signal, the image input unit 202 causes the imaging device 2 to capture an image on which the pattern is projected. Step S22 is repeated until the projection and imaging of all pattern images are completed by the determination in step S23. After completion, the captured image is sent to the image processing unit 203.

(Step S24)
In step S24, the label assigning unit 203 that has received the captured image provides a binary label to the captured image on which the pattern is projected.

(Step S25)
In step S25, the label changing unit 204 changes the label based on the features of the MMSW so that the restrictions on the pattern group are maintained.

  In this embodiment, the labeling error of the binarization result input from the label assigning unit 203 is corrected using the feature of the MMSW pattern. The MMSW has a feature that the hamming distance of the pattern edge position is 1 like the gray code. Therefore, after the binarization of all patterns is completed, if the label is changed so that the hamming distance of the edge position becomes 1, a binarization result in which the pattern group constraint is maintained can be obtained.

  A method for correcting a labeling error using the features of the MMSW pattern will be described in detail. Here, in the MMSW pattern of FIG. 7, the flow of processing will be described by taking the 511th to 515th pixels that are continuous in the horizontal direction as an example. FIG. 9 shows a projection pattern of MMSW in the continuous pixels from 511 to 515 pixels. As shown in FIG. 9, the Hamming distance of the pattern edge position of the MMSW is 1, and the brightness changes only in the part surrounded by the frame. For each projection pixel, a corresponding code is determined. For example, the code corresponding to the 513th pixel is 0000110011, and the MMSW uses the correspondence table to associate the projection pixel with the code. Hereinafter, a projection pixel corresponding to a certain code is referred to as a corresponding pixel. The result of binarizing the captured image of each pattern is shown in FIG. However, in FIG. 10, it is assumed that a part of the binarization error is included.

  First, an error candidate having a possibility of error is detected. In the MMSW, since each pattern is a thick stripe, it is considered that a different label is hardly shaken locally. Therefore, as in the gray pixel of FIG. 11, in each pattern, the label changes as white-black-white (1-0-1) or black-white-black (0-1-0) for each pixel. Since the center pixel is likely to be erroneous, it is determined as an error candidate. Further, as in the gray frame in FIG. 11, adjacent pixels in which the white label and the black label that do not include an error candidate change are extracted and set as edge positions.

  Next, a method for determining whether an error candidate label is an error based on the classification of FIG. 12 and changing the label as necessary will be described.

(Step S1201)
First, an error candidate pixel that is isolated as in the region 6001 of FIG. 11 is regarded as an error, and is changed to a black label if it is a white label, and to a white label if it is a black label.

(Step S1202)
Next, processing of pixels in which error candidates are continuous as in regions 6002 and 6003 will be described. In step S1202, it is determined whether one of the pixels is an edge position as in the region 6002. If one of the pixels is an edge position, the white label of the pixel at the edge position, that is, the right pixel of the region 6002 is changed to a black label so that the Hamming distance between that pixel and the pixel at the other edge position is 1. . Because the surface of the measurement target object in these two pixels is projected as a discontinuous surface with a Hamming distance of 2, the portion corresponding to the pattern edge position is projected on the continuous surface, This is because it is more likely that the Hamming distance is 2 because the right pixel of the region 6002 is incorrect.

(Step S1203)
Next, in step S1203, it is determined whether or not both pixels are edge positions with respect to the pixel in which one pixel is determined not to be an edge position in step S1202. Then, as in the region 6003, when both pixels of consecutive error candidates are edge positions, the label is changed so that the Hamming distance becomes 1 and the difference between the corresponding pixels becomes small. . This is because it is considered that the corresponding pixel frequently changes in a plurality of continuous imaging pixels does not occur unless the measurement target object surface has a sharp uneven shape. In the example of FIG. 11, the corresponding pixel of the right pixel in the region 6003 is 68, and the corresponding pixel of the left pixel is 512. When the black label of the right pixel is changed to the white label, the Hamming distance is 1, the corresponding pixel is 513, and the difference between the corresponding pixels is 1. On the other hand, when the white label of the left pixel is changed to the black label, the Hamming distance is 1, but the corresponding pixel is 511, and the difference between the corresponding pixels is 443. Therefore, since the difference in the corresponding pixels between adjacent imaging pixels becomes smaller when the label of the right pixel is changed, the label may be changed by regarding the right pixel as an error. Also, if the consecutive error candidate pixels are not at the edge position, it cannot be determined which is the error, so the labels of both pixels may be changed, or the label of one of the pixels may be changed. Also good.

  By using the features of the MMSW pattern group in this way, a more likely binarization result as shown in FIG. 13 with the pattern group constraints maintained is obtained. Error candidate detection need not be limited to each pixel, and may be detected every two or more pixels. Further, the edge position need not be limited to adjacent pixels, and may have a certain pixel width. The obtained binarization result is output to the edge detection unit 205.

(Step S26)
In step S26, the edge detection unit 205 detects a subpixel edge based on the binarization result, and obtains a correspondence relationship between the projection coordinate and the image coordinate by spatial encoding.

(Step S27)
In step S <b> 27, the three-dimensional coordinate calculation unit 206 calculates the three-dimensional coordinates from the correspondence between the projected coordinates and the image coordinates, and sends them to the output unit 207. Then, the output unit 207 outputs the three-dimensional coordinate data, and the process ends.

  In this way, using the MMSW configured with thick stripes as the space division pattern group and using the features of the MMSW pattern group, the effect of noise generated on the captured image is reduced, and the three-dimensional coordinates are highly accurate. Can be calculated.

(Third embodiment)
The apparatus described in the third embodiment is a three-dimensional measurement apparatus using pattern projection. When performing binarization labeling indicating the presence / absence of projection using an MMSW pattern or the like, an ambiguous area is used. Temporarily assigns a label indicating that it is ambiguous, and finalizes binarization labeling based on spatial neighborhood information (distribution) in the subsequent processing. By doing in this way, it becomes possible to prevent the identification error of the measurement target area from being affected by noise, and it is possible to calculate highly accurate three-dimensional coordinates.

[Device configuration]
An information processing apparatus 3100 according to the third embodiment includes a projection control unit 301, an image input unit 302, a label attaching unit 303, a label changing unit 304, an edge detecting unit 305, a second label changing unit 306, and a three-dimensional coordinate calculation. A unit 307 and an output unit 308. The information processing apparatus 3100 according to the third embodiment of the present invention will be described below with reference to the block diagram of FIG. In the third embodiment, the configuration of FIG. 1 is partially changed. Here, the description will focus on the parts different from the first embodiment.

  The projection control unit 301 holds an MMSW pattern image as in the second embodiment, and a description thereof is omitted because the same processing is performed. The projection pattern need not be limited to the MMSW, but a pattern that provides high contrast is desirable.

  The label attaching unit 303 assigns ternary (white, gray, black) labels according to the luminance value of the captured image on which the pattern is projected. The obtained ternarization result is output to the label changing unit 304.

  The label changing unit 304 optimizes the ternarization result input from the label assigning unit 303 as an initial value using information that there is a high possibility that the label is spatially close in the captured image, decide. The obtained ternary optimization result is output to the edge detection unit 305.

  The edge detection unit 305 detects a subpixel edge (boundary position) from the ternary optimization result input from the label change unit 304. Details of the processing will be described later. The obtained subpixel edge is output to the second label changing unit 306.

  The second label changing unit 306 determines the white label and black label of the ambiguous region (gray label) according to the subpixel edge input from the edge detection unit 305, and performs spatial encoding. Details of the processing will be described later.

  Other configurations are the same as those in the first embodiment, and are omitted.

[Measurement processing]
A measurement process performed by the three-dimensional measurement apparatus 3000 according to the third embodiment will be described with reference to the flowchart of FIG.

(Step S31)
In step S31, when the three-dimensional measurement apparatus 3000 is activated, an initialization process is executed. The initialization process includes a process of starting up the projection apparatus 1 and the imaging apparatus 2, a process of reading calibration data and projection patterns of the projection apparatus 1 and the imaging apparatus 2, and the like.

(Step S32)
In step S <b> 32, the projection control unit 301 causes the projection apparatus 1 to project the MMSW pattern and sends a control signal to the image input unit 302. Upon receiving the control signal, the image input unit 302 causes the imaging device 2 to capture an image on which the pattern is projected, and sends the captured image to the label applying unit 303.

(Step S33)
In step S <b> 33, the label attaching unit 303 that has received the captured image provides a ternary label to the captured image on which the pattern is projected. The label assigning unit 303 assigns a ternary (white, gray, black) label according to the luminance value of the captured image on which the pattern is projected. In spatial coding using light and dark patterns such as gray code, normally captured images are labeled and identified as binary values (white and black), but here ambiguous and undecidable parts are not forcibly binarized. The gray label is given and binarized in the latter part. The ternarization result g _p at a certain pixel p is obtained by using the luminance I of the captured image, the luminance threshold t for separating the white label and the black label, and the luminance range w of the gray label.

It is expressed as 1 represents a white label, 0 represents a gray label, and -1 represents a black label. As in the first embodiment, the threshold value t may be set to the same value as the threshold value used for binarization in normal spatial coding. The gray label luminance range w is determined based on the reliability of the pixel of interest p described later. By performing the ternarization, it is possible to accurately determine the label by the second label changing unit 306 in the subsequent stage without forcibly binarizing the ambiguous portion near the threshold. Furthermore, thanks to the use of the MMSW pattern, the contrast is relatively high and the reliability is high, so that the ratio of gray labels can be kept low. The ternarization result obtained in this way is output to the label changing unit 304.

  The reliability and the luminance range w of the gray label will be described. The reliability is a value that can be determined based on a label error rate, an SN ratio, a luminance level, and the like at the target pixel p. For example, in two images obtained by projecting and imaging a pattern consisting only of a bright part and a pattern consisting only of a dark part on the measurement target, the brightness of the image projected from the pattern consisting only of the dark part is higher It is considered that the pixel has a low SN ratio of the projection pattern and an error has occurred. Therefore, a window having an appropriate size with respect to the pixel of interest p is considered, the ratio of pixels in which errors occur in the pixels in the vicinity region is defined as a label error rate, and the difference between 1 and the label error rate is defined as the reliability. To do. The lower the reliability, the harder it is to decide whether the label should be a white label or a black label, so the number of pixels labeled on the gray label tends to be very large. Therefore, the brightness range w of the gray label is set such that the value of w increases as the reliability of the pixel of interest p increases, and the value of w decreases as the reliability decreases. By setting the threshold value in this way, the number of pixels labeled on the gray label can be kept within a certain range.

(Step S304)
Next, the label changing unit 304 that receives the ternary label optimizes and determines (changes) the label of each pixel based on the spatial neighborhood information to a plausible label. That is, the label changing unit 304 optimizes the ternary result input from the label assigning unit 303 as an initial value by using information that the label is likely to be the same in the spatial vicinity in the captured image, Change the label. The fringe pattern is a pattern having a certain stripe width, and generally, the imaging device 2 has a higher spatial resolution than the projection device 1. Corresponds to the imaging pixel. Therefore, when a neighboring pixel (such as eight neighbors) of the pixel of interest in the captured image is viewed, it is highly likely that the neighboring pixel has the same label as the pixel of interest. Utilizing this, for example, if the label of the target pixel is determined by taking the majority of each label in the neighboring pixels, a more likely label can be given to the target pixel. Here, a cost function C as described below is designed to obtain a label combination G that minimizes the cost.

  V represents a set of all pixels, and E represents a set of neighboring pixels corresponding to all pixels. Further, p is a target pixel, and q is a subscript indicating a neighboring pixel. The data cost D indicates which label is appropriate when viewed as the pixel of interest p alone, and the smooth cost W indicates which label is appropriate when the relationship between the label of the neighboring pixel q and the pixel of interest p is viewed. λ represents a weight, but the larger the λ, the greater the effect of the smooth cost W, resulting in a stronger smoothed result. In the design of the data cost D and the smooth cost W, only the label information is used in the above, but the luminance information of the captured image may be used. There are various methods for minimizing the cost function C, such as Iterated Conditional Mode, Graph Cut, Belief Propagation, and any of them may be used. In this way, labeling errors that occur due to the effect of noise are reduced. The obtained ternary optimization result is output to the edge detection unit 305.

(Step S35)
In step S <b> 35, the edge detection unit 305 detects a subpixel edge based on the determined ternary optimization result.

  The processing by the edge detection unit 305 is shown in FIG. Since it is considered that an edge exists near the gray label, first, the ternarization result is scanned in the horizontal direction of the image (the vertical direction of the image if the edge is horizontal), and the gray label and the black and white label in the vicinity thereof are scanned. An area is extracted and set as an edge candidate point. Then, a subpixel edge that passes near the center of the edge candidate point group is obtained by performing function fitting on the edge candidate point. The obtained subpixel edge is output to the second label changing unit 403.

(Step S36)
In step S36, the second label changing unit 306 detects the edge position based on the obtained subpixel edge, and changes the gray label to a binary black and white label.

  The processing of the second label changing unit 306 is shown in FIG. First, an edge position (boundary position) that is a pixel unit position through which a subpixel edge passes is obtained. Gray labels other than the edge position are determined on the left and right sides of the edge position (up and down if the edge is in the horizontal direction). For example, in the case of FIG. 17, the label is changed to a black label when it is on the left of the edge position, and a white label when it is on the right. Next, the gray label at the edge position is changed to a label having a larger area in the pixel on the left and right with the subpixel edge as a boundary. As in the case of the lower part of FIG. 17, when the subpixel edge is leftward in a pixel with an edge position, the area on the right side is large, and the white label is changed. In this way, the edge position and the binarization result are obtained. The edge position need not be limited to the pixel through which the subpixel edge passes, and may have a certain pixel width.

(Step S37)
In step S37, it is determined whether the projection, imaging, and processing of all pattern images have been completed. Steps S32 to S36 are repeated until it is determined that the processing is completed, and the correspondence between the projection coordinates and the image coordinates is obtained by spatial encoding. When the pattern to be processed still remains, the ternarization result of the obtained pattern and the subpixel edge are stored, and the next pattern is processed. When all the patterns have been processed, the multi-value quantization results and subpixel edge information of each pattern are integrated, spatial coding is performed, and the correspondence between the projected coordinates and the image coordinates is obtained. The obtained correspondence between the projected coordinates and the image coordinates is output to the three-dimensional coordinate calculation unit 307.

(Step S38)
In step S <b> 38, the three-dimensional coordinate calculation unit 307 calculates the three-dimensional coordinates from the correspondence between the projected coordinates and the image coordinates, and sends them to the output unit 308. Then, the output unit 308 outputs the three-dimensional coordinate data, and the process ends.

  Note that the process of step S32 need not be included in the repetition of the figure. After all patterns are projected and imaged in step S32, the processing from step S33 to step S36 may be repeated, or parallel processing may be performed.

  In this way, by using spatial neighborhood information using a pattern group that does not include a pattern with a narrow stripe width, the influence of noise generated in the captured image is reduced, and three-dimensional coordinates are calculated with high accuracy. be able to.

(Fourth embodiment)
The apparatus described in the fourth embodiment is a three-dimensional measuring apparatus, and the projected pattern has multiple levels of brightness levels. When performing labeling indicating the presence or absence of projection, the levels are higher than the projected brightness level. A label indicating that the area is ambiguous is assigned to the ambiguous area. The ambiguous label is corrected to a label that matches the projection brightness level in the subsequent processing based on spatial proximity information, thereby correcting the identification error of the measurement target area caused by noise and reducing the number of projected images. To calculate highly accurate three-dimensional coordinates.

[Device configuration]
Hereinafter, the configuration of the apparatus according to the present embodiment will be described. An information processing apparatus 4100 according to the fourth embodiment includes a projection control unit 401, an image input unit 402, a label applying unit 403, a label changing unit 404, an edge detecting unit 405, a second label changing unit 406, and three-dimensional coordinate calculation. Unit 407 and output unit 408. In the fourth embodiment, the configuration of FIG. 15 is partially changed. Therefore, here, the description will focus on the parts different from the third embodiment.

  The projection control unit 401 generates a pattern image having luminance levels (m = 4 in FIG. 19) in m stages (P1> P2>...> Pm) of P1, P2,. The projection control unit 1 is controlled to project the patterns one by one in order, and at the same time, a control signal is output to the image input unit 402. However, the projection and imaging of the patterns are in no particular order, and it is possible to project and image all the patterns simultaneously by separately projecting the wavelengths of the light of each pattern and using an imaging device corresponding to each wavelength. In the example of the pattern shown in FIG. 19, (1) to (4) are luminance calibration patterns for obtaining the observed luminance when each luminance level is projected onto the target object, and (5) to (7) are It is an encoding pattern for space division. By using a multi-value pattern, the identifiable area per sheet increases, so that the correspondence between the projection apparatus 1 and the imaging apparatus 2 can be obtained with a smaller number of projections. For example, in a normal light / dark pattern such as a gray code, the number of uniquely identifiable areas is doubled every time one new encoded pattern is projected. In the example shown in FIG. Therefore, the number of uniquely identifiable areas per new coding pattern is quadrupled.

  The label changing unit 404 changes the label using the fact that there is a high possibility that the label is in the spatial vicinity.

  The edge detection unit 405 obtains a subpixel edge from the multivalued result.

  The second label changing unit 406 determines the multi-value quantization result as the projection luminance level based on the subpixel edge.

  The label assigning unit 403 assigns L1, L2,... Lm labels corresponding to m-level luminance values of the captured image on which the pattern is projected, and undecided labels U that do not belong to them.

  As in the third embodiment, the label changing unit 404 optimizes the multi-value quantization result input from the label assigning unit 403 as an initial value, and determines a label. The obtained multi-value optimization result is output to the edge detection means 405.

  The edge detection unit 323 detects the subpixel edge from the multi-value optimization result input from the label change unit 404, as in the third embodiment. The obtained subpixel edge is output to the second label changing unit 406.

  The second label changing unit 406 changes the undecided label U to the label Li corresponding to the luminance level of the projection pattern in accordance with the subpixel edge input from the edge detecting unit 405, and performs spatial encoding. The processing of the second label changing unit 406 is similar if the gray label described in the third embodiment is replaced with the undetermined label U, and the white label and the black label are replaced with a label pair included in the set Φ described later. Since this is a process, the description is omitted. In this way, the edge position and the multivalued result are obtained.

  Other configurations are the same as those in the third embodiment, and are therefore omitted.

[Measurement processing]
Measurement processing by the three-dimensional measurement apparatus 4000 of the fourth embodiment will be described with reference to the flowchart of FIG. However, since many of them are the same as those in the third embodiment, the description thereof will be omitted, and only steps that are different will be described.

(Step S42)
In step S <b> 42, the projection control unit 401 causes the projection apparatus 1 to project a multi-value pattern as shown in FIG. 19 and sends a control signal to the image input unit 402. Upon receiving the control signal, the image input unit 402 causes the imaging device 2 to capture an image on which the pattern is projected, and sends the captured image to the label applying unit 403.

(Step S43)
In step S43, the label attaching unit 403 that has received the captured image provides a multi-value label to the captured image on which the pattern is projected. The label assigning unit 403 assigns L1, L2,... Lm labels corresponding to m-level luminance values of the captured image on which the pattern is projected, and undecided labels U that do not belong to them. By introducing such a label U, it is possible to determine the label more accurately by using spatial information at a later stage without forcibly determining the ambiguous and undecidable part at this stage. Therefore, it is possible to reduce labeling errors in the vicinity of edges that are difficult to label.

A method of attaching the multi-value label shown above will be described. First, the brightness calibration patterns shown in (1) to (4), which are the patterns in which all the projection pixels in the projection pattern shown in FIG. 19 have the brightness level Pi (i = 1, 2,..., M), In an image captured by being projected onto a target object, the luminance value observed at a certain pixel p is defined as I _p (Pi). Then, the encoding patterns (5) to (7) shown in FIG. 19 are sequentially projected and imaged. For an image in which each coding pattern is projected, the multi-value quantization result g _{p at the} pixel p is the luminance I _{p of} the captured image, and the luminance range regarded as the luminance label at each luminance level is w _p (Pi). Then

It is expressed as The luminance range w _p (Pi) may be determined based on the standard deviation in the luminance value I _p (Pi) of the camera. However, the present invention is not limited to this, and the value of w _p (Pi) may be given by a ratio such as 0.1 times I _p (Pi), or an appropriate fixed value may be used. The obtained multi-value quantization result is output to the label changing unit 404. Note that the undetermined label U is not necessarily limited to one label, and a multi-value label may be given depending on which luminance value is present.

(Step S44)
Here, a cost function C as described below is designed to obtain a label combination G that minimizes the cost (evaluates the cost).

  The description of each symbol is omitted for those that are the same as in the third embodiment, and only the newly defined set Φ and functions f and h will be described. The set Φ is a set having a pair of labels (Li, Lj) (a set of pixels) corresponding to luminance levels positioned adjacent to each other in the projection pattern as elements, and is defined for each projection pattern. For example, in (5) of FIG. 19, Φ = {(L1, L2), (L2, L3), (L3, L4)}, and in (6), Φ = {(L1, L2), (L2, L3). , (L3, L4), (L4, L1)}. The function f used in the data cost calculation is a function of a label corresponding to the luminance value, returns a subscript indicating the order of the luminance value, and the cost increases as the data cost D changes to a label with a significantly different luminance level. give. The function h used for the smooth cost calculation is the cost 0 if the two labels to be compared are the same, the cost 1 if the label is different and one of the labels is an undetermined label U, and the labels are different and neither is an undetermined label. If it is a label pair that can be placed together, the cost is calculated as 2. If the label is different and neither of them is an undetermined label, and it is a label pair that cannot be adjacent as a projected pattern, it is calculated as cost 3. However, the method of determining the cost of D or W is not limited to this, and the cost value and conditions vary depending on the nature of the projection pattern. The obtained multi-value optimization result is output to the edge detection unit 405.

(Step S45)
In step S45, the edge detection unit 405 detects a subpixel edge from the multilevel optimization result input from the label change unit 404, as in the third embodiment. In the third embodiment, the gray label and the neighboring black and white label are used as the edge candidate points. In the fourth embodiment, when the label U and the neighboring label are a label pair included in the set Φ, Is an edge candidate point. Then, a subpixel edge that passes near the center of the edge candidate point group is obtained by performing function fitting on the edge candidate point.

(Step S46)
In step S46, the second label changing unit 406 detects the edge position based on the obtained subpixel edge, and changes the undetermined label to a label corresponding to the luminance level of the projection pattern. The second label changing unit 406 changes the undecided label U to the label Li corresponding to the luminance level of the projection pattern in accordance with the subpixel edge input from the edge detecting unit 405, and performs spatial encoding. The process of the second label changing unit 406 is similar to the process described in the third embodiment by replacing the gray label with the undetermined label U and replacing the white label and the black label with the label pair included in the set Φ. Therefore, the description is omitted.

  In this way, by projecting a multi-value pattern consisting of multi-level luminance levels, labeling with undetermined labels, and using spatial neighborhood information, the effect of noise generated on the captured image can be reduced. The three-dimensional coordinates can be calculated with high accuracy with a reduced number of projections.

(Fifth embodiment)
In the first embodiment and the second embodiment, the method of calculating highly accurate three-dimensional coordinates by using the features of the space division pattern group has been described. In the third and fourth embodiments, a method of calculating highly accurate three-dimensional coordinates by giving a label indicating that it is ambiguous and using spatial neighboring information has been described. Therefore, in the fifth embodiment, these two methods are integrated, and the three-dimensional coordinates with higher accuracy are calculated by correcting and determining the identification error of the measurement target region that has occurred under the influence of noise.

[Device configuration]
An information processing apparatus 5100 according to the fifth embodiment includes a projection control unit 501, an image input unit 502, a label applying unit 503, a label changing unit 504, an edge detecting unit 505, a second label changing unit 506, and three-dimensional coordinate calculation. A unit 507 and an output unit 508.

  The information processing according to the fifth embodiment of the present invention will be described below with reference to the block diagram of FIG. Here, as a method in which the first embodiment and the third embodiment are integrated, a method using the features of the gray code pattern group and spatial proximity information will be described.

  The label assigning unit 503 assigns a ternary label according to the brightness value of the captured image.

  The label changing unit 504 may change the label using the pattern characteristics in the same manner as in the first and second embodiments, and then the label may be the same in the spatial vicinity as in the third and fourth embodiments. Change the label to take advantage of the high nature. Detailed processing will be described later.

  The edge detection unit 505 obtains a subpixel edge from the ternarization result.

  The second label changing unit 506 may change the label by looking at the labels of neighboring pixels as in the third embodiment, but the gray code pattern is changed to a sub-pixel edge by a label of a lower frequency pattern. Since the left and right labels (up and down if the edge is in the horizontal direction) are determined, the labels may be changed according to this characteristic. For example, as shown in FIG. 2, gray code pattern 1 is bright, and pattern 2 has a subpixel edge in pattern 3 in the dark area, left is bright and right is dark, while pattern 1 is bright, The subpixel edges of pattern 3 in the bright area of pattern 2 are determined such that the left side is dark and the right side is bright. When the pattern to be processed still remains, the binarization result of the obtained pattern and the subpixel edge are stored, and the edge position is output to the label changing unit 504.

  In this way, the ternary ambiguous label is processed in order from the low-frequency pattern of the Gray code, and a more probable label is given, and then the label is optimized, so that the first embodiment or A label with fewer errors than in the third embodiment can be obtained.

[Measurement processing]
The measurement process by the three-dimensional measurement apparatus 5000 of the fifth embodiment will be described with reference to the flowchart of FIG.

  Step S53, step S55, step S56, and step S57 are substantially the same as those in the third embodiment, and other steps are substantially the same as those in the first embodiment.

  Note that the processing in step S52 need not be included in the repetition of the figure. The processes from step S53 to step S57 may be repeated after all patterns are projected and imaged in step S52, or they may be processed in parallel. Moreover, it is not necessary to perform the process of step S53 and step S54 in the order shown to a figure, and you may replace an order suitably.

  As described above, by using the features of the gray code pattern group and the spatial neighborhood information, it is possible to reduce the influence of noise generated in the captured image and calculate the three-dimensional coordinates with higher accuracy.

(Modification)
In each of the first to fifth embodiments, an example in which three-dimensional measurement is performed has been described. However, even if calibration for projection mapping is performed based on the correspondence relationship between the obtained projection coordinates and imaging coordinates. Good. Normally, the imaging device 2 is not required when displaying content in projection mapping, but it is necessary to obtain a correspondence relationship to which region of the projection target the projection light is irradiated during calibration. Therefore, the imaging apparatus 2 may be used to capture an image of the pattern being projected based on any one of the first to fifth embodiments, and the mapping relationship of the projection light with respect to the projection target may be obtained.

(Other embodiments)
The present invention can also be realized by executing the following processing.

  That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

  This program and a computer-readable storage medium storing the program are included in the present invention.

Claims (9)

Image obtaining means for obtaining a plurality of images obtained by imaging a target object in which a plurality of types of patterns having a bright part and a dark part are projected in time series by the projecting means;
An attaching unit for attaching a label corresponding to the luminance of the pixel to the pixels in the plurality of images;
The given label is changed based on the restriction that the boundary does not occur in another pattern at a position where the boundary between the bright part and the dark part occurs in one of the plurality of types of projected patterns. Change means to
Correspondence relation deriving means for deriving a correspondence relation between the position of the pattern on the projection plane of the projection means and the position of the pattern in the acquired image based on the changed label. Information processing device.   The changing unit changes the label based on whether or not a change in a label of an adjacent pixel in each of the images applied by the applying unit exists in a change in luminance of the projected plurality of patterns. The information processing apparatus according to claim 1. In the changing unit, a label attached to a pixel in an image acquired from a low-frequency pattern among the plurality of projected patterns is applied to a pixel in an image acquired from a high-frequency pattern. The information processing apparatus according to claim 1, wherein the label assigned by the assigning unit is changed so as to be more likely than the label. Said applying means, the brightness of pixels in the image, and if it is larger than the threshold value, in a smaller, information according to any one of claims 1 to 3, characterized in that given different labels respectively Processing equipment. It said pattern, and an information processing apparatus according to any one of claims 1 to 4, characterized in that a gray code pattern. On the basis of the derived correspondence relationship information three-dimensional coordinate derivation means for deriving the three-dimensional coordinates of the target object surface, according to any one of claims 1 to 5, characterized by further comprising Processing equipment. The information processing apparatus according to claim 6 , wherein the three-dimensional coordinate deriving unit derives the three-dimensional coordinates of the surface of the target object using a position that is a boundary of the code as a measurement point. An image acquisition step of acquiring a plurality of images obtained by imaging a target object in which a plurality of types of patterns having a bright part and a dark part are projected in time series by a projecting unit;
An applying step of applying a label corresponding to the luminance of the pixel to the pixels in the plurality of images;
The given label is changed based on the restriction that the boundary does not occur in another pattern at a position where the boundary between the bright part and the dark part occurs in one of the plurality of types of projected patterns. Change process to
And a correspondence derivation step of deriving a correspondence between the position of the pattern on the projection plane of the projection unit and the position of the pattern in the acquired image based on the changed label. Information processing method . The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 7 .

Images