JP2005043233A - Three-dimensional shape measuring device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent reduction of the intensity difference between stripes, even when the number of gradations of the stripe is increased. <P>SOLUTION: A projection pattern is a combination of 7 kinds of luminance stripes formed by dividing 256 gradations into 6 steps, and arrangement is adopted so that each set of adjacent stripe does not have one-step level difference. The arrangement is equivalent to dividing the 256 gradations into 3 steps (4 levels of 0, 85, 170 and 255) on the particular level difference between each set of adjacent stripes. When arrangement is adopting by using 7-level stripes so that each set of adjacent stripe does not have one-step level difference, the number of one-set stripes can be about 28 in order that the same combination of adjacent stripes does not appear in one set, and the combination of the same adjacent stripes is not repeated in a range of at least 28 stripes, and since the intensity difference between the stripes is sufficiently large, the probability of a code retrieval error or the like is reduced. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a three-dimensional shape measurement technique for acquiring distance information to a target object using a pattern projection method.

  As a method for measuring the shape of an object, a method called a pattern projection method in which a reference pattern is projected onto the object and photographing is performed with a CCD camera or the like from a direction different from the direction in which the reference pattern light is projected. There is. The photographed pattern is deformed depending on the shape of the object. By associating the photographed deformation pattern with the projected pattern, the object can be measured three-dimensionally. The problem with the pattern projection method is how to make the correspondence between the deformed pattern and the projected pattern simple and easy to avoid. Various pattern projection methods have been proposed in the past.

  For example, the technique disclosed in Patent Document 1 includes a projector that projects an encoded pattern, a first camera that captures a projection pattern from the optical axis direction of the projector, and a projection pattern from a direction different from the optical axis direction of the projector. A second camera that shoots the image, and assigns a new code to an area where the change amount of the shooting pattern by the first camera relative to the projection pattern is equal to or greater than a predetermined value, and uses the assigned code to take a shooting pattern by the second camera Is a three-dimensional image capturing apparatus configured to generate first distance information from the first distance information and obtain a three-dimensional image based on the first distance information and the luminance information obtained from the first camera. A stripe pattern (sometimes called a slit pattern) is used as a projection pattern. By re-encoding the projection pattern using the pattern photographed by the first camera placed on the same optical axis, three-dimensional measurement can be performed with high accuracy.

In Patent Document 2, a plurality of mask patterns for determining the exposure amount of stripe light are prepared, and a coded pattern of a plurality of gradations is projected by sequentially switching the mask patterns during one frame exposure time of the camera.
JP 2000-65542 A JP 2002-131031 A

  However, when projecting a multi-level pattern as described above, the three-level pattern in the former example is one set of six stripes, and the latter example is a set of sixteen stripes with four masks. A repeated pattern will be projected. In this way, when the repetition cycle is small, there is a problem that the probability of erroneous matching during code search processing increases. In order to avoid this problem, in the former example, the number of levels may be increased, and in the latter example, the types of masks are increased within one frame of the camera.

  However, when the number of pattern levels is increased in this way, adjacent pattern areas are affected by signal deterioration factors such as the MTF (amplitude transfer function) of the projection system and the imaging system and the S / N of the CCD camera. A new problem arises that the possibility of detecting a new value increases. Also, when measuring an object with a dark surface color, the reflectance is small, so the dynamic range of the pattern is reduced on the image captured by the camera, the gradation step for each stripe is reduced, and an incorrect value is obtained in code processing. There are also problems such as calculating.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a three-dimensional shape measurement technique capable of accurate pattern detection in a three-dimensional shape measurement method using a plurality of levels of projection patterns.

  According to this invention, in order to achieve the above-mentioned object, the configuration as described in the claims is adopted. Here, prior to explaining the invention in detail, a supplementary explanation will be given of the claims.

  That is, according to the first aspect of the present invention, in order to achieve the above-described object, a stripe pattern coded with a plurality of levels of luminance obtained by dividing all white to all black substantially at equal intervals is used as a subject. In a three-dimensional shape measuring apparatus that obtains a three-dimensional image of the subject based on photographing the subject with a camera arranged so as to deviate from the optical axis of the projector that projects onto the projector, the brightness level difference between two adjacent stripes is A stripe pattern arranged so as to have two or more levels is used.

  According to the second aspect of the present invention, in order to achieve the above-mentioned object, a stripe pattern encoded with a plurality of levels of luminance obtained by dividing all white to all black at substantially equal intervals is used as a subject. In a three-dimensional shape measuring apparatus that obtains a three-dimensional image of the subject based on photographing the subject with a camera that is shifted from the optical axis of the projector that projects onto the projector, the color distribution is projected onto a uniform subject. The input data of the projector is determined on the basis of the gamma characteristic so that the stripe pattern has a plurality of levels of brightness obtained by dividing all white to all black at equal intervals. In this case, the input data of the projector may be determined based on the gamma characteristics of the camera, not the gamma characteristics of the projector. Alternatively, it may be determined based on both gamma characteristics.

  In the three-dimensional shape measuring apparatus according to the first aspect of the present invention, the projector is a data projector, and the stripe pattern projected on the subject having a uniform color distribution is substantially all white to all black. The input data of the data projector may be determined based on the gamma characteristics of the data projector so as to obtain a plurality of levels of brightness divided at intervals.

  In the three-dimensional shape measuring apparatus according to the first aspect of the present invention, the projector is a slide projector, and the stripe pattern projected onto the subject subject having a uniform color distribution substantially ranges from all white to all black. Pattern data of a medium on which the stripe pattern is drawn (for example, but not limited to, a metal film deposited on a slide film or a transparent substrate) so as to have a plurality of levels of brightness divided at equal intervals, You may make it determine based on the gamma characteristic of the said slide projector.

  Furthermore, in the above-described dimensional shape measuring apparatus, the stripe pattern projected onto the subject having a uniform color distribution has a plurality of levels obtained by dividing all white to all black on the photographed image by the camera at substantially equal intervals. The input data of the data projector or the pattern data of the slide film may be determined based on the gamma characteristic of the camera so that the brightness is obtained.

  The present invention can be realized not only as an apparatus or a system but also as a method. Of course, part of such an invention can be configured as software. Of course, software products used to cause a computer to execute such software are also included in the technical scope of the present invention.

  These and other aspects of the invention are set forth in the appended claims and will be described in detail below with reference to examples.

  According to the present invention, for example, the object is detected by a camera arranged so as to be offset from the optical axis of a projector that projects a stripe pattern encoded with a plurality of levels of brightness divided at equal intervals from all white to all black. Since a 3D shape measuring apparatus that obtains a 3D image of the subject based on shooting uses a pattern arranged so that the luminance level difference between two adjacent stripes is 2 levels or more, a code search error Can be reduced. In addition, by inputting a pattern in consideration of the gamma characteristics of the pattern projection system and the gamma characteristics of the imaging system, a desired stripe pattern can be obtained, so that code search errors can be reduced.

  An embodiment of a three-dimensional image measuring apparatus according to the present invention will be described with reference to the system schematic diagrams of FIGS.

  FIG. 1 is a configuration diagram of a three-dimensional image measurement apparatus according to an embodiment of the present invention. In FIG. 1, the three-dimensional image measurement apparatus is a pattern projection apparatus (for example, a liquid crystal projector) 10 that projects a pattern for three-dimensional measurement. , An imaging device (for example, a CCD camera, also referred to as a first camera) 20, a triangulation imaging device (for example, a CCD camera, also referred to as a second camera) 30, and other calculation processing device (not shown). Composed. Reference numeral 40 denotes a half mirror, and reference numeral 50 denotes an object. The basic configuration of this three-dimensional measuring apparatus is the same as the configuration disclosed in Patent Document 1 (Japanese Patent Laid-Open No. 2000-65542). The pattern projector 10 uses a liquid crystal projector, a DLP (trademark) projector, or a slide projector. The projection pattern input to the pattern projector 10, for example, a liquid crystal projector, uses a stripe pattern with shading as shown in FIG. 2, and projects the pattern onto an object (object) shown on the right side of FIG. When using a slide projector, the projection pattern is formed on a slide film, or a metal film or the like is vapor-deposited on a glass pattern, and the transmittance is controlled by the film thickness or the retinal point pattern.

  FIG. 3 shows a luminance profile in the horizontal direction of pattern data input to a liquid crystal projector, for example. The projection pattern (pattern data) is a combination of seven types of luminance stripes in which 256 gradations are divided into six levels, but the adjacent stripes are arranged so as not to have a one-level level difference. This is equivalent to dividing 256 gradations into three stages (four levels of 0, 85, 170, and 255) as far as the level difference between adjacent stripes is concerned. In the case of repeating normal four-level stripes into one set, the limit is to set one set to 12 stripes so that each combination of adjacent stripes does not appear in one set. Repeatedly, a stripe pattern including a large number of stripes is formed. If the same combination of adjacent stripes overlaps in the vicinity, it causes a code search error. On the other hand, if seven levels of stripes are used and the adjacent stripes do not have a level difference of one step as in this embodiment, the same combination of adjacent stripes appears in one set. In order to avoid this, the number of stripes in one set can be reduced to about 28. In the range of at least 28 stripes, combinations of the same adjacent stripes do not overlap, and the probability of code search errors also decreases.

  FIG. 4 shows a pattern projection pattern. An imaging device (first camera 20) and a projection device (pattern projection device 10) are arranged on the same optical axis by a half mirror 40 or the like, an imaging device is prepared for triangulation measurement, and a stripe pattern as shown in FIG. 2 is projected. To do. Re-encoding is performed from the image (first camera image) observed by the image sensor (first camera 20) of the same optical axis, and the image (second image) observed by the measurement image sensor (second camera 30). A three-dimensional distance image (distance) is calculated using the camera image.

  FIG. 5 shows a configuration example for calculating a distance image. In this figure, the pattern projection apparatus 10 projects a coded pattern onto the object 50. This pattern is stored in the frame memory 110. A projection pattern on the object 50 is imaged by the first camera 20 for monitoring and the second camera 30 for triangulation, and stored in the pattern image memories 120 and 150, respectively.

  The area dividing unit 130 divides the pattern image stored in the pattern image memory 120 into an area where the projection pattern (light) from the pattern projection apparatus 10 has sufficiently arrived (also referred to as area 2) and an area where it has not reached (also referred to as area 1). To divide. For example, for an area where the intensity difference between adjacent stripes is less than or equal to the threshold, it is determined that the projection pattern has not reached sufficiently, and the projection pattern has sufficiently reached an area where the intensity difference between stripes is greater than or equal to the threshold. Determined as an area. As described below, an edge pixel serving as a boundary line is calculated and a distance is calculated for an area where the projection pattern has sufficiently reached. For areas where the projection pattern does not reach sufficiently, distance calculation based on parallax is performed separately. Although not specifically described here, refer to Patent Document 1 for details.

  The recoding unit 160 extracts stripes for the extracted region 2, divides each stripe in the vertical direction for each stripe width, generates a square cell, and recodes the cell. This will be described in detail later.

  The code decoding unit 170 determines the code of each cell (edge) of the pattern image from the second camera 30 for triangulation stored in the pattern image memory 150 using the code from the recoding unit 160. . As a result, the x coordinate of the pixel at the measurement point p (edge) in the pattern image of the pattern image memory 150 and the irradiation direction (slit angle) θ from the light source are determined, and the distance Z is measured by equation (1) described later. (See FIG. 14). The three-dimensional image memory 180 stores the distance and the luminance value of the object acquired from the first camera 20 (stored in the luminance value memory 140) as three-dimensional image data.

  Details of the calculation of the three-dimensional shape in this configuration example will be further described.

  Re-encoding is performed according to the flowchart shown in FIG. 7 using the pattern image photographed by the first camera 20 for monitoring on the same optical axis and the pattern image used for light projection. First, the region of the pattern image photographed by the first camera 20 is divided. A region where the intensity difference between adjacent stripes is less than or equal to the threshold is extracted as region 1 where the projection pattern from the pattern projection apparatus 10 has not reached, and a region where the intensity difference between stripes is greater than or equal to the threshold is extracted as region 2. In step S10, the edge pixel that is the boundary line for the region 2 is calculated.

  Stripes are extracted from the extracted region 2 and each stripe is divided in the vertical direction for each stripe width to generate a square cell. The average value of the intensity is taken for each generated cell, and the average value is set as the intensity of each cell (S11). Compare the intensities between the corresponding cells in order from the center of the image, and if the pattern changes due to factors such as the reflectance of the object, the distance to the object, etc. Code generation and assignment are performed (S12 to S16).

  FIG. 8 is a simplified example for simplicity, but the left side stripe row of FIG. 8 is a light projection pattern coded by the arrangement of stripes. The intensity is 3 (strong), 2 (medium), 1 (weak) is assigned. The right side of FIG. 8 is obtained by extracting the stripes photographed by the first coaxial camera 20 in the direction perpendicular to the stripes by the cell width. In the example at the upper right in FIG. 8, since a new code appears with the intensity changing in the third cell from the left, a new code of 0 is assigned. In the example at the lower right in FIG. 8, since the existing code appears in the third cell from the left and the second cell from the top, a new code (232, 131) from the cell sequence (vertical) The code is expressed by (line, horizontal line). This re-encoding is equivalent to projecting a complex pattern such as a two-dimensional pattern at a site where the shape of the object is rich in change, and projecting a simple pattern at a site where there is little change. This process is repeated, and recoding is performed by assigning unique codes to all cells.

  As an example, FIGS. 11 and 12 respectively show simplified images obtained by the first camera 20 and the second camera 30 when the pattern of FIG. 10 is projected onto the object of FIG. In this example, FIG. 13 is obtained as a new coded pattern on the surface of the plate.

  Next, stripes are extracted from the stripe image obtained by the second camera 30 and divided into cells as before. For each cell, the code of each cell is detected using the recoded code, and the irradiation direction θ from the light source is calculated based on the detected code. The distance Z is calculated by Equation (1) using θ of the cell to which each pixel belongs, the x coordinate on the image taken by the camera 2, the focal length F and the base line length L which are camera parameters. FIG. 14 shows the relationship among the measurement point p, the irradiation direction θ from the light source, the x coordinate on the image taken by the second camera 30, the focal length F that is a camera parameter, and the baseline length L. .

  Z = FL / (x + Ftan θ) (1)

  This calculation is actually performed using the x-coordinate of the cell boundary, and the x-coordinate at this time is calculated in a unit smaller than the pixel resolution of the camera to improve the measurement accuracy. The x-coordinate value is obtained from the brightness average values d1 and d2 of appropriate pixels of the cells on both sides of the edge pixel calculated previously and the brightness de of the edge pixel. A pixel position de ′ (indicated by x for convenience in the figure) corresponding to the luminance de is obtained from a straight line obtained by linear interpolation from the pixel positions p1 and p2 adjacent to the edge pixel and the luminance average values d1 and d2, and this is the x coordinate value. . (See Figure 15)

  Here, the projection pattern is a combination of luminance stripes for each of the 42 gradations or 43 gradations in which 256 gradations are divided into 6 stages, but the adjacent stripes are arranged so as not to have a level difference of one stage. Therefore, the intensity difference between stripes is large compared to a pattern in which luminance stripes simply divided into six levels are combined, and it is easy to calculate the x coordinate value. FIG. 16 shows the luminance distribution of the pixel unit of the camera when the intensity difference between stripes is large and small. The average luminance value of several pixels on both sides of the edge pixel can take a range indicated by the thickness of the gray horizontal line due to the influence of noise of each pixel. As shown on the left of FIG. 16, when the intensity difference is large, when the calculation method of FIG. 15 is applied, the influence of noise on the luminance average value of several pixels on both sides of the edge pixel is expressed by the thickness of each gray line. As a result, the x coordinate value is determined within the range of a. However, when the intensity difference is small, the ratio of the noise component of the luminance average value of several pixels on both sides of the edge pixel becomes relatively large with respect to the intensity difference, which causes ambiguity in the calculation of the x coordinate value. The range becomes the range of b, and as a result, the measurement accuracy is lowered. For this reason, it is desirable that the difference in intensity between stripes is large.

  FIG. 6 shows a configuration example for obtaining the x coordinate. In FIG. 6, the edge right neighboring pixel position input unit 210, the edge right cell luminance average value input unit 220, the edge left neighboring pixel position input unit 230, the edge left cell luminance average value input unit 240, and the edge luminance input unit 250 respectively. d1, p1, d2, p2, and de are supplied to the interpolation calculation unit 200 to calculate the x coordinate as described above.

  In this embodiment, since the pattern is constrained so that the intensity difference between stripes is 2 levels or more, the x coordinate value can be measured with high accuracy.

  Next, correction of gamma characteristics of the projector and camera will be described.

  When a liquid crystal projector is used as the pattern projector (projector) 10, an output image projected on an input image of the liquid crystal projector generally has a gamma characteristic. Therefore, even if 256 gradations are divided into 6 equal parts in the input image, the brightness of the projected pattern is not evenly 7 levels. Therefore, the gamma characteristic must be taken into account when creating the input image. In the projector having the gamma characteristic of the output level on the vertical axis with respect to the input level on the horizontal axis as shown in FIG. 17, the input image shows luminance gradation values of 0, 148, 185, 203, 222, 237, and 255 from the graph. It is desirable to have a stripe pattern. The same applies to the case where the pattern projector (projector) 10 is a DLP projector. When the pattern projector (projector) 10 is a slide projector, it is necessary to prepare data in consideration of the gamma characteristics of the slide film when creating the film.

  Similarly, a gamma characteristic also exists in a CCD camera. Therefore, when it is desired to obtain a stripe pattern having a uniform luminance interval on a camera image, an input image that further considers the gamma characteristic of the CCD camera is required.

  The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the invention. For example, in the above embodiment, an example in which the level is equally divided into seven levels has been described, but it goes without saying that other levels may be set. Further, when 256 gradations are equally divided into 7 levels, 255 is not divisible by 6, so that it is included in the technical scope of the present invention to round the interval of each level to 42 gradations or 43 gradations.

  Further, it goes without saying that the present invention is effective for projection light having invisible wavelengths such as near infrared in addition to visible wavelengths.

  In the above example, the pattern is generated with the constraint that the intensity difference between stripes is 2 levels or more and the gamma characteristic is corrected. However, the intensity difference is not 2 levels or more. However, by correcting the gamma characteristic, as a result, it can be assured that the x-coordinate can be measured with relatively high accuracy. The present invention also includes a configuration that enables highly accurate measurement of the x-coordinate value only by correcting the gamma characteristic. In some cases, the intensity difference between stripes can be restricted to three levels or more.

It is a figure which shows the apparatus structure of the Example of this invention. It is a figure which shows an example of the pattern and subject for demonstrating the said Example. It is a figure which shows the example of a stripe pattern of the above-mentioned Example. It is a schematic diagram explaining operation | movement of the said Example. It is a block diagram explaining the structural example of the said Example. It is a figure explaining the circuit structural example which performs the interpolation calculation of the coordinate x on the image of the 2nd camera of the measurement point of the said structural example. It is a flowchart explaining operation | movement of the said Example. It is a figure explaining coding of the above-mentioned example. It is an arrangement diagram of a camera and a subject for explaining the coding of the embodiment. It is a pattern diagram for demonstrating the encoding of the said Example. It is a figure which shows the example of the monitor image of the 1st camera of the above-mentioned Example. It is a figure which shows the example of the monitor image of the 2nd camera 2 of the above-mentioned Example. It is a figure explaining the part to which the brightness | luminance changed in the to-be-photographed object in the above-mentioned Example. It is a figure explaining distance calculation in the above-mentioned example. It is calculation explanatory drawing of the edge coordinate of the above-mentioned Example. It is a figure explaining the difference of the edge coordinate calculation by the gradation difference in the said Example. It is a gamma characteristic and pattern input value calculation explanatory drawing of the light projector (projector) of the said Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Pattern projection apparatus 20 1st camera 30 for a monitor 2nd camera 40 for a triangulation 50 Half mirror 50 Object 110 Frame memory 120 Pattern image memory 130 Area division part 140 Luminance value memory 150 Pattern image memory 160 Recoding part 170 Code decoding unit 180 Three-dimensional image memory 200 Interpolation calculation unit 210 Edge right neighboring pixel position input unit 220 Edge right cell luminance average value input unit 230 Edge left neighboring pixel position input unit 240 Edge left cell luminance average value input unit 250 Edge luminance input Part

Claims (8)

  Taking a picture of the subject with a camera arranged offset from the optical axis of the projector that projects a stripe pattern encoded with a plurality of levels of luminance, which is divided from substantially white to all black at substantially equal intervals. On the basis of the three-dimensional shape measuring apparatus for obtaining a three-dimensional image of the subject, a three-dimensional shape using a stripe pattern arranged so that a difference in luminance level between two adjacent stripes is 2 levels or more is used. measuring device.   Taking a picture of the subject with a camera arranged offset from the optical axis of the projector that projects a stripe pattern encoded with a plurality of levels of luminance, which is divided from substantially white to all black at substantially equal intervals. Based on the three-dimensional shape measuring apparatus for obtaining a three-dimensional image of the subject, the stripe pattern projected onto the subject having a uniform color distribution has a plurality of levels in which all white to all black are substantially equally spaced. The three-dimensional shape measuring apparatus is characterized in that input data of the projector is determined based on a gamma characteristic of the projector so as to obtain a brightness of.   The projector is a data projector, and the data projector is configured such that a stripe pattern projected onto a subject having a uniform color distribution has a plurality of levels of brightness obtained by dividing all white to all black at substantially equal intervals. The three-dimensional shape measuring apparatus according to claim 1, wherein the input data is determined based on gamma characteristics of the data projector.   The projector is a slide projector, and the stripe pattern is drawn so that the stripe pattern projected onto the subject having a uniform color distribution has a plurality of levels of brightness divided from all white to all black at equal intervals. The three-dimensional shape measuring apparatus according to claim 1, wherein pattern data of a medium to be processed is determined based on gamma characteristics of the slide projector.   The input data of the data projector so that the stripe pattern projected onto the subject having a uniform color distribution has a plurality of levels of brightness obtained by dividing all white to all black at equal intervals on the image taken by the camera. 5. The three-dimensional shape measuring apparatus according to claim 3, wherein pattern data of the medium is determined based on gamma characteristics of the camera.   Taking a picture of the subject with a camera arranged offset from the optical axis of the projector that projects a stripe pattern encoded with a plurality of levels of luminance, which is divided from substantially white to all black at substantially equal intervals. Based on the three-dimensional shape measuring apparatus for obtaining a three-dimensional image of the subject, the stripe pattern projected onto the subject having a uniform color distribution has a plurality of levels in which all white to all black are substantially equally spaced. The three-dimensional shape measuring apparatus is characterized in that the input data of the projector is determined based on the gamma characteristic of the camera so that the brightness of   A three-dimensional shape that obtains a three-dimensional image of the subject based on photographing the subject with a camera that is arranged off the optical axis of a projector that projects a stripe pattern encoded with a plurality of levels of light intensity onto the subject. The measuring apparatus uses a stripe pattern arranged so that a difference in light intensity level between two adjacent stripes in the stripe pattern is N (N is a positive number of 2 or more). Dimensional shape measuring device.   A three-dimensional shape that obtains a three-dimensional image of the subject based on photographing the subject with a camera that is arranged off the optical axis of a projector that projects a stripe pattern encoded with a plurality of levels of light intensity onto the subject. In the measurement method, a three-dimensional shape using a stripe pattern arranged so that a difference in light intensity level between two adjacent stripes in the pattern is N (N is a positive number of 2 or more) level or more. Measuring method.

Images