JP2016217833A - Image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system that can highly accurately restore a three-dimensional shape even when a measurement front surface of a measured object is highly glossy and the like.SOLUTION: An image processing device 4 takes in a projection pattern image projected onto a measured object from a projector 2, and a shooting pattern image shot by a camera 3; calculates a corresponding point between the projection pattern image and the shooting pattern image; and restores a three-dimensional shape of the measured object from the calculated corresponding point. The image processing device 4 comprises: a correlation calculation unit 42 that calculates a matching degree of the projection pattern image and the obtained shooting pattern image by a correlation from the taken-in projection pattern image and shooting pattern image, in which the matching degree corresponds to a certain area of a front surface of the measured object; and a first order reflection light acquisition unit 43 that discriminates and acquires a shooting pattern image by first order reflection light from the front surface of the measured object on the basis of the correlation degree calculated by the correlation calculation unit 42.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing system and an image processing method.

  For example, in the FA (Factory Automation) field of various assembly lines in factories, many active stereo type measuring devices capable of non-contact measurement are commercially available for parts inspection and recognition. As such a non-contact measurement type, for example, a laser probe, a sensing method using a line laser, and the like have been conventionally known, but in recent years, a so-called projector / camera that can be measured in an area (plane). The method is drawing attention.

  Such a projector-camera method for three-dimensional measurement is already known (for example, Patent Document 1), and a pattern image is projected from a projector (projection apparatus) onto a measured object and projected onto the measured object. The obtained pattern image is photographed with a camera (photographing device), corresponding points are acquired from the captured image, the three-dimensional shape is restored from the acquired corresponding points, and the object to be measured is three-dimensionally measured.

  For example, in Patent Document 1, a pattern image is projected from a projection optical unit (projector) onto an object to be measured, the pattern image projected onto the object to be measured is captured by an image sensor (camera), and the phase is shifted. A projector / camera type three-dimensional shape measuring apparatus configured to obtain height information of an object to be measured using phase information calculated from a plurality of pattern images acquired in the above state is disclosed.

  According to the technique described in Patent Document 1, it is described that measurement can be performed with high accuracy even when the measurement surface of the object to be measured is in a wide range from a low scattering portion to a high scattering portion. However, for example, when the measurement surface of the object to be measured has a high gloss and concave curved shape, the pattern image (pattern light) projected from the projection optical unit (projector) is mutually reflected by this measurement surface. As a result, there is a problem that the projected pattern image and the captured image cannot be correctly associated by superimposing the projected pattern image. For this reason, the three-dimensional shape cannot be accurately restored, and the three-dimensional measurement cannot be performed correctly.

  Therefore, an object of the present invention is to provide an image processing system capable of accurately restoring a three-dimensional shape even when the measurement surface of the object to be measured has high gloss or the like.

  In order to achieve the above object, an image processing system according to the present invention shoots the pattern image projected onto the measurement object by projecting means for projecting a predetermined pattern image onto the surface of the measurement object. An image capturing means for acquiring a pattern image, a projection pattern image projected onto the object to be measured, and the image capturing pattern image are captured, a corresponding point between the projection pattern image and the image capturing pattern image is calculated, and from the calculated corresponding point An image processing system comprising: an image processing unit that restores a three-dimensional shape of the object to be measured, wherein the image processing unit uses a surface of the object to be measured from the captured projection pattern image and the captured pattern image A correlation calculation unit that calculates the degree of matching between the projection pattern image and the obtained captured pattern image corresponding to a certain area by correlation, and the correlation calculation unit calculates Based on the correlation degree that, the is characterized in that it has a first order reflection light acquisition unit that acquires by discriminating the captured pattern image by the primary reflected light from the surface of the measurement subject, a.

  The image processing system according to the present invention correlates the degree of matching between the projection pattern image and the obtained shooting pattern image corresponding to a certain area on the surface of the object to be measured from the projection pattern image and the shooting pattern image. Based on the calculated degree of correlation, it is possible to discriminate and obtain a captured pattern image based on the primary reflected light from the surface of the measured object.

  Therefore, for example, when the surface of the object to be measured is highly glossy and concavely curved, the projection pattern image and the obtained captured pattern image are correctly associated with each other by mutual reflection on the surface of the object to be measured. It is possible to suppress the inconvenience that cannot be performed, discriminate and obtain a captured pattern image based only on the primary reflected light from the surface of the object to be measured, and restore the three-dimensional shape with high accuracy.

1 is a schematic diagram showing an appearance of an image processing system according to an embodiment. 1 is a block diagram showing a schematic configuration of an image processing system according to an embodiment. Schematic which shows the structure of the image projection unit of a projector. The figure which shows an example of the pattern image projected on a to-be-measured object. The perspective view which shows an example of a to-be-measured object. The figure which shows an example of the imaging | photography pattern image image | photographed with the camera with respect to the measurement surface where the pattern image was projected. 6 is a flowchart illustrating image processing by the image processing system according to the embodiment. The schematic diagram for demonstrating the mutual reflection generate | occur | produced on the measurement surface of a to-be-measured object. The schematic diagram for demonstrating the state from which the correspondence between a projection pattern and an imaging | photography pattern image is not calculated | required correctly. The figure which showed an example of the projection pattern of M series corresponding to a certain area | region, and each acquired imaging pattern image. The figure which showed the result of having calculated the height of each correlation of each obtained imaging pattern image. The schematic diagram for demonstrating the matching process in the modification 1. FIG. The schematic diagram for demonstrating the matching process in the modification 2. FIG. (A), (b), (c) is a figure which shows another example of the pattern image each projected on a to-be-measured object.

  Hereinafter, the present invention will be described based on the illustrated embodiments. FIG. 1 is a schematic diagram showing an external appearance of an image processing system according to the present embodiment, and FIG. 2 is a block diagram showing a schematic configuration of the image processing system.

  As shown in FIG. 1, an image processing system 1 according to the present embodiment includes a projector 2 as a projection unit, a camera 3 as a photographing unit, and an image processing device 4 as an image processing unit. The image processing device 4, the projector 2, and the camera 3 are connected by wire.

  As shown in FIG. 2, the projector 2 includes an image signal input unit 11, an image signal processing unit 12, a device drive control unit 13, and an image projection unit 14 as main components, and a projection image is measured. Project to the object 5. The projector 2 is disposed substantially in front of the measured object 5.

  The image signal input unit 11 inputs an image signal output from the image signal output unit 40 of the image processing device 4. The image signal processing unit 12 performs predetermined image processing (for example, processing such as contrast, brightness, saturation, hue, RGB gain, and sharpness) on the image signal to generate a projection image signal. The device drive control unit 13 performs drive control of devices (light source 20, color wheel 21, MDM element 25 (see FIG. 3), etc.) in the image projection unit 14 in accordance with the generated projection image signal.

  As shown in FIG. 3, the image projection unit 14 converts the white light emitted from the light source 20 into light of each color of RGB by a rotating color wheel 21, and relay lens system 22, plane mirror 23, concave mirror 24, etc. Then, the light is incident on a well-known DMD (Digatal Micromirror Device) element 25 as an image forming element. Then, the DMD element 25 drives and controls a plurality of micromirrors based on image data for each display pixel in synchronization with the incident timing of light of each color (RGB), and generates a projection image corresponding to the projection image signal. Projection is performed on the measurement object 5 through the projection optical system 26. In the present embodiment, the projection image projected onto the measurement object 5 is, for example, a pattern image such as a checker pattern (checkered pattern) as shown in FIG.

  As shown in FIG. 2, the camera 3 includes a photographic lens 30, a solid-state imaging device 31 such as a CCD, and a signal processing unit 32 as main components, and the measured object 5 incident through the photographic lens 30. A subject image is formed on the solid-state image sensor 31. Then, the signal processing unit 32 takes in an electrical signal (pixel output signal) output from the solid-state imaging device 31, performs predetermined signal processing (for example, processing such as shoeing correction, Bayer conversion, and color correction) for display. Or an image signal that can be recorded. The camera 3 is arranged toward the measured object 5 with a predetermined distance from the projector 2 on the same plane as the projector 2.

  In the present embodiment, the object to be measured 5 is, for example, a metal part having a measurement surface 5a curved in a concave shape as shown in FIG. 5 and the surface of which is subjected to high gloss processing (mirror surface processing). 5, a pattern image (for example, a checker pattern) is projected from the projector 2 on the measurement surface 5a side. The object to be measured 5 is not limited to a metal part having a concavely curved measurement surface 5a as shown in FIG. 5 and the surface of which has been subjected to high gloss processing (mirror processing). Other surface shapes or other materials may be used as long as they have been glossed.

  The image processing apparatus 4 includes an image signal input unit 41 that inputs an image signal from the signal processing unit 32 of the camera 3, a correlation calculation unit 42, a primary reflected light acquisition unit 43, a three-dimensional shape restoration processing unit 44, The display unit 45 (see FIG. 1), a control unit 46 that controls the entire processing of the image processing apparatus 4, and the image signal output unit 40 that outputs an image signal corresponding to the pattern image to the projector 2 side are main components. As prepared. The image processing device 4 is configured by a personal computer, for example.

  The correlation calculation unit 42 receives the projection pattern image captured from the image signal output unit 40 and the captured pattern image captured from the camera 3 via the image signal input unit 41 (for example, the captured image as shown in FIG. 6). The degree of matching between the obtained captured pattern image and the projected pattern image corresponding to a certain area on the surface of the measurement object 5 is calculated by correlation (details of the processing operation of the correlation calculation unit 42 will be described later).

  The primary reflected light acquisition unit 43 discriminates and acquires a captured pattern image based on primary reflected light from the surface of the measurement object 5 based on the degree of correlation calculated by the correlation calculation unit 42 (acquires primary reflected light). Details of the processing operation of the unit 43 will be described later). The three-dimensional shape restoration processing unit 44 calculates corresponding points between the two-dimensional captured pattern image obtained by the primary reflected light acquired by the primary reflected light acquiring unit 43 and the projection pattern image, and measures from the calculated corresponding points. The three-dimensional shape of the surface 5a is restored.

  Strictly speaking, the component of the primary reflected light from the measurement surface 5a photographed by the camera 3 includes a reflected light component equal to or higher than the secondary reflected light from the measurement surface 5a, a component of multiple reflected light from the measurement surface 5a, and the like. Although the component of a scattered light etc. is contained slightly, in this embodiment, it defines only as "primary reflected light".

  Next, image processing by the image processing system 1 according to the present embodiment will be described with reference to a flowchart shown in FIG.

  First, an image signal is output from the image signal output unit 40 of the image processing apparatus 4 to the image signal input unit 11 of the projector 2, and a pattern image (for example, a checker pattern image as shown in FIG. 4) is received from the projector 2. Projection is performed on the measurement surface 5a of the measurement object 5 (step S1).

  Then, the measurement surface 5a on which the pattern image is projected is photographed by the camera 3 (step S2). This captured image is, for example, a two-dimensional image in which a pattern image (checker pattern image) is projected on the measurement surface 5a as shown in FIG. The captured image is stored in a storage unit (not shown) in the image processing device 4. Shooting by the camera 3 is performed by, for example, inputting a shooting ON signal from the image processing device 4 to the camera 3 or turning on the shooting button of the camera 3 by the measurer.

  Then, for example, when a photographed image as shown in FIG. 6 (a photographed image when a pattern image such as a checker pattern is projected from the projector 2 onto the measurement surface 5a) is photographed by the camera 3, this projection is performed. A pattern (hereinafter referred to as “projection pattern”) is associated with an image (captured pattern image) captured by the camera 3 (step S3). When the projector 2 includes the DMD element 25 as an image forming element as in the present embodiment, the projection pattern and the captured pattern image are associated with each other by a known matching method based on the encoding of the DMD element 25 for each pixel. It can be carried out. The captured pattern image can be displayed on the display unit 45 of the image processing device 4.

  By the way, in this embodiment, since the measurement surface 5a of the measured object 5 is curved in a concave shape and the surface thereof is subjected to high gloss processing (mirror processing), for example, as shown in FIG. When a pattern image (pattern light) L is projected from the projector 2, secondary reflected light L2 and the like are generated by mutual reflection in addition to the primary reflected light L1 on the measurement surface 5a. In FIG. 8, only the secondary reflected light L2 is shown as the disturbance light that adversely affects the processing in the primary reflected light acquisition unit 43 and the like later, but other than this, the third order or higher on the measurement surface 5a. Reflected light, multiple reflected light, reflected light of light incident from an external light source, and the like are also included as disturbance light.

  In this way, when a pattern image is projected onto the measurement surface 5a that is concavely curved and whose surface is subjected to high gloss processing (mirror surface processing), a captured pattern image captured by the camera 3 by the mutual reflection shown in FIG. A part of the pattern image is superimposed on an arbitrary area of the image, and the correspondence between the projection pattern and the captured pattern image cannot be obtained correctly.

  For example, when an arbitrary area A is selected from the captured pattern image shown in FIG. 6, and a projection pattern P as shown in FIG. For example, two patterns a and b are generated by mutual reflection at 5a, and the two patterns a and b are superimposed on the area A, and for example, a pattern c is obtained as a captured pattern image. Therefore, the correspondence between the projection pattern a (projection pattern image) and the pattern d (photographing pattern image) cannot be obtained correctly.

  Therefore, in the present embodiment, adjustment is made so that the movement or individual difference of the DMD element 25 of the projector 2 matches the projection pattern so that the coding pattern obtained by a well-known matching method has M-sequence characteristics. The degree of matching between the projection pattern and the captured pattern image in an arbitrary region of the captured image is calculated as a correlation (step S4). The process of step S4 is executed by the correlation calculation unit 42.

  In order to match the movement of the DMD element 25 of the projector 2 and individual differences with the projection pattern, a pattern image (for example, a checker pattern image as shown in FIG. 6) is projected in advance on the white plane. Calibration for encoding the projected image is performed, and adjustment is performed so that a sequence close to the M sequence appears.

  When the correspondence between the projection pattern and the photographic pattern image is not correctly obtained due to mutual reflection or the like on the measurement surface 5a (see FIG. 9), in the process of step S4, for example, the photographic pattern image shown in FIG. When an arbitrary area A is selected, and an M-sequence projection pattern M as shown in FIG. 10 is projected corresponding to the area A, for example, by calculating the matching degree by correlation, for example, the pattern Assume that a captured pattern image having a pattern such as A, B, and C is obtained.

  FIG. 11 is a diagram showing the results of calculating the respective correlation heights of the patterns A, B, and C described above. In FIG. 11, the horizontal axis represents the phase of each pattern A, B, and C, and the vertical axis represents the correlation height of each pattern A, B, and C.

  Since the M series is a pattern showing a high correlation, the pattern A is calculated as the pattern having the highest correlation (high correlation pattern) from the patterns A, B, and C shown in FIG. 11 (step S5). Since the pattern A having the highest correlation is the pattern having the highest matching rate, the captured pattern image by only the primary reflected light from the measurement surface 5a is discriminated and acquired while suppressing the influence of the mutual reflection component or the like (step S6). ). As a result, the projection pattern on the measurement surface 5a and the acquired captured pattern image are correctly associated with each other. The process of step S6 is executed by the primary reflected light acquisition unit 43.

  Then, corresponding points (coordinates) between the projection pattern (projection pattern image) on the measurement surface 5a and the acquired captured pattern image are calculated (step S7), and the three-dimensional shape of the measurement surface 5a is calculated from the calculated corresponding points (coordinates). Restoration is performed (step S8). The restored three-dimensional shape of the measurement surface 5 a of the measured object 5 can be displayed on the display unit 45 of the image processing device 4.

  Thus, according to the present embodiment, the degree of matching between the projection pattern image and the obtained shooting pattern image corresponding to a certain area of the measurement surface 5a of the measurement target object 5 from the projection pattern image and the shooting pattern image. Can be discriminated and acquired based on the calculated degree of correlation, and the captured pattern image by the primary reflected light from the measurement surface 5a of the measured object 5 can be discriminated.

  Therefore, when the measurement surface 5a of the measurement object 5 is highly glossy and concavely curved as in this embodiment, the pattern projected by mutual reflection or the like on the measurement surface 5a of the measurement object 5 It is possible to accurately restore the three-dimensional shape of the measurement surface 5a by suppressing problems that prevent the image and the captured pattern image from being correctly associated.

<Modification 1>
In the above embodiment, as shown in FIG. 10 and FIG. 11, the matching degree is calculated by correlation to perform matching with high correlation. However, in order to further improve matching accuracy, for example, FIG. As shown in the first modification example, the region when the matching degree is calculated by correlation using the epipolar geometry for the captured pattern image (for example, three regions M1, M2, and M3 on the epipolar line E in FIG. 12). M3) may be limited, and a region having the highest correlation among the limited regions may be calculated.

  The epipolar line E as shown in FIG. 12 can be calculated from the basic matrix from the arrangement positions of the projector 2 and the camera 3 of the geometrically calibrated image processing system 1.

  In this way, by using the epipolar geometry for the captured pattern image and limiting the area when calculating the degree of matching by correlation, the accuracy of matching is further improved, and further, matching calculation in unnecessary areas can be performed. Since it becomes unnecessary, the calculation efficiency can be improved.

<Modification 2>
In Modification 1, epipolar geometry is used for the captured pattern image to limit the area for calculating the degree of matching by correlation. For example, as shown in FIG. A region M4 (hereinafter referred to as “error region M4”) in which the error position is highly correlated due to the influence of light or the like, and a matching correct region M5 in which the correlation of the matching degree is slightly lower than the error region M4 are calculated. It may be done. In this case, if the degree of matching is determined based on the level of correlation, a matching error may occur using the error region M4 instead of the correct solution region M5. When the error area M4 is adopted, the restoration accuracy of the three-dimensional shape is lowered.

  Therefore, in the second modification, in order to prevent such a matching error, a projection pattern (projection pattern image) on the measurement surface 5a, and each captured pattern image acquired corresponding to the error region M4 and the correct region M5 are used. Corresponding points (coordinates) are calculated, and the three-dimensional shape of the measurement surface 5a is restored from the calculated corresponding points (coordinates).

  Next, in order to determine which of the error area M4 and the correct answer area M5 is the correct area, the control unit 46 obtains two-dimensional coordinates obtained by perspective projection conversion from the three-dimensional point position of the restored three-dimensional shape. An error between the coordinates corresponding to the three-dimensional position in the system and the pattern coordinates of the projection pattern (projection pattern image) on the measurement surface 5a is determined. In this error determination, the smaller error (that is, the correct answer area M1) is determined as a normal position.

  In this way, even when the error region M4 is calculated on the epipolar line E due to the influence of disturbance light or the like but the matching degree shows a high correlation, the correct region M5 is selected without adopting the error region M4. Since it can be employed, only the primary reflected light can be discriminated and the three-dimensional shape can be accurately restored.

  In the above-described embodiment, the checker pattern image is used as the pattern image projected onto the measurement surface 5a. However, the present invention is not limited to this. For example, FIGS. 14 (a), 14 (b), and 14 (c). As shown in FIG. 6, the entire pattern may be a dark gray lattice pattern image, a dark gray dot pattern image, a gradation image, or the like. The gradation image shown in FIG. 14C is a dark tone to high tone gradation from darker to darker from the top to the bottom.

  Further, in the above-described embodiment, the projector 2 and the camera 3 are separately arranged with a predetermined interval. However, the camera 3 is integrally arranged on the outer surface (for example, the upper surface) of the projector 2. But you can. Furthermore, a configuration in which the image processing apparatus 4, the projector 2, and the camera 3 are connected via a network constructed by a wireless communication network may be employed.

DESCRIPTION OF SYMBOLS 1 Image processing system 2 Projector 3 Camera 4 Image processing apparatus 5 Object to be measured 5a Measurement surface 14 Image projection unit 25 DMD element 30 Shooting lens 31 Solid-state image sensor 32 Signal processing part 42 Correlation calculation part 43 Primary reflected light acquisition part 44 3 Dimensional shape restoration processing section E Epipolar line

Japanese Patent No. 4077754

Claims (7)

Projecting means for projecting a predetermined pattern image on the surface of the object to be measured, photographing means for photographing the pattern image projected on the object to be measured and obtaining a photographed pattern image, and projecting on the object to be measured Image processing means for capturing a projection pattern image and the captured pattern image, calculating corresponding points between the projected pattern image and the captured pattern image, and restoring the three-dimensional shape of the measured object from the calculated corresponding points; In an image processing system comprising:
The image processing means correlates a degree of matching between the projection pattern image obtained and the captured pattern image corresponding to a certain area on the surface of the object to be measured from the captured projection pattern image and the captured pattern image. A correlation calculation unit to calculate;
A primary reflected light acquisition unit that discriminates and acquires a captured pattern image based on primary reflected light from the surface of the measurement object based on the degree of correlation calculated by the correlation calculation unit. A featured image processing system.   When a projection pattern image is projected from the projection unit onto the measurement object and a plurality of captured pattern images are obtained with respect to the projection pattern image, the correlation calculation unit corresponds to the pattern having the highest correlation. The image processing system according to claim 1, wherein the captured pattern image is calculated as a pattern having the highest matching degree.   The primary reflected light acquisition unit discriminates and acquires a captured pattern image of a pattern having the highest matching degree calculated by the correlation calculating unit as a captured pattern image by primary reflected light from the surface of the measurement object. The image processing system according to claim 2.   The correlation calculation unit uses geometric calculation on the obtained captured pattern image to limit the number of matching areas to a predetermined number, and has the highest correlation among the limited areas. 4. The image processing system according to claim 1, wherein the region is calculated as a captured pattern image having the highest degree of matching. 5.   After calculating a corresponding point between the projection pattern image and the captured pattern image calculated by the correlation calculation unit, and restoring the three-dimensional shape of the measured object from the calculated corresponding point, the three-dimensional point of the three-dimensional shape The error between the coordinates corresponding to the three-dimensional position in the two-dimensional coordinate system obtained by perspective projection conversion from the position and the pattern coordinates of the projection pattern image is determined, and the calculated three-dimensional if the error is within a predetermined range The image processing system according to claim 1, wherein it is determined that the shape restoration accuracy is high.   The image processing system according to claim 1, wherein a measurement surface of the object to be measured has high gloss. Projecting means for projecting a predetermined pattern image on the surface of the object to be measured, photographing means for photographing the pattern image projected on the object to be measured and obtaining a photographed pattern image, and projecting on the object to be measured Image processing means for capturing a projection pattern image and the captured pattern image, calculating corresponding points between the projected pattern image and the captured pattern image, and restoring the three-dimensional shape of the measured object from the calculated corresponding points; In an image processing method of an image processing system comprising:
A first step of calculating a degree of matching between a projection pattern image and an obtained shooting pattern image corresponding to a certain area on the surface of the object to be measured from a correlation between the captured projection pattern image and the shooting pattern image; ,
And a second step of discriminating and obtaining a captured pattern image by primary reflected light from the surface of the measurement object based on the degree of correlation calculated in the first step. Processing method.