WO2011145285A1

WO2011145285A1 - Image processing device, image processing method and program

Info

Publication number: WO2011145285A1
Application number: PCT/JP2011/002561
Authority: WO
Inventors: 洋川崎; 亮古川; 立昌佐川; 八木　康史
Original assignee: 有限会社テクノドリーム二十一
Priority date: 2010-05-17
Filing date: 2011-05-09
Publication date: 2011-11-24
Also published as: JP2011242183A

Abstract

Provided are an image processing device, an image processing method, and a program, which enable measurement of the shape of a moving object at a high density and a high frame rate. A two-dimensional image of an object (30) present in a three-dimensional space is captured by an image processing device (10), a camera (28), a projector (24), and a projector (26), and the three-dimensional shape of the object is restored from the captured two-dimensional image. The projector (24) projects a horizontal pattern onto the object (30), and the projector (26) projects a vertical pattern onto the object (30). The two-dimensional image is acquired by capturing an image of pattern light produced by the reflection of the patterns by the object (30) by the camera (28), and the three-dimensional image is restored from the two-dimensional image by the image processing device (10).

Description

Image processing apparatus, image processing method, and program

The present invention relates to an image processing device, an image processing method, and a program, and more particularly, to an image processing device, an image processing method, and a program for restoring a three-dimensional shape of an object displayed in an input two-dimensional image.

There is a need for three-dimensional measurement of a target object having high density and high frame rate and dynamic motion. For example, it is desired to restore the shape at a high density and a high frame rate for a change in muscles of an exercising person and a structural analysis at the moment when an object bursts. *

In such a practical three-dimensional measurement, the active measurement method has been considered as a main solution. Particularly for dynamically changing objects, a method using a projector capable of acquiring a wide range in a short time is suitable. However, there is an essential problem in using the active method by the projector for high-density shape measurement of the moving object. *

Two main methods of three-dimensional measurement using a projector are known: a time encoding method and a spatial encoding method. Since the time encoding method is a method of projecting a plurality of patterns, it is essentially not suitable for measuring a moving object at a high frame rate. Also, when using a plurality of projectors to cover a wide range, it is difficult to achieve synchronization. On the other hand, in the case of the spatial encoding method, the projection pattern of each projector is invariant and it is suitable for measuring a moving object because only one shot image is required, but the pattern itself is complicated, so Problems such as color and shape interference occur, and image processing is often difficult. *

If the shape can be restored from an image of one frame on which a fixed monochromatic line pattern is projected, all of the above problems can be solved. As such a method, a one-shot restoration method (shape from ground pattern: SFG) based on a coplanarity constraint (COP) using a grid pattern composed of vertical and horizontal straight lines has been recently proposed. (Non-patent document 1), (Non-patent document 2), (Non-patent document 3). *

In practical three-dimensional measurement, an active method of projecting light from a sensor has been frequently used. In particular, many methods using a video projector have been proposed for efficiency (Non-Patent Document 4) and (Non-Patent Document 5). Regarding the active method using a projector, two types of methods, a time encoding method and a spatial encoding method, have been proposed. *

In recent years, research on shape restoration of dynamic scenes using high-speed cameras and DLP projectors has been conducted. Weise et al. Proposed a system combining pattern projection and stereo vision based on the phase shift method (Non-patent Document 6). Narasiman et al. Proposed a method for reconstructing a shape by identifying a high-speed time-series pattern generated by a DLP projector (Non-patent Document 7). In addition, studies have been conducted to reduce the number of patterns necessary for shape restoration (Non-Patent Document 8) and (Non-Patent Document 5). *

On the other hand, in the spatial encoding method, since the pattern is fixed, it is possible to restore the shape from only one frame in the video, which is suitable for dynamic scene measurement (Non-Patent Document 9) (Non-Patent Document 10). ), (Non-Patent Document 11), (Non-Patent Document 12), (Non-Patent Document 13). *

Furthermore, one method for dealing with the above-described problem is described in Patent Document 1.

JP 2009-3000277 A

However, the techniques described in the above documents have the following problems.

In the SFGs described in Non-Patent Document 1 to Non-Patent Document 3, in order to obtain a unique solution, the density information of the pattern line interval is used. For this reason, there is a restriction that both vertical and horizontal patterns cannot be made sufficiently dense. This not only causes a decrease in measurement density, but also causes a problem in that the shape cannot be restored because the intersections tend to be insufficient in a thin or small shape.

In the technical matters described in Non-Patent Document 4 and Non-Patent Document 5, when a time encoding method is used, stable high-precision restoration is possible in a static scene, but a plurality of different patterns can be projected. Because it is necessary, it is inherently difficult to use for dynamic scenes.

In the methods described in Non-Patent Document 6 to Non-Patent Document 8, these methods can acquire depth information at a high frame rate, but it is necessary to recognize a time-series code. Must move at a constant speed or less. Furthermore, high-precision synchronization of the equipment to be used is required.

In the methods described in Non-Patent Document 9 to Non-Patent Document 13, in many cases, a complicated pattern is used, so that it is affected by the texture to be observed, or the spatial pattern information cannot be identified at the depth edge portion. The problem that the error becomes large occurs. Furthermore, in order to measure a wide range, when patterns are projected from a plurality of projectors onto the same observation target, the patterns interfere with each other, so that the separation is not easy.

Furthermore, in the technique described in Patent Document 1, the remaining one-dimensional indeterminacy has been eliminated by matching the projected pattern with the obtained solution, but matching is required. Minute calculation time has increased.

The present invention has been made to solve the various problems described above, and the main object of the present invention is to measure the shape of a moving object at a high density and a high frame rate, and obtained as a result. An object of the present invention is to provide an image processing apparatus, an image processing method, and a program for appropriately restoring the three-dimensional shape of an object projected on a two-dimensional image.

The present invention is an image processing device that restores a three-dimensional shape from a two-dimensional image, and the two-dimensional image includes a first light projecting unit that projects a first pattern onto an object existing in a three-dimensional space; Photographing the first pattern light and the second pattern light reflected by the object, second light projecting means for projecting the second pattern intersecting the surface of the first pattern and the object onto the object; A first curve that is the first pattern obtained by the imaging means for obtaining a two-dimensional image and projected on the object in the two-dimensional image; and the second curve projected on the object in the two-dimensional image. A first calculation unit that detects a second curve as a pattern and calculates an intersection coordinate that is a coordinate of an intersection between the first curve and the second curve; the intersection coordinates; the first light projecting unit; Parameters of the second light projecting means and the photographing hand A second calculation unit that determines a first correspondence that is a correspondence between the first curve and the first pattern and a second correspondence that is a correspondence between the second curve and the second pattern, from the parameters of: By calculating the three-dimensional coordinates of the object of the portion irradiated with the first pattern light and the second pattern light from the first correspondence, the second correspondence, or both, the third shape is restored. And a calculation unit.

Furthermore, the present invention is an image processing method for restoring a three-dimensional shape from a two-dimensional image, wherein the two-dimensional image includes first light projecting means for projecting a first pattern onto an object existing in a three-dimensional space. Photographing the first pattern light and the second pattern light reflected by the object; and second light projecting means for projecting the second pattern intersecting the surface of the object to the object. And a first curve that is the first pattern obtained by the imaging means for obtaining a two-dimensional image and projected onto the object in the two-dimensional image, and the first curve projected onto the object in the two-dimensional image. A first step of detecting a second curve that is two patterns and calculating an intersection coordinate that is a coordinate of an intersection of the first curve and the second curve; the intersection coordinates; the first light projecting means; Parameters of the second light projecting means, and A second step of determining a first correspondence that is a correspondence between the first curve and the first pattern and a second correspondence that is a correspondence between the second curve and the second pattern from the parameters of the shadow means; The three-dimensional shape is restored by calculating the three-dimensional coordinates of the object of the portion irradiated with the first pattern light and the second pattern light from the first correspondence, the second correspondence, or both. An image processing method comprising: three steps.

Furthermore, the present invention is a program for causing an image processing apparatus to execute a function of restoring a three-dimensional shape from a two-dimensional image, and the two-dimensional image projects a first pattern onto an object existing in the three-dimensional space. First light projecting means, second light projecting means for projecting a second pattern intersecting the first pattern and the surface of the object to the object, the first pattern light reflected by the object, and the first light A first curve, which is the first pattern, obtained by photographing means for photographing a two-pattern light to obtain a two-dimensional image and projected onto the object in the two-dimensional image, and the object in the two-dimensional image A first function that detects a second curve that is the second pattern projected onto the first pattern and calculates an intersection coordinate that is a coordinate of an intersection between the first curve and the second curve; the intersection coordinates; 1 light projecting means and the second light projecting means A first correspondence that is a correspondence between the first curve and the first pattern and a second correspondence that is a correspondence between the second curve and the second pattern are determined from the parameters and the parameters of the photographing unit. The three-dimensional shape is calculated by calculating the three-dimensional coordinates of the object of the portion irradiated with the first pattern light and the second pattern light from the two functions and the first correspondence, the second correspondence, or both. And a third function for restoring the program.

The present invention can obtain a solution without indefiniteness only from information on the intersections of patterns projected on an object. Therefore, since it is not necessary to use pattern density information, both vertical and horizontal patterns can be made dense. In the background art, particularly in the restoration of a thin area, it is difficult to detect the connection relation of the grid pattern and the restoration may fail. However, the present invention greatly reduces this problem.

Further, in the present invention, since a plurality of projectors are used, it is possible to greatly reduce the portion where the pattern is shielded and the shape cannot be measured.

Furthermore, in the present invention, a technique using a line ID based on a de Bruijn sequence is adopted to improve the accuracy of the solution. Furthermore, a technique using adjacent information of detected lines is adopted. In this way, dense one-shot shape measurement using a plurality of projectors and one camera system was realized. In addition, a one-shot restoration linear solution using a grid pattern was realized. Furthermore, it has become possible to measure moving objects at a high frame rate.

(A) is a figure which shows the state which acquires the two-dimensional image of an object using the image processing apparatus of this invention, (B) is a figure which shows the structure of an image processing apparatus. It is a flowchart which shows the image processing method of this invention. It is a figure which shows the positional relationship of the camera and projector which are contained in the image processing apparatus of this invention. In the image processing apparatus of this invention, it is a figure which shows the pattern position on the image surface of a projector. It is a figure which shows the experiment scene using the image processing apparatus of this invention. It is a figure which shows the setup of evaluation experiment, (A) shows an input image, (B) is a top view which shows arrangement | positioning of a projector, (C) is the side view. It is the figure which displayed the decompression | restoration result by three methods using the present invention on the same coordinate system, (A) is a restoration result by the method A using only the restraint of a grid point, (B) is restraint by adjacent information. (C) is a restoration result by the method C using adjacent information and line ID. It is a figure which shows the evaluation experiment with respect to the case where the distance of the axis | shafts of two pencils is short using this invention, (A) is an input image, (B) is a figure which shows arrangement | positioning of a projector, (C) Is a side view. (A) And (B) is a table | surface which shows the precision improvement by utilization of adjacent information and line ID. It is a figure which shows the effect of this invention, the image arrange | positioned on the left side is based on background art, The image arrange | positioned on the right side is based on this form, The image of the upper stage is an input image, The image of interruption Is a grid graph obtained by line detection, and the lower image is a three-dimensional shape restoration result. It is a figure which shows the effect of this invention, the figure arrange | positioned at the left side is an input image, the figure arrange | positioned at the center is the detected grid pattern, and the figure arrange | positioned at the right side is a three-dimensional shape restoration result. is there.

<First Embodiment: Image Processing Device>
With reference to FIG. 1, the structure of the image processing apparatus 10 which concerns on embodiment of this invention is demonstrated. FIG. 1A is a diagram illustrating an example of the entire apparatus according to the present embodiment, and FIG. 1B is a diagram illustrating a configuration of the image processing apparatus 10.

Referring to FIG. 1A, in the present embodiment, an image processing apparatus 10, a camera 28 (photographing means), a projector 24 (first light projecting means), and a projector 26 (second light projecting means). A two-dimensional image of the object 30 existing in the three-dimensional space is photographed, and the three-dimensional shape of the object is restored from the photographed two-dimensional image. Here, the camera 28 and the projectors 24 and 26 have been calibrated. That is, each internal parameter and the rigid body transformation parameter between apparatuses are known. Furthermore, the camera 28, the projectors 24 and 26, and the image processing apparatus 10 may be regarded as the image processing apparatus of this embodiment.

The projector 24 has a function of projecting light including a horizontal pattern onto the object 30 that is a subject. For example, an apparatus such as a video projector can be considered. In addition, line laser projectors may be arranged or combined. Alternatively, the laser light source may be irradiated in a plurality of directions by a prism or a beam splitter.

The projector 26 projects a vertical pattern on the object 30. Other features of the projector 26 are the same as those of the projector 24. Since a fixed pattern is projected from each projector, there is no need for synchronization between the camera 28 and the projectors 24 and 26.

The pattern light projected from the projector 24 and the pattern light projected from the projector 26 intersect on the surface of the object 30. The intersecting angle is arbitrary.

Here, in this embodiment, the two projectors can essentially restore the shape with a single color pattern, but use a color pattern to improve accuracy and stability. Specifically, stable vertical / horizontal line detection and separation are realized by using line detection based on belief propagation and a color code based on a debroin sequence which is a periodic pattern. This method is described in Non-Patent Document 10 and Non-Patent Document 13, for example. Furthermore, this matter is described in Joaquim Salvi, Joan Battle, and El Musta Muaddiv. Arobust-coded pattern projection for dynamic 3D scene mea-surlement. Pattern Recognition, 19 (11): 1055 1065, 1998. ).

The camera 28 is a means for photographing the object 30 and employs a solid-state imaging device such as a CCD image sensor. Specifically, the camera 28 images pattern light in which the pattern projected from the projectors 24 and 26 is reflected by the object 30. Furthermore, the intersection point where these pattern lights intersect is also photographed by the camera 28. A two-dimensional image is taken by the camera 28, and data based on the two-dimensional image is subjected to image processing by the image processing apparatus 10, whereby the three-dimensional shape of the object 30 is restored.

With reference to FIG. 1B, a configuration of an image processing apparatus 10 that restores a three-dimensional shape from a two-dimensional image will be described.

The image processing apparatus 10 according to the present embodiment mainly includes an image processing unit 12, a control unit 14, an input unit 16, a storage unit 18, a display unit 20, and an operation unit 22. The general function of the image processing apparatus 10 is to perform image processing on an input two-dimensional image, restore a three-dimensional shape, and output the image. Further, the embodied image processing apparatus 10 may be a computer such as a personal computer in which an application (program) for executing a predetermined function is installed, or dedicated to image processing configured to execute the predetermined function. It may be configured as a device. Furthermore, the respective parts constituting the image processing apparatus 10 are electrically connected to each other via a bus.

The image processing unit 12 is a part that performs a main image processing function, and includes a first calculation unit 12A, a second calculation unit 12B, and a third calculation unit 12C.

The first calculation unit 12A has a function of calculating the vertical curve of the vertical pattern and the horizontal curve of the horizontal pattern projected on the object 30 and the intersection coordinates of both patterns from the captured two-dimensional image.

The second calculator 12B has a function of determining the correspondence from the parameters of each projector, the parameters of the camera, and the light spot coordinates. Specifically, the first correspondence between the vertical pattern projected from the projector 26 and the vertical curve detected from the image and the second correspondence between the horizontal pattern projected from the projector 24 and the horizontal curve detected from the image are determined. is doing.

The third calculator 12C has a function of determining the three-dimensional coordinates of the portion irradiated with both pattern lights from the obtained first correspondence, second correspondence, or both.

Details of each part constituting the above-described image processing unit will be described in detail below as an image processing method. Here, the number of calculation units included in the image processing unit 12 is not necessarily the above-described three, and the number of calculation units may be increased or decreased as necessary.

The control unit 14 is a part that controls the operation of the entire image processing apparatus 10 (the image processing unit 12, the input unit 16, the storage unit 18, and the display unit 20).

The input unit 16 is a part where information is input to the image processing apparatus 10 from the outside. In the present embodiment, a moving image or a still image that is a two-dimensional image is input.

The storage unit 18 is a fixed storage disk represented by an HDD (Hard Disk Drive), a removable storage disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disk), a fixed or removable semiconductor memory, or the like. is there. In the present embodiment, the storage unit 18 stores a two-dimensional image before processing, a three-dimensional shape restored from the two-dimensional image, data during treatment, and the like.

Further, the storage unit 18 stores a program for causing the image processing apparatus 10 to execute the following image processing method. This program is called when the user operates the operation unit 22, and executes the functions of the respective parts described above so as to restore the three-dimensional shape data from the input two-dimensional image data.

The display unit 20 is, for example, a liquid crystal display, a CRT (Cathode Ray Tube), or a video projector, and displays an input two-dimensional image and a three-dimensional shape restored based on the two-dimensional image.

The operation unit 22 is, for example, a keyboard or a mouse. When the user operates the operation unit 22, the image processing apparatus 10 restores a three-dimensional shape from the two-dimensional image.

<Second Embodiment: Image Processing Method>
A method for restoring the three-dimensional shape of an object projected on a two-dimensional image using the image processing apparatus 10 having the above configuration will be described below.

Referring to FIG. 2, the image processing method of the present embodiment includes step S10 for capturing a two-dimensional image of an object, step S11 for extracting a vertical pattern curve and a horizontal pattern curve from the captured two-dimensional image, Step S12 for detecting the intersection of the pattern curve, Step S13 for deriving a simultaneous equation of the pattern curve from the intersection, and Step S14 for obtaining the three-dimensional shape of the pattern curve from the simultaneous equation. Here, deriving simultaneous equations in step S13 is equivalent to determining the correspondence between each pattern and each curve.

The image processing method embodied in this embodiment will be described below.

First, in step S10, pattern light is projected toward the object 30 from the projector 24 and the projector 26 using the system shown in FIG. Here, the projector 24 projects lateral pattern light that extends linearly in the lateral direction onto the object 30. Then, a horizontal pattern light extending linearly in the vertical direction is projected from the projector 26 onto the object 30. As a result, both pattern lights are projected onto the surface of the object 30, and the intersection of both pattern lights exists on the surface of the object 30. The object 30 in this state is photographed by the camera 28. Thereby, a two-dimensional image of the object 30 irradiated with each pattern light is obtained. Image data based on the two-dimensional image is transmitted to the image processing apparatus 10.

In step S11, a vertical pattern curve and a horizontal pattern curve are extracted from the two-dimensional image. In order to stably detect a color code from a two-dimensional image, it is desirable to reduce the total number of colors. Therefore, the line detection method proposed in Non-Patent Document 2 is used to identify the vertical and horizontal patterns while using the same color.

The debroin sequence used in this embodiment is used to determine the line ID given to each line. A DeBruin sequence of length n and number of symbols q is a number sequence of length qn. If a partial number sequence of length n is observed, the position in the number sequence can be uniquely specified. When the projection pattern is coded with two or more symbols that can be distinguished in the image, the correspondence between the projection pattern and the observation pattern is uniquely determined by matching a sequence of length n. In Non-Patent Document 10, Non-Patent Document 10, and Non-Patent Document 13, large q and n values are used to determine a globally unique ID. Use a pattern generated using values.

This is because, in this embodiment, the deburin ID is used only for correcting erroneous estimation due to the influence of noise or the like, and therefore it is sufficient if local uniqueness is ensured. Because of the small q and n, the number of continuous patterns required to determine the ID is small, and the ID can be expressed with a small number of colors, so that it is less susceptible to shape irregularities and image processing. In this embodiment, the number of colors q = 2 and the cord length n = 3 are used. That is, the period of the color pattern is 8 and the line ID given to each line is 0 to 7. In addition, an erroneous connection of lines that occurs at an occluded edge or the like can also be eliminated by using this deburin ID.

The position of the intersection of the vertical pattern and the horizontal pattern is calculated with sub-pixel accuracy (step S12). Further, since adjacent intersections can be found by using the continuity of the detected lines, a grid graph in which the intersections are connected in a lattice shape is obtained as a result of line detection. By using the intersection coordinates and the grid graph that is the connection information of the intersections, three-dimensional reconstruction is realized by the method described below.

Here, a restoration method using two projectors and one camera which are the minimum configuration will be described. When a set of parallel lines is projected from a projector, the trajectory of each line in the three-dimensional space is a plane, and all the planes share a single straight line. That is, it is an element of pencil of planes (representing a set of surfaces sharing the same straight line, hereinafter referred to as “pencil”) around this straight line. When a plane is represented by three parameters and the three-dimensional vector is regarded as a point in a three-dimensional space, this point is called a plane dual, and the space is called a dual space. A plane dual is a point, and a pencil dual is a straight line. From this, a plane in a specific pencil can be expressed with one parameter in the same way as a point in a straight line is expressed with one parameter. At this time, it is possible to create a linear equation relating to the parameters of the pattern plane from the intersection position of the grid (step S13).

So far, Non-Patent Document 2 has proposed a method of projecting vertical and horizontal lines by a single projector to restore three-dimensionally. Also, this method is described in Ryo Furuka, Hiroshi Kawasaki, Ryusuke Sagawa, and Yasushi Yagi. Shape from grid pattern based on co-narity constraints for one-shot scanning. IPSJ Transaction on Computer Vision and Applications, 1: 139 157, 2009. But it is also mentioned.

In the above document, the intersection of vertical and horizontal lines is called a grid point. From the above document, the axis through which the vertical pattern plane passes and the axis through which the horizontal pattern plane passes intersect at the optical center of the projector. As a result, the constant term of the linear equation obtained from the grid point information is eliminated, so that the configured simultaneous equations are always indefinite.

In the above document, line density information is given to a pattern, and the remaining one-dimensional indefiniteness is solved by matching the projected pattern with the obtained solution. Further, Non-Patent Document 3 eliminates this indefiniteness by using the debrew-in ID.

On the other hand, in this embodiment, a unique solution is obtained by using a plurality of projectors and projecting the vertical plane and the horizontal plane with different projectors. That is, as shown in FIG. 3, the vertical pattern plane share a line l _v passing through the optical center of the projector (that is, a pencil to axis l _v.). Similarly horizontal pattern plane share l _h. It is unknown which pattern projected from the projector corresponds to each of the curves irradiated with vertical and horizontal pattern light observed by the camera. This information is restored from the equation for the observed curve. When l _v and l _h positions the projector so that the skewed, the resulting equation becomes a linear equation with a constant term, generally solution has no ambiguity. For this reason, a solution can be uniquely determined only from a linear equation. Pattern plane p

Represented by At this time, the three-dimensional vector p = (p ₁ , p ₂ , p ₃ ) ^T is a plane parameter vector. The vector p is also called the original plane dual vector. The space of vector p is also called the dual space of the original space. Some vertical pattern plane and _{_{v a,} v a} and different vertical pattern planes and _{v b.} At this time, _a set of planes including l _v that is an intersection of v _a and v _b is expressed by the following expression.

This equation has the same form as the equation of a straight line in the three-dimensional space, and represents that a set of vertical pattern planes is included in a straight line passing v _a and v _b in the dual space. In order to express this straight line, using an arbitrary point v ₀ on the straight line and a direction vector (or a normalized infinity point) v _inf of the straight line,

Can be expressed. Here, μ is a parameter that can be defined for each line on the image projected by the projector.

v ₀ can be arbitrarily selected from a set of planes including l _v . Further, the plane containing the l _v, but only plane passing through the optical center of the camera exists, v _inf coincides with the normal vector of the plane. Equation (1) assumes that the plane does not pass through the optical center of the camera, but as the plane approaches the optical center of the camera, the parameter p approaches the infinity point in the direction of v _{inf in} dual space. .

Similarly, the set of horizontal patterns

It is assumed that

Assume that an intersection of a certain vertical pattern v and horizontal pattern h is observed at (s, t) in normalized camera coordinates. At this time, if the dual vectors of v and h are v and h, and u = (s, t, −1) ^T ,

It is. Substituting equations (4) and (2),

Get.

Since the above equation is obtained for each grid point, a linear simultaneous linear equation can be created from this. For the i-th vertical plane, the parameter μ in equation (2) is μ _i , and for the j-th horizontal plane, the parameter ρ in equation (4) is ρ _j . Also, assuming that K grid points are detected, the k-th grid point u _k = (s _k , t _k , −1) is the α (k) -th vertical plane and β (k) -th horizontal Suppose that it is an intersection with a plane. At this time,

Holds for k = 1,..., K.

Although a mathematical expression similar to the mathematical expression 7 is also given in Non-Patent Document 2, as a significant difference from the above-mentioned document, in this embodiment, the mathematical expression 7 has a constant term, and this term is not erased by variable substitution or the like. . In Non-Patent Document 2, the constant term can be deleted as it is or by replacing variables. Therefore, the solution of the linear equation is not unique, but in this embodiment, the only solution is obtained from the linear equation.

Furthermore, when adjacent information between patterns is given, a constraint equation using the information can be created. In particular, when the difference in μ between adjacent lines on the pattern is a constant value (corresponding to a set of pattern planes arranged at equal intervals in the dual space), the constraint equation is linear as described below. convenient. Therefore, in the present embodiment, it is assumed that μ has a linear relationship that changes at a constant interval in adjacent patterns as described above, and the following discussion will proceed.

As will be described later, this can always be established by adjusting the interval of the pattern to be projected or by devising the arrangement of the projectors, so that the generality of the method is not lost by this assumption. Consider a case in which L pairs of patterns are adjacent. If the l th pair is the γ (l) th vertical plane and the δ (l) th vertical plane,

It becomes. D is a constant and can be determined in advance from how to take v ₀ and v _inf and the order relationship of μ _{γ (l)} and μ _{δ (l)} .

The pattern plane of each detected curve can be determined by solving the linear simultaneous equations relating to μ _i and ρ _j obtained from the equations (7) and (8).

In the actual calculation, the matrix equation Ax = b is generated. Where x is a vector in which the detected parameters μ _i and ρ _j of the vertical and horizontal patterns are arranged. If there are sufficient intersections and adjacent information, the number of rows of A is larger than the number of columns, so a solution is obtained by x = (A ^T A) ⁻¹ A ^T c using a pseudo inverse matrix. For the calculation of (A ^T A) ⁻¹ , an equation solving method by LU decomposition may be used.

The solution obtained by the above equation represents the position information of the corresponding pattern plane for the observed curve of the vertical pattern or the horizontal pattern. When this information is obtained, the curve can be reconstructed three-dimensionally by the principle of triangulation. Here, curves observed in both the vertical pattern and the horizontal pattern may be used (step S14).

The above position information may use the solution obtained from the equation as it is, but the projected pattern plane is finite and the position is known. May be the corresponding plane. Thereby, if the accuracy of the solution obtained from the equation is high, three-dimensional restoration without errors other than calibration errors can be performed.

Next, the arrangement of the projector and camera will be described.

In general, even if the line pattern on the image of the projector is at a constant interval, μ in Equation (3) does not have a value proportional to the pattern coordinates (that is, the dual of the pattern plane does not have a constant interval). For this reason, in order for μ to be linear according to the pattern, it is necessary to calculate the pattern to be projected every time according to the rigid transformation between the projector and the camera. In this case, the pattern is changed every time the positional relationship between the projector and the camera is changed, and convenience is lost.

As another method, even when a pattern with a constant interval is projected, an arrangement is possible in which the positional relationship between μ and the image plane of the projector is linear. A plane passing through the optical center of the projector and parallel to the image plane of the projector Faugeras. Three-dimensional computer vision: A geo-metric viewpoint. The MIT press, Cambridge, MA, 1993. The focal plane of the projector (focal plane) will be referred to.

As shown in FIG. 4, the optical center of the camera is included in the focal plane of the projector, and the dual of the pattern plane including the optical axis of the projector is v ₀ . At this time, a set of vertical pattern planes generated from the equidistant grids become equidistant points even in the dual space. This is intuitively shown by the fact that the position of the pattern at infinity on the image plane coincides with the point at infinity in the dual space (the plane including the optical center of the camera). In this embodiment, if the projector and the camera are arranged in this way, the adjacent relationship of the patterns can be incorporated into the linear equation.

As an example of such an arrangement, the front direction of the vertical pattern projector (first light projecting means) and the direction from the camera toward the optical center of the vertical pattern projector (first light projecting means) are orthogonal and horizontal. There is an arrangement in which the front direction of the pattern projector (second light projecting means) and the direction from the camera toward the optical center of the horizontal pattern projector (second light projecting means) are orthogonal to each other.

If the above arrangement is not satisfied, the adjacency relationship does not become a linear equation, but can be used as a nonlinear constraint condition of the solution. Next, an explanation will be given regarding the accuracy improvement using the devourin series. As described above, the three-dimensional position of the pattern plane including the detected curve can be uniquely determined from the linear simultaneous equations. However, in practice, if an erroneous line and grid point are detected due to an error in image processing, an error may occur in the obtained solution. Therefore, in the present embodiment, the error of the solution is corrected by using the periodic line ID attached to the curve by the debroin sequence. In this embodiment, since it is assumed that the camera and the projector have been calibrated, the parameter of the pattern plane projected from the projector and the line ID of the debroin series of each plane are known. At this time, possible solutions are limited to those whose surface positions and line IDs coincide with these planes. The solution is corrected using this principle. Specifically, in the vicinity of the solution for the detected curve obtained above, the projection pattern IDs are matched with each other, and whether or not the equation (5) that is the condition of each intersection is established. Investigate. This condition represents that the distance on the image between the projected line of two planes on the camera and the detected grid point is actually zero. Therefore, these distances are obtained for each grid point, the sum of squares thereof is obtained, and a solution having a small value is selected.

In the above procedure, when searching for “around a solution for a curve”, if a variable is changed for all the planes, it is necessary to search a large dimensional space. In order to shorten the calculation time, it is efficient to obtain an eigenvector corresponding to the minimum eigenvalue of (A ^T A) for the coefficient matrix A of the linear equation and search for a solution in the direction of this vector. This is justified in the above equation because the direction in which errors are likely to occur is often a one-dimensional space. When the adjacency relationship becomes a non-linear constraint, the adjacency relationship may be used as a means for improving accuracy in the same manner as the accuracy improvement described above.

When the adjacent relationship between patterns is used in a linear equation, a straight line shared by the vertical pattern (first light projection means axis) and a straight line shared by the vertical pattern (second light projection means axis) are special. Except in some cases, a unique solution is required even if they meet. However, when the adjacent relationship between patterns is not used in a linear equation, the equation becomes indefinite when these axes intersect. For this reason, when the adjacent relationship cannot be obtained, it is necessary to arrange the projector so that these axes do not intersect.

As described above, a solution method different from the method of directly solving the linear equation is also conceivable. As already described, the projected pattern plane is finite and the position is known, so the solution (correspondence between the observed curve and the projected known pattern plane) is finite. Therefore, a solution that satisfies Equations 7 and 8 may be obtained by a search by combination optimization. In this case, when the correspondence with respect to a certain curve A is adopted as a hypothesis, the position of the curve is determined, and the position of the curve B intersecting with the curve A or the curve C adjacent to the curve A is calculated from the equations 7 and 8. Information is obtained and the hypothesis of B or C correspondence is also limited. In this way, by finding the correspondence of the pattern plane to the curve one after another, the solution can also be obtained by obtaining the hypothesis that best satisfies the conditions of Equations 7 and 8 from the limited hypotheses.

<Third Embodiment: Experimental Results>
In the present embodiment, experimental results using the above-described image processing apparatus and image processing method of the present embodiment will be described.

In the experiment of this embodiment, as shown in FIG. 5, one camera and two projectors were arranged and measured. The arrangement of each device was adjusted each time depending on the observation target, and in the measurement of a person, the distance between the camera and the projector was about 1 m, and the relative angle was about 25 degrees. The camera resolution is 1280 × 960 pixels, the frame rate is 15 FPS, and the projector resolution is 1024 × 768 pixels.

First, the evaluation of the accuracy improvement of the solution using the adjacent information and the line ID will be described. Specifically, regarding the effect of shape restoration using adjacent information and line ID, a cube was observed under the following three conditions, and the three-dimensional restoration results were compared. (Method A): Use only constraints obtained from grid points (Method B): Use constraints between grid points and adjacent information (Method C): Solution correction using line ID in addition to constraints between grid points and adjacent information Use of the restored point data was applied to two planes, and the root mean square error (RMSE) and the angle between the two planes were evaluated. In the experimental environment, the distance from the camera to the observation target was 1.6 m.

The arrangement of the input image and the projector is shown in FIG. FIG. 7 shows the result of restoring the three-dimensional shape of the two surfaces of the cube, and the table of FIG. 9A shows the angle between the planes and the RMS error in the plane fitting. From these results, it is understood that the angle between the two surfaces approaches 90 degrees by using the adjacent information (method B) and the line ID (method C), so that the accuracy of the shape is improved. Compared to the time encoding method using the gray code, the method C obtained the same result as the gray code method except for errors due to the line detection because the correct correspondence was obtained by the correspondence using the line ID.

The RMS error when the three-dimensional shape restoration result was applied to a plane increased each time additional information was given. This is a natural result for the following reasons. When there is no additional information, restoring the vertical and horizontal lines minimizes the distance between the vertical and horizontal lines at the intersection to minimize the distance between the vertical and horizontal lines. As a result, the RMS error during plane fitting is small. Become. On the contrary, when the constraint of adjacency or line ID is added, the deviation between the vertical line and the horizontal line becomes large, and the error at the time of plane fitting becomes large. This is because in an actual system, it is difficult to perform calibration with sub-pixel accuracy or less over the entire screen.

Next, the case where the distance between the axes of two pencils was short was evaluated using the projector arrangement shown in FIG. Similarly, a cube is measured, and the result of restoring the three-dimensional shape is applied to two planes. The angle between the surfaces and the RMS error at the time of fitting are shown in FIG. The distance between the camera and the observation target was 3.7 m. Under this condition, method A failed due to a rank drop, but methods B and C showed that three-dimensional shape restoration can be achieved by using constraints based on adjacent information.

Next, the shape restoration of a small object will be described. Specifically, a comparison was made with the one-shot shape restoration (non-patent document 2) by the conventional method using only one projector. In the conventional method, a one-dimensional search is necessary to determine one degree of freedom remaining in the linear solution, but in this embodiment, a dense pattern can be used as compared with Non-Patent Document 2. The advantage is that a fine shape such as a thin finger can be easily restored.

FIG. 10 is a comparison of the results of both methods. The upper row is an input image, the middle row is a grid graph obtained by line detection, and the lower row is a three-dimensional shape restoration result. It can be seen that the conventional method shown on the left side lacks a finger due to a lack of pattern, whereas the present embodiment shown on the right side can restore the shape without missing the finger.

Finally, the upper body of a moving person was measured in order to confirm the dense shape measurement of the human body. FIG. 11 shows the shape restoration results for a plurality of poses. In this figure, column (A) is an input image, and the result of determining line detection and line ID by line detection processing is column (B). The line ID is represented by the color of each line (in this case, shading). Column (C) shows the result of the shape restoration result using the line detection result. In some parts such as the flank and shoulders, there is a blind spot from one projector, and there is a part where only a vertical or horizontal pattern is projected. However, it can be seen that the shape is restored by connecting the pattern to a portion that is not a blind spot. This is one of the advantages of this embodiment using a plurality of projectors.

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 12 Image processing part 12A 1st calculation part 12B 2nd calculation part 12C 3rd calculation part 14 Control part 16 Input part 18 Storage part 20 Display part 22 Operation part 24 Projector 26 Projector 28 Camera 30 Object

Claims

An image processing apparatus that restores a three-dimensional shape from a two-dimensional image, wherein the two-dimensional image is a first light projecting unit that projects a first pattern onto an object existing in a three-dimensional space; and the first pattern And a second light projecting means for projecting a second pattern intersecting the surface of the object to the object, and photographing the first pattern light and the second pattern light reflected by the object to obtain a two-dimensional image. And a first curve that is the first pattern projected onto the object in the two-dimensional image and the second pattern that is projected onto the object in the two-dimensional image. A first calculation unit that detects two curves and calculates an intersection coordinate that is a coordinate of the intersection between the first curve and the second curve; the intersection coordinates, the first light projecting unit, and the second light projecting unit Parameters of the means, and of the photographing means A second calculation unit for determining a first correspondence that is a correspondence between the first curve and the first pattern and a second correspondence that is a correspondence between the second curve and the second pattern from the parameter; The third calculation for restoring the three-dimensional shape by calculating the three-dimensional coordinates of the object of the portion irradiated with the first pattern light and the second pattern light from one correspondence, the second correspondence, or both And an image processing apparatus comprising: a unit.
2. The apparatus according to claim 1, wherein the first light projecting unit and the second light projecting unit project a first pattern image and a second pattern image, which are images of a plurality of line segments, into a three-dimensional space. An image processing apparatus according to 1.
The third calculation unit calculates the three-dimensional shape by solving a first linear equation obtained from parameters of the intersection coordinates, the first light projecting unit, and the second light projecting unit. The image processing apparatus according to claim 1.
The third calculation unit calculates the three-dimensional shape by solving as a combination optimization problem that determines the correspondence with the intersection coordinates detected by the first calculation unit as a constraint. The image processing apparatus according to claim 1.
The image processing apparatus according to claim 1, wherein an axis of the first light projecting unit and an axis of the second light projecting unit do not intersect in a three-dimensional space.
The first light projecting unit and the second light projecting unit are arranged so that a first condition that a set of planes through which the first pattern or the second pattern passes in a three-dimensional space is arranged at equal intervals in a dual space is satisfied. 6. The image processing apparatus according to claim 1, further comprising a light projecting unit.
The first condition is that the front surface of the first light projecting unit and the direction vector from the image capturing unit to the first light projecting unit are perpendicular to each other, and the front surface of the second light projecting unit and the image capturing unit The image processing apparatus according to claim 6, wherein the image processing apparatus is satisfied by acquiring the two-dimensional image in a state where a direction vector to the second light projecting unit is a right angle.
The first pattern or the second pattern is provided with identification information capable of identifying each of the patterns into several classes. In the third calculation unit, the identification information and the first pattern or the The image processing apparatus according to claim 1, wherein a constraint condition that the classes of the second pattern match is used.
The image processing apparatus according to claim 8, wherein the identification information is provided by setting the color of the pattern included in the first pattern or the second pattern to two or more colors.
The said 2nd calculation part calculates | requires the said 1st correspondence or the said 2nd correspondence, or both using the relational expression of the said adjacent 1st patterns, or the said 2nd adjacent patterns. The image processing apparatus according to claim 1.
An image processing method for restoring a three-dimensional shape from a two-dimensional image, wherein the two-dimensional image is a first light projecting unit that projects a first pattern onto an object existing in a three-dimensional space; and the first pattern And a second light projecting means for projecting a second pattern intersecting the surface of the object to the object, and photographing the first pattern light and the second pattern light reflected by the object to obtain a two-dimensional image. And a first curve that is the first pattern projected onto the object in the two-dimensional image and the second pattern that is projected onto the object in the two-dimensional image. A first step of detecting two curves and calculating an intersection coordinate which is a coordinate of an intersection between the first curve and the second curve; the intersection coordinates, the first light projecting means and the second light projecting means Parameters and the photographing means A second step of determining, from the parameters, a first correspondence that is a correspondence between the first curve and the first pattern, and a second correspondence that is a correspondence between the second curve and the second pattern; A third step of restoring the three-dimensional shape by calculating the three-dimensional coordinates of the object of the portion irradiated with the first pattern light and the second pattern light from the correspondence, the second correspondence, or both; An image processing method comprising:
A program for causing an image processing apparatus to execute a function of restoring a three-dimensional shape from a two-dimensional image, wherein the two-dimensional image is a first light projecting unit that projects a first pattern onto an object existing in a three-dimensional space. Photographing the first pattern light and the second pattern light reflected by the object; and second light projecting means for projecting the second pattern that intersects the first pattern and the surface of the object to the object. And a first curve that is the first pattern projected onto the object in the two-dimensional image and the first curve projected onto the object in the two-dimensional image. A first function that detects a second curve that is a second pattern and calculates an intersection coordinate that is a coordinate of an intersection between the first curve and the second curve; the intersection coordinates; the first light projecting means; Parameters of the second light projecting means And a first correspondence that is a correspondence between the first curve and the first pattern and a second correspondence that is a correspondence between the second curve and the second pattern are determined from the parameters of the imaging means and the parameters of the photographing means. The three-dimensional shape is calculated by calculating the three-dimensional coordinates of the object of the portion irradiated with the first pattern light and the second pattern light from the two functions and the first correspondence, the second correspondence, or both. A third function that restores the program, and a program that executes