JP2021009140A - Measuring apparatus, imaging apparatus, measuring system, control method and program - Google Patents

Abstract

To realize good measuring accuracy while suppressing increase of an apparatus scale or reduction of usability.SOLUTION: A measuring apparatus comprises: a projection device that projects a predetermined pattern onto a subject; an imaging system that images an image group from at least two or more different view points, distance among the view points being shorter than distance between the projection device and the view point; and relative position calculation means that calculates a relative position of the projection device relative to the at least one view point from a pattern image position on the image group and a physical relationship between the view points, and acquires distance information of the subject from the relative position and the pattern image position on an image in the view point.SELECTED DRAWING: Figure 1

Description

The present invention relates to a measuring device, an imaging device, a measuring system, a control method and a program, and more particularly to a measuring technique in which a plurality of methods of distance measurement including a stereo method are used in combination.

Conventionally, a technique for acquiring distance information and shape information of a subject by using the principle of triangulation from an image obtained by imaging, a so-called stereo method, has been proposed. For example, there is a passive stereo method in which the shape of a subject is measured by the principle of triangulation using a parallax image pair acquired by imaging the same subject with two imaging devices placed at different positions. In addition, there is a pattern projection stereo method as a method for measuring the shape of an object having a small texture or in a dark place. In the pattern projection stereo method, a predetermined pattern is projected onto a subject by a projection device and an image is taken by an imaging device placed at a position different from that of the projection device. The positional relationship between the projection device and the imaging device and the image obtained by the imaging are displayed. From the positional relationship of the patterns of, the distance information and shape information of the subject are acquired using the principle of triangulation. Further, for the purpose of improving accuracy and the like, a hybrid type in which the pattern projection stereo method and the passive stereo method are used in combination has also been proposed (Patent Document 1).

JP-A-2002-286415

Patent Document 1 proposes a hybrid type in which a passive stereo method and a pattern projection stereo method are used in combination, but a specific configuration and method thereof are not disclosed.

Since the stereo method uses the principle of triangulation, it is important that the positional relationship between the two elements that generate parallax is default (known) and stable in order to obtain high measurement accuracy. Here, the two elements are two imaging devices in the passive stereo method, and a projection device and an imaging device in the pattern projection stereo method. Therefore, when the passive stereo method and the pattern projection stereo method are used in combination, the positional relationship between the two imaging devices and the positional relationship between one imaging device and the projection device are known and stable. It is important to have high measurement accuracy.

Further, in principle of triangulation, the longer the distance (baseline length) between two elements that generate parallax, the more advantageous it is to obtain higher measurement accuracy.

However, in order to know and stabilize the positional relationship between the two elements in the presence of temperature environment and vibration, it is necessary to firmly fix the two elements, which makes the configuration of the measurement system complicated and large. There are challenges. Furthermore, if the baseline length, which is the distance between the two elements, is increased in order to measure with high accuracy, the mechanical stability is lowered, which causes a problem that the mechanism for holding the two elements is further increased in size. there were.

On the other hand, if the positional relationship between the projection device and the image pickup device is fixed, the range in which the position and orientation of the projection can be adjusted is limited, for example, by projecting a pattern according to the shape of the subject so as not to cause shadows. , There is also a problem that usability is reduced.

Further, since a texture can be formed on the subject by projecting a pattern using a projection device, it is possible to measure the shape by the passive stereo method even in an object with a small texture or in a dark place. However, in this case, the pattern projection is only for the purpose of improving the passive stereo method, and is different from the configuration in which the pattern projection stereo method is used together. Therefore, there is a problem that the measurement stability and accuracy are lower than those of the hybrid type.

The present invention aims to alleviate at least one of the problems of the prior art, and is a measuring device that realizes good measurement accuracy while suppressing an increase in the scale of the device and a decrease in usability. It is an object of the present invention to provide an imaging device, a measurement system, a control method and a program.

In a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures image groups from at least two different viewpoints.
The distance between the viewpoints is shorter than the distance between the projection device and the viewpoints.
It has a relative position calculation means for obtaining a relative position of the projection device with respect to at least one of the viewpoints from the pattern image position on the image group and the positional relationship between the viewpoints.
A measuring device characterized in that distance information of the subject is acquired from the relative position and the pattern image position on the image at the viewpoint.

According to the present invention, good measurement accuracy can be realized while suppressing an increase in the scale of the apparatus and a decrease in usability.

Block diagram showing the functional configuration of the measurement system according to the embodiment of the present invention A flowchart illustrating the distance measurement process executed by the measuring device 150 according to the first embodiment. The figure for demonstrating the projection mode of the pattern in 1st Embodiment. A flowchart showing a modified example of the distance measurement process executed by the measuring device 150 according to the first embodiment. A flowchart showing another modification of the distance measurement process executed by the measuring device 150 according to the first embodiment. The figure which illustrated the method of calculating the position and the posture of the projection apparatus 110 which concerns on 1st Embodiment. Block diagram showing the functional configuration of the measurement system according to the second embodiment The figure for demonstrating the detailed configuration of the imaging system 120 which concerns on 2nd Embodiment A flowchart illustrating the distance measurement process executed by the measuring device 150 according to the third embodiment. The figure which illustrated the captured image obtained in the state which made the pattern projection which concerns on 3rd Embodiment A flowchart showing a modified example of the distance measurement process executed by the measuring device 150 according to the third embodiment. A flowchart showing another modification of the distance measurement process executed by the measuring device 150 according to the third embodiment. A flowchart illustrating the distance measurement process executed by the measuring device 150 according to the fourth embodiment. The figure which illustrated the captured image obtained in the state which made the pattern projection which concerns on 4th Embodiment

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

One embodiment described below includes two imaging devices (cameras) and one projection device as an example of a measurement system, and a measuring device to which these are connected, and measures the distance from the imaging device to the subject. An example in which the present invention is applied to a possible measurement system will be described. However, the present invention can be applied to any device capable of measuring the distance in the depth direction using the passive stereo method in the state where the pattern is projected.

[First Embodiment]
FIG. 1 shows a configuration example of the measurement system 100 in the present invention.

In FIG. 1, the measurement system 100 includes a projection device 110 that projects a pattern on a subject, an imaging system 120 that acquires a parallax image based on a luminous flux from the subject, and a measurement device 150.

The projection device 110 is composed of a projection optical system 111, a pattern unit 112 having an illumination unit such as a spatial modulator, a reticle, and an LED. Reference numeral 113 denotes an exit pupil of the projection optical system 111, and 114 is a projection axis representing the projection direction of the projection device 110. The projection device 110 has a control unit (not shown) such as a microcomputer including a ROM in which a control program of each block of the projection device 110 is recorded and a RAM used as an expansion area of the control program, and has a pattern unit 112. It controls and projects a pattern (light) on the subject.

The imaging system 120 is composed of a first camera 130 and a second camera 140, and images a group of images from two different viewpoints. The first camera 130 has a first image pickup optical system 131 and a first image pickup element 132 such as a CCD or a CMOS sensor, with the first entrance pupil 133 of the first image pickup optical system 131 as a viewpoint. An image is taken with the image pickup axis 134 of 1 as the image pickup direction. The second camera 140 has a second image pickup optical system 141 and a second image pickup element 142 such as a CCD or a CMOS sensor, with the second entrance pupil 143 of the second image pickup optical system 141 as a viewpoint. An image is taken with the image pickup axis 144 of 2 as the image pickup direction. The distance between the first entrance pupil 133 and the second entrance pupil 143 is set to be shorter than the distance between the exit pupil 113 of the projection device and the first entrance pupil 133 of the imaging system 120. ing. Further, each of the first camera 130 and the second camera 140 performs image processing such as development and correction on the image signal output from the control unit including the CPU, ROM, and RAM, and the image sensor, and obtains the captured image of digital data. It is provided with a configuration generally provided in a digital camera, such as an image processing unit for generating.

The measuring device 150 includes a control unit 151 including a CPU, a memory 152, a relative position calculation unit 154, and a distance shape calculation unit 153. The control unit 151 reads the operation program of each block stored in the non-volatile portion of the memory 152 (hereinafter, referred to as ROM for convenience), expands the operation program of each block into another area of the memory 152, and executes the operation of each block. Control. Further, the measuring device 150 is connected to the imaging system 120 and the projection device 110, which are external devices, and can transmit and receive data to and from each device.

Next, the calculation flow of the distance shape performed by the measurement system 100 will be described with reference to FIG. 2A. Each process of this flow is executed by the control unit 151 of the measuring device 150 executing the corresponding control program, or by operating each block under the control of the control unit 151 or the control unit 151.

In step S1501, the control unit 151 gives an instruction to project a pattern to the projection device 110 via the communication unit, and also instructs the image pickup system 120 to take an image in a state where the pattern is projected to the image pickup system 120 via the communication unit. A pair of parallax images obtained by imaging is acquired from. As shown in FIG. 2B, the projection device 110 receives an instruction from the measurement device 150 and projects the pattern 200 onto the subject in the direction of the projection axis 114 by the pattern unit 112 and the projection optical system 111. In response to the instruction from the measuring device 150, the imaging system 120 captures a parallax image of the subject and the pattern 200 with the first camera 130 and the second camera 140.

In step 1502, the distance shape calculation unit 153 calculates the primary distance information of the subject from the positions of the subject and the pattern image on the captured parallax image and the positional relationship of each viewpoint by a known method of the passive stereo method.

Next, in step S1503, the relative position calculation unit 154 determines the position and orientation of the projection device 110 with respect to the position of the first entrance pupil 133 of the imaging system 120 from the position of the primary distance information of the subject and the pattern image on the parallax image. The position of a certain exit pupil 113 and the projection axis 114 are calculated. In the present embodiment, the position and orientation of the projection device 110 will be described as being relative to the position and orientation of the first camera 130. However, the position / orientation relative to the position / orientation of the second camera 140 may be used.

Next, in step S1504, the distance shape calculation unit 153 uses a known method of the pattern projection stereo method from the calculated position / orientation of the projection device 110, the position / orientation of the imaging system 120, and the position of the pattern image on the parallax image. Calculate the secondary distance information of the subject.

Then, in step S1505, the control unit 151 stores the calculated secondary distance information in a storage unit (not shown), or outputs the calculated secondary distance information to an external device via a recording unit, a communication unit, or the like. At this time, the distance (baseline length) between the exit pupil 113 of the projection device 110 and the first entrance pupil 133 of the imaging system 120 is between the first entrance pupil 133 and the second entrance pupil 143 of the imaging system 120. Longer than the distance (baseline length). Therefore, the secondary distance information obtained by the pattern projection stereo method by the projection device 110 and the imaging system 120 has higher distance accuracy than the primary distance information obtained by the passive stereo method by the imaging system 120. Therefore, it is only necessary to maintain the positional relationship between the first camera 130 and the second camera of the imaging system 120 at a baseline length shorter than the baseline length required to obtain the desired measurement accuracy. , High distance measurement accuracy can be obtained with a simpler configuration.

Further, as shown in FIG. 3A, the control unit 151 integrates the primary distance information obtained by the passive stereo method and the secondary distance information obtained by the pattern projection stereo method in addition to the processing performed in FIG. 2A to integrate the subject. Step S1506 for obtaining the distance information may be performed. Each process of the flow of FIG. 3A is also carried out by the control unit 151 of the measuring device 150 or by operating each block under the control of the control unit 151. For example, depending on the relationship between the position of the projection device 110 and the shape of the subject, there may be a region where the pattern is not projected as a shadow of the subject. At this time, the passive stereo method may be able to acquire the primary distance information by using the pattern of the subject as a clue, and by integrating the primary distance information and the secondary distance information, it is possible to obtain the distance information without any loss.

Next, according to step S1503, the control unit 151 calculates the position and orientation of the projection device 110, the position of the exit pupil 113, and the projection axis 114 from the primary distance information of the subject and the position of the pattern image on the parallax image. Will be described with reference to FIG. FIG. 4 shows a case where a pattern is projected onto the subject 300 to measure the distance. A pattern is projected from the exit pupil 113 of the projection device 110 in the direction of the projection axis 114, and parallax images are acquired for the first entrance pupil 133 of the first camera 130 and the second entrance pupil 143 of the second camera 140. To. First, among the patterns formed on the surface of the subject 300, the primary distance information of arbitrary two places (pattern 201, pattern 202) is calculated, and the position of the first entrance pupil 133 of the first camera 130 is obtained. Find the relative position coordinates of each pattern. The relative position coordinates with respect to the first entrance pupil 133 are, for example, orthogonal to the first entrance pupil 133 as the origin and the straight line connecting the first entrance pupil 133 and the second entrance pupil 143 as one axis. It is a position coordinate in a Cartesian coordinate system in which another axis is set in the direction of the movement. Next, when the projection angles of the pattern 201 and the pattern 202 are set to θ1, a circle 310 in which θ1 is the inscribed angle is obtained through the two points of the pattern 201 and the pattern 202. At this time, the exit pupil 113 of the projection device is present at any position on the circumference of the circle 310. The projection angle θ1 is known information about the projection device 110, and is determined from, for example, the geometric relationship between the pattern unit 112 and the projection optical system 111. Next, among the patterns formed on the surface of the subject 300, the primary distance information of another one place (pattern 203) is calculated, and the pattern is relative to the position of the first entrance pupil 133 of the first camera. Find the position coordinates. When the projection angles of the pattern 202 and the pattern 203 are set to θ2, a circle 311 passing through the two points of the pattern 202 and the pattern 203 and having θ2 as the inscribed angle is obtained. At this time, the exit pupil 113 of the projection device is present at any position on the circumference of the circle 311. That is, the intersection of the circle 310 and the circle 311 that is not the pattern 202 is the position of the exit pupil 113. In this embodiment, a pattern on the projection axis is selected as the pattern 202. At this time, the straight line including the line segment connecting the position of the exit pupil 113 and the position of the pattern 202 becomes the projection axis 114. In this way, the position of the exit pupil 113 and the projection axis 114, which are the positions and postures of the projection device 110, can be calculated from the primary distance information of the subject and the position of the pattern image on the parallax image.

In the present embodiment, the relative position information of three places on the surface of the subject is used, but the position of the exit pupil 113 is performed with higher accuracy by using the information of four or more places and using an averaging process or the like. And the projection axis 114 may be obtained. Further, when there is prior information regarding the coordinates of the position of the exit pupil 113 and the direction of the projection axis 114, the position of the exit pupil 113 and the projection axis 114 can be obtained from the relative position information of the two locations. For example, when the inclination of the projection axis 114 with respect to the coordinate system with the first entrance pupil 133 as the origin is known, the circle 310 obtained from the relative position information at two locations and the projection axis 114 passing through the position of the pattern 202 The intersection of is the position of the exit pupil 113. The method of obtaining the position of the exit pupil 113 of the projection device 110 and the projection axis 114 by using the geometric relationship as shown in FIG. 4 is not limited to the process shown in the present embodiment. Any method may be used in which the relative position of the projection device with respect to at least one viewpoint is obtained by using the pattern image position on the parallax image acquired by the imaging system 120 and the positional relationship between the viewpoints as input information, and intermediate data such as primary distance information can be obtained. A method that is not used may be used.

In the present embodiment, the position and orientation of the projection device 110 are plotted in a two-dimensional plane including a line segment connecting the first entrance pupil 133 and the second entrance pupil 143 of the imaging system 120 by using FIG. Explained how to ask for. If the position of the exit pupil 113 of the projection device 110 is not in this two-dimensional plane, the geometric relationship is formed even in a plane orthogonal to the plane shown in FIG. 4, including the first entrance pupil 133 of the first camera 130. The position and orientation of the projection device 110 are obtained by using. It is possible to obtain a three-dimensional position / orientation by synthesizing the position / orientation obtained in each plane. It is preferable that the projection device 110 is arranged at a position on an extension of a straight line connecting the first entrance pupil 133 and the second entrance pupil 143 of the imaging system 120. As a result, the position and orientation of the projection device 110 need only be obtained in the two-dimensional plane, and the amount of calculation can be reduced, so that the speed and power consumption can be suppressed.

With the above configuration, in the present embodiment, it is only necessary to maintain the positional relationship with the baseline length shorter than the baseline length required to obtain the desired measurement accuracy, and the distance measurement accuracy is stable and high with a simpler configuration. Can be obtained.

Further, since the position and orientation of the projection device 110 can be obtained when necessary, the position and orientation of the projection device 110 can be made variable without being fixed to a predetermined state. As a result, the direction and position of projecting the pattern can be determined so that the shape of the subject does not block the subject, and the conditions of the measurable subject can be greatly expanded. The position and orientation of the projection device 110 need not always be obtained for each measurement. For example, the position / orientation of the projection device 110 may be obtained when the position / orientation of the projection device 110 is actively changed or when there is a concern about a change in the positional relationship due to a disturbance such as an environmental temperature change or vibration. ..

In the pattern to be projected, a line segment extending in a direction perpendicular to the straight line connecting the first entrance pupil 133 and the second entrance pupil 143 of the imaging system 120 is in the direction of the straight line (in a pair of parallax images). It is preferable that the pattern is a striped pattern arranged substantially periodically in the direction in which parallax occurs). With this configuration, a parallax image with high contrast in the parallax direction of the imaging system 120 can be acquired, and the measurement accuracy of the primary distance information by the passive stereo method can be improved. The projection of such a suitable pattern can also be realized by dynamically constructing the pattern based on the captured image obtained by capturing the test pattern in the imaging system 120 after the test pattern is projected.

Further, it is desirable that at least one line segment (striped) of the striped pattern has a different color and shape from the other line segments. As a result, it is possible to suppress erroneous correspondence when taking correspondence between parallax images, and it is possible to reduce measurement errors. In particular, the calculation of the projection axis 114 can be facilitated by making the color and shape of the pattern on the projection axis 114 different from those of other line segments. Further, the pattern 200 projected to measure the primary distance information is placed on the imaging system side in advance so that when the imaging system 120 recognizes each projected pattern image, the relative positional relationship between the patterns can be known. It is also preferable that information on patterns (positional relationships between them) is stored. As a result, the imaging system 120 can easily estimate the positional relationship in the pupil division direction of the subject at the position where each pattern is projected from the positional relationship between the patterns (for example, the positional relationship between the stripes in the striped pattern). You can.

Further, the pattern used when obtaining the relative position of the projection device by the passive stereo method and the pattern used when acquiring the distance information by the pattern projection stereo method may be different. Further, the measurement system 100 may perform measurement in the flow shown in FIG. 3B in addition to the processing shown in FIG. 2A. Each process of the flow of FIG. 3B is also carried out by the control unit 151 of the measuring device 150 or by operating each block under the control of the control unit 151. After step S1503, as step S1507, the projection device 110 projects a pattern generally used in pattern projection stereo such as random dots and grades, and the subject on which the pattern is projected is imaged by the imaging system 120 to form a pair of parallax images. Is obtained. With this configuration, a pattern suitable for each method can be used, and position calculation and distance measurement with higher accuracy become possible. Further, if the pattern used in the pattern projection stereo method is projected as a pattern suitable for the shape of the subject, such as changing the fineness of the pattern by using the primary distance information, the distance information can be acquired with higher accuracy. ..

[Second Embodiment]
Next, the second embodiment will be described with reference to FIGS. 5A and 5B. As shown in FIG. 5A, the image pickup system 120 is different from the first embodiment in that the camera 400 is configured. The processing flow includes the example shown in FIG. 2A and is the same as that of the first embodiment.

The camera 400 is composed of an image pickup optical system 401 and an image pickup element 402. The image sensor 402 has two photoelectric conversion units in one pixel. As shown in FIG. 5B, the pixel of the image sensor 402 is composed of a microlens 403, a first photoelectric conversion unit 404, and a second photoelectric conversion unit 405. The photoelectric conversion unit and the pupil of the imaging optical system 401 are in an optically conjugate relationship via the microlens 403, and the pupil of the imaging optical system is divided to acquire parallax images from different viewpoints. There is. The image obtained by using the image signal from the first photoelectric conversion unit 404 is an image with the second entrance pupil 143 in the first embodiment as a viewpoint, which corresponds to a part of the pupil region of the imaging optical system 401. It is an image corresponding to. Further, the image obtained by using the image signal from the second photoelectric conversion unit 405 of each pixel corresponds to the other part of the pupil region of the imaging optical system 401, and the first incident in the first embodiment. The image is based on the pupil 133. Further, the image obtained by adding the image signal from the first photoelectric conversion unit 404 of each pixel and the image signal from the second photoelectric conversion unit 405 of each pixel has the incident pupil 406 as a viewpoint. It becomes an image.

As described above, in the present embodiment, a pair of parallax images can be acquired by one imaging optical system 401 and one imaging element 402, so that the present embodiment is different from the first embodiment using a plurality of cameras. It is possible to construct a more mechanically stable image pickup system with a short baseline length, and it is possible to stably obtain high distance measurement accuracy with a simpler configuration.

Furthermore, by acquiring a normal image captured without projecting a pattern, it is possible to obtain measured values from the same viewpoint as the normal image, and by adding it to the image as meta information or using it for recognition, the recognition accuracy is improved. It can be used as an image pickup device.

[Third Embodiment]
Next, a third embodiment will be described with reference to FIGS. 6A and 6B. Each process of the flow of FIG. 6A is also carried out by the control unit 151 of the measuring device 150 or by operating each block under the control of the control unit 151. FIG. 6A is a processing flow, and the pattern is projected and imaged in step S1511. FIG. 6B is an example of an image from the viewpoint of the first entrance pupil 133 acquired by the imaging system 120 by projecting a pattern with the projection device 110 using the measurement system 100 shown in FIG. 5A. The pattern to be projected is a striped pattern in which line segments extending in a direction perpendicular to the straight line connecting the first entrance pupil 133 and the second entrance pupil 143 of the imaging system 120 are arranged substantially periodically in the direction of the straight line. It is a pattern. The subject is a plane and a hemispherical object placed on the plane.

In step S1512, the distance shape calculation unit 153 calculates the primary distance information of the subject from the parallax image captured by the passive stereo method.

Next, in step S1513, the distance shape calculation unit 153 generates surface normal information from the stripe spacing of the striped pattern image in any of the parallax images. The stripe spacing of the striped pattern image represents the inclination of the surface in the periodic direction of the stripes, and is the same as the stripe spacing of the projection pattern when the surfaces face each other in the observation direction, and the stripe spacing becomes the same when the surface normal is inclined. It gets narrower. As shown in FIG. 6B, for example, the stripes in the plane region 210 are straight and evenly spaced. On the other hand, the fringes in the curved region 211 become curved, and the fringe spacing changes depending on the location.

In step S1513, the distance shape calculation unit 153 obtains the local fringe interval for each place in the image, and thereby, the surface normal information at the local position of the subject can be acquired. Further, in step S1514, the distance shape calculation unit 153 integrates the primary distance information and the surface normal information to generate highly accurate distance shape information.

Then, in step S1515, the control unit 151 stores the calculated high-precision distance shape information in a storage unit (not shown), or outputs it to an external device via a recording unit, a communication unit, or the like.

The primary distance information obtained by the passive stereo method and the surface normal information using the fringe spacing are in a differential / integral relationship. That is, the integral of the surface normal and the distance information, or the derivative of the surface normal and the distance information are essentially the same. Therefore, the distance shape calculation unit 153 can compare these and reduce the variation by removing or averaging the obvious error values. Alternatively, variation can be suppressed by using information obtained by integrating the surface normals for the flat surface portion and using distance information for the step amount in the stepped portion. By integrating the primary distance information and the surface normal information in this way, it is possible to acquire the distance shape information with higher accuracy than the distance shape information obtained by a single measurement method. The order of steps S1512 and S1513 may be interchanged.

At this time, by additionally performing the process of step S1516 as shown in FIG. 7A, when the pattern used when obtaining the primary distance information by using the passive stereo method and the surface normal information are acquired from the stripe spacing. The pattern used may be different. Each process of the flow of FIG. 7A is also carried out by the control unit 151 of the measuring device 150 or by operating each block under the control of the control unit 151. In step S1516, the control unit 151 gives an instruction to project a pattern to the projection device 110 via the communication unit, and also instructs the image pickup system 120 to take an image in a state where the pattern is projected to the image pickup system 120 via the communication unit. A pair of parallax images obtained by imaging from the above are newly acquired. Here, it is desirable that the stripe spacing is narrower when the surface normal information is acquired in step S1516 than when the pattern used when obtaining the primary distance information in S1511 is obtained. As a result, spatially dense surface normal information can be acquired. When obtaining the surface normal information, the orthogonal striped patterns may be switched and projected, or the grid pattern or the dotted line pattern obtained by synthesizing the orthogonal striped patterns may be projected. With this configuration, the normals in the orthogonal axial directions can be obtained, so that the surface normal information of the subject can be acquired with higher accuracy.

Further, the pattern at the time of acquiring the surface normal information may be selected based on the primary distance information. For example, if the distance from the projection device is far, the projection pattern image becomes coarse, and if it is close, the projection pattern image becomes fine. Therefore, the pattern size is controlled so as to have the stripe spacing required for acquiring surface normal information according to the distance. .. In addition, since the direction in which the surface is tilted can be roughly understood from the primary distance information, if a striped pattern in the direction orthogonal to the tilted direction is selected and projected, or a fine pattern is projected, the amount of tilt of the surface ( The amount of inclination of the surface normal) can be obtained with high accuracy.

Further, the pattern used in the passive stereo method may be selected based on the surface normal information. If the surface normal is tilted, the striped pattern image becomes too coarse, the parallax images cannot correspond to each other, and the distance measurement accuracy is lowered. Therefore, the projection pattern is made finer on the tilted surface. As a result, distance information can be acquired with high accuracy. As a result, it is possible to project in a pattern suitable for the shape of the subject, so that the distance information can be acquired with higher accuracy.

Further, the pattern used when acquiring the surface normal information is a striped shape in which line segments extending in a direction perpendicular to a straight line connecting the viewpoints of the projection device 110 and the imaging system 120 are arranged substantially periodically in the straight line direction. It is desirable that the pattern is. With this configuration, the change in the stripe spacing with respect to the surface normal becomes large, and the surface normal can be detected with high accuracy.

Further, as in the example shown in FIG. 7B, the distance information acquired by the pattern projection stereo method may be used. Each process of the flow of FIG. 7B is also carried out by the control unit 151 of the measuring device 150 or by operating each block under the control of the control unit 151.

In step S1517, the relative position calculation unit 154 calculates the position and orientation of the projection device 110 with respect to the position of the imaging system 120 from the primary distance information of the subject and the position of the pattern image on the parallax image.

Next, in step S1518, the distance shape calculation unit 153 uses the pattern projection stereo method to secondary the subject from the calculated position / orientation of the projection device 110, the position / orientation of the imaging system 120, and the position of the pattern image on the parallax image. Calculate the distance information.

In step S1514, the distance shape calculation unit 153 compares the primary distance information, the secondary distance information, and the surface normal information, and removes and averages the error values to obtain more stable and highly accurate distance shape information. Can be obtained. As described above, in the present embodiment, more accurate distance shape information can be obtained by integrating the primary distance information obtained by the passive stereo method and the surface normal information using the stripe spacing to calculate one distance shape information. Can be generated.

[Fourth Embodiment]
In the present embodiment, the boundary region of the subject is extracted by analyzing the continuity of the stripes in the image based on the stripe pattern projected by the projection device 110, and the information is used to obtain the distance shape information with high accuracy. To get. The processing flow according to this embodiment will be described with reference to FIG. 8A. Each process of the flow of FIG. 8A is also carried out by the control unit 151 of the measuring device 150 or by operating each block under the control of the control unit 151.

In step S1521, the control unit 151 gives an instruction to project the striped pattern to the projection device 110 via the communication unit, and also instructs the imaging system 120 to take an image with the striped pattern projected via the communication unit. , Acquire a pair of parallax images obtained by imaging from the imaging system 120.

Next, in step S1522, the distance shape calculation unit 153 generates boundary portion information of the subject. In FIG. 6B, the stripes in the flat region 210 have a constant line direction, and the stripes in the curved region 211 have a changing line direction, but the change is gradual. On the other hand, in the region (boundary portion) 212 where the change in distance is large, the direction of the line changes rapidly. FIG. 8B is an example of an image of a subject composed of a flat surface and a flat object placed at a position separated from each other in the depth direction on the front side, and a striped line at the boundary portion 222 of the subject having a step in the depth direction. Is interrupted and becomes discontinuous. By analyzing the continuity of the fringes in this way, it is possible to extract a region where the change in distance is large, that is, a boundary portion of the subject.

In step S1523, the distance shape calculation unit 153 calculates the primary distance information of the subject from the parallax image captured by the passive stereo method.

Next, in step S1524, the distance shape calculation unit 153 generates highly accurate distance shape information from the primary distance information using the boundary portion information of the subject. In principle, the stereo method is less accurate in the boundary region where the distance change is large. Therefore, for the extracted boundary region, the distance shape information is created by extending the primary distance information in each region sandwiching the boundary region to the boundary portion without using the primary distance information obtained by the stereo method. To do. For example, in the example shown in FIG. 8B, the primary distance information of the region 221 is used on the region 221 side across the boundary portion 222, and the primary distance information of the region 220 is used on the region 220 side. In the example shown in FIG. 6B, the primary distance information of the region 210 is used on the region 210 side across the boundary portion 212, and the primary distance information on the region 211 side so that the surface normals in the region 211 change continuously. To generate. In this way, by extracting the boundary region of the subject based on the shape of the stripes and obtaining the distance information based on the boundary region, it is possible to obtain the distance shape information with high accuracy even at the boundary portion where the distance change is large. At this time, by using the secondary distance information obtained by the pattern projection stereo method and the surface normal information obtained from the stripe spacing, it is possible to obtain the distance shape information of the subject with higher accuracy.

Then, in step S1525, the control unit 151 stores the calculated high-precision distance shape information in a storage unit (not shown), or outputs it to an external device via a recording unit, a communication unit, or the like.

As described above, in the present embodiment, the boundary region of the subject is extracted by projecting the striped pattern light by the projection device 110 and analyzing the continuity of the stripes in the image including the subject and the pattern light. Distance shape information can be acquired with higher accuracy by using the information of the boundary region.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100 Measuring system 110 Projection device 120 Imaging system 130 First camera 140 Second camera 150 Measuring device

Claims (24)

In a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures image groups from at least two different viewpoints.
The distance between the viewpoints is shorter than the distance between the projection device and the viewpoints.
It has a relative position calculation means for obtaining a relative position of the projection device with respect to at least one of the viewpoints from the pattern image position on the image group and the positional relationship between the viewpoints.
A measuring device characterized in that distance information of the subject is acquired from the relative position and the pattern image position on the image at the viewpoint. The measuring device according to claim 1, wherein the projection device has a variable position and posture. The measuring device according to claim 1 or 2, wherein the projection device is arranged at a position on an extension line of a straight line connecting the two viewpoints. The measuring device according to any one of claims 1 to 3, wherein the pattern used when obtaining the relative position and the pattern used when acquiring the distance information are different. Any of claims 1 to 4, wherein the pattern is a striped pattern in which line segments extending in a direction perpendicular to a straight line connecting the two viewpoints are arranged substantially periodically in the direction of the straight line. The measuring device according to item 1. The measuring device according to claim 5, wherein at least one line segment of the striped pattern is different in color and / or shape from the other line segments. The fifth or sixth aspect of the present invention, wherein the surface normal information at the local position of the subject is acquired by using the local stripe spacing of the pattern image on the image of the striped pattern. Measuring device. The measuring device according to claim 7, wherein the distance information of the subject is acquired from the surface normal information and the pattern image position on the image at the relative position and the viewpoint. Obtaining the distance information of the subject from the surface normal information, the pattern image position on the image at the relative position and the viewpoint, the distance between the viewpoints, and the pattern image position on the image with respect to each viewpoint. The measuring device according to claim 7 or 8. The pattern used when acquiring the surface normal information and the pattern used when acquiring the pattern image position are different, and the pattern used when acquiring the surface normal information has a narrower stripe spacing of the pattern. The measuring device according to any one of claims 7 to 9, wherein the measuring device is characterized. The pattern is a pattern including a line segment.
According to claim 1, the boundary region of the subject is extracted based on the shape of the pattern image of the line segment on the image, and the distance information of the subject is acquired by using the information of the boundary region. The measuring device according to any one of 10. In a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures image groups from at least two different viewpoints.
The pattern includes a striped pattern in which line segments extending in a direction perpendicular to the straight line connecting the two viewpoints are arranged substantially periodically in the direction of the straight line.
Using the local stripe spacing of the pattern image on the image at least one viewpoint of the striped pattern, the surface normal information at the local position of the subject is acquired.
A measuring device characterized in that distance information of the subject is acquired from the surface normal information, the distance between the viewpoints, and the pattern image position on the image with respect to each viewpoint. The pattern used when acquiring the surface normal information and the pattern used when acquiring the pattern image position are different, and the pattern used when acquiring the surface normal information has a narrower stripe spacing of the pattern. The measuring device according to claim 12. The pattern used when acquiring the surface normal information is a striped pattern in which line segments extending in a direction perpendicular to a straight line connecting the projection device and the viewpoint of the imaging system are arranged substantially periodically in the direction of the straight line. The measuring device according to claim 12 or 13, wherein the measuring device includes a pattern. In a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures image groups from at least two different viewpoints.
The pattern is a pattern including a line segment.
A measuring device characterized in that a boundary region of a subject is extracted based on the shape of the pattern image of the line segment on the image, and distance information of the subject is acquired by using the information of the boundary region. The measuring device according to any one of claims 1 to 15, wherein the imaging system divides the pupil of one optical system and images a group of images from a plurality of viewpoints. The measuring device according to any one of claims 1 to 16 is included.
The imaging system is an imaging device characterized in that an image captured without projecting the pattern is also acquired. It is configured to include a projection device that projects a predetermined pattern on a subject, an imaging system that captures image groups from at least two different viewpoints, and a projection device and a measuring device connected to the imaging system. It is a measurement system
The distance between the viewpoints is shorter than the distance between the projection device and the viewpoints.
The measuring device is
It has a relative position calculation means for obtaining a relative position of the projection device with respect to at least one of the viewpoints from the pattern image position on the image group and the positional relationship between the viewpoints.
A measurement system characterized in that distance information of the subject is acquired from the relative position and the pattern image position on the image at the viewpoint. It is configured to include a projection device that projects a predetermined pattern on a subject, an imaging system that captures image groups from at least two different viewpoints, and a projection device and a measuring device connected to the imaging system. It is a measurement system
The pattern includes a striped pattern in which line segments extending in a direction perpendicular to the straight line connecting the two viewpoints are arranged substantially periodically in the direction of the straight line.
The measuring device is
Using the local stripe spacing of the pattern image on the image at least one viewpoint of the striped pattern, the surface normal information at the local position of the subject is acquired.
A measurement system characterized in that distance information of the subject is acquired from the surface normal information, the distance between the viewpoints, and the pattern image position on the image with respect to each viewpoint. It is configured to include a projection device that projects a predetermined pattern on a subject, an imaging system that captures image groups from at least two different viewpoints, and a projection device and a measuring device connected to the imaging system. It is a measurement system
The pattern is a pattern including a line segment.
The measuring device is characterized in that a boundary region of a subject is extracted based on the shape of the pattern image of the line segment on the image, and the distance information of the subject is acquired by using the information of the boundary region. Measurement system to do. It is a control method of a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures an image group from at least two different viewpoints.
The distance between the viewpoints is shorter than the distance between the projection device and the viewpoints.
It has a relative position calculation step of obtaining a relative position of the projection device with respect to at least one of the viewpoints from the pattern image position on the image group and the positional relationship between the viewpoints.
A control method characterized in that distance information of the subject is acquired from the relative position and the pattern image position on the image at the viewpoint. It is a control method of a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures an image group from at least two different viewpoints.
The pattern includes a striped pattern in which line segments extending in a direction perpendicular to the straight line connecting the two viewpoints are arranged substantially periodically in the direction of the straight line.
Using the local stripe spacing of the pattern image on the image at least one viewpoint of the striped pattern, the surface normal information at the local position of the subject is acquired.
A control method characterized in that distance information of the subject is acquired from the surface normal information, the distance between the viewpoints, and the pattern image position on the image with respect to each viewpoint. It is a control method of a measuring device having a projection device that projects a predetermined pattern on a subject and an imaging system that captures an image group from at least two different viewpoints.
The pattern is a pattern including a line segment.
A control method characterized in that a boundary region of a subject is extracted based on the shape of the pattern image of the line segment on the image, and distance information of the subject is acquired by using the information of the boundary region. A program for causing a computer to function as the measuring device according to any one of claims 1 to 16.

Images