JP6541070B2 - Three-dimensional information restoration apparatus and three-dimensional information restoration method - Google Patents

Description

  The present disclosure relates to a three-dimensional information restoration apparatus and a three-dimensional information restoration method.

  Conventionally, a stereo camera is known in which two imaging units are fixed to one case so that the same subject is imaged by the left and right imaging units. The stereo camera captures an image of a subject from a plurality of different directions, and records information in the plane direction and information (three-dimensional information) in the depth direction.

  For example, three-dimensional information restoration including an image input unit, a corresponding point detection unit, a basic matrix calculation unit, a translation calculation unit, a rotation calculation unit, and a distance calculation unit as an apparatus for restoring three-dimensional information An apparatus is known (see, for example, Patent Document 1).

  In this three-dimensional information restoration apparatus, the image input unit inputs two images capturing a three-dimensional rigid body. The corresponding point detection unit detects a corresponding point between two images. The basic matrix calculation unit calculates a basic matrix from a three-dimensional rotation matrix and a three-dimensional translation vector between two images. The translation calculation unit calculates a three-dimensional translation vector. The rotation calculator calculates a three-dimensional rotation matrix. The distance calculation unit calculates the distance between the camera and the corresponding point in the three-dimensional space.

Unexamined-Japanese-Patent No. 09-237341 gazette

  In the three-dimensional information restoration device described in Patent Document 1, when two imaging devices are separately installed to form a stereo camera system, the three-dimensional image of the target point is obtained from the left and right images captured by the two imaging devices. Coordinates may not be restored correctly.

  The present disclosure has been made in view of the above circumstances, and provides a three-dimensional information restoration apparatus and a three-dimensional information restoration method capable of improving the restoration accuracy of three-dimensional coordinates restored from two captured images.

The three-dimensional information restoration apparatus of the present disclosure comprises a port, an input device, and a processor. The port acquires a first image captured by the first imaging device and a second image captured by the second imaging device. The processor detects a plurality of first corresponding point pairs in which the first feature point in the first image and the second feature point in the second image correspond to a plurality of first corresponding point pairs. Based on the three-dimensional coordinates in which the first feature point is back-projected. The input device indicates the position in the first image and the second image of the second corresponding point pair, with any corresponding point pair in the first image and the second image as the second corresponding point pair. Position information and distance information indicating the distance from the first imaging device to the second corresponding point pair are input. The processor restores the three-dimensional coordinates of the second corresponding point pair based on the input position information of the second corresponding point pair , and inputs the three-dimensional coordinates of the restored second corresponding point pair. The accuracy of the three-dimensional coordinates is evaluated based on the distance information, and when the accuracy is less than a predetermined value, the first corresponding point pair is changed to recalculate the parameters for deriving the three-dimensional coordinates. , Recalculate restoration of three-dimensional coordinates of the input second corresponding point pair based on the recalculated parameter.

  According to the present disclosure, the restoration accuracy of three-dimensional coordinates restored from two captured images can be improved.

Schematic diagram showing a schematic configuration example of a stereo camera system in the first embodiment Block diagram showing a configuration example of a PC in a stereo camera system (A), (B) and (C) are schematic diagrams for explaining an example of parameters for deriving three-dimensional coordinates A diagram for explaining an example of the outline of the operation of a stereo camera system Flow chart showing an example of initial calibration performed at the time of initial setting The figure which shows the example of a screen where the several corresponding point used as a candidate for specifying a corresponding point, and the corresponding line which connects these corresponding points were drawn in the image on either side. The figure which shows the example of a screen where the error message of the corresponding point and the mark were added in the screen of FIG. The figure which shows the example of a screen where guidance was added so that a distant point may be specified as a corresponding point in the screen of FIG. Block diagram showing a configuration example of a PC in the second embodiment Flow chart showing an example of recalibration performed during operation 10 is a flowchart showing an example of recalibration performed at the time of operation following FIG. 10 Diagram showing an example screen where template matching is performed during recalibration Figure showing an example screen with guidance for corresponding point re-searching added Flow chart showing another example of the initial calibration performed at the time of initial setting Flow chart showing another example of recalibration performed during operation

  Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, the detailed description may be omitted if necessary. For example, detailed description of already well-known matters and redundant description of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. It is to be understood that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and they are not intended to limit the claimed subject matter.

First Embodiment
[Configuration etc.]
FIG. 1 is a schematic view showing a schematic configuration example of a stereo camera system 5 in the first embodiment. The stereo camera system 5 includes, for example, a first camera 10, a second camera 11, and a PC (Personal Computer) 20. The first camera 10 and the second camera 11 are connected to, for example, the PC 20 through the cables 18A and 18B, respectively. The first camera 10 and the second camera 11 are connected, for example, via a cable 18C.

  The stereo camera system 5 is an example of a three-dimensional information restoration system. The PC 20 is an example of a three-dimensional information restoration apparatus. The first camera 10 and the second camera 11 are an example of an imaging device.

  Although not illustrated, the first camera 10 and the second camera 11 include, for example, an image sensor for capturing an image including a subject, and an output device (for example, communication device) for transmitting image data of the captured image to the PC 20 . As an imaging device, a surveillance camera, an in-vehicle camera, an industrial camera, a medical camera, and a consumer camera are mentioned, for example.

  For example, in response to an image acquisition request from the PC 20, the first camera 10 captures a first image (a first camera image, for example, an image on the left side) of a predetermined scene including a subject and captures the first image Transfer the image data to the PC 20.

  The second camera 11 captures, for example, a second image (second camera image, for example, an image on the right side) of a predetermined scene including the subject according to an image acquisition request from the PC 20 and a synchronization signal from the first camera 10 The image data of the imaged second image is transferred to the PC 20. That is, the second camera 11 images the same subject included in the same scene from a direction different from that of the first camera 10.

  The first camera 10 and the second camera 11 respectively have a first housing 13 and a second housing 14 and are, for example, fixed cameras fixed to a ceiling, a wall, and other positions. The first camera 10 and the second camera 11 are PTZ cameras capable of panning, tilting and zooming. That is, the first camera 10 and the second camera 11 may be cameras capable of operating at least one of pan, tilt and zoom, or a fixed camera in which the pan direction, tilt direction and zoom magnification are fixed. May be

  In the first camera 10 and the second camera 11, the focal length, the optical axis coordinates, and the distortion correction coefficient are known. The first camera 10 and the second camera 11 can output an image obtained by performing distortion correction on a captured image based on, for example, a distortion correction coefficient. Therefore, the images captured by the first camera 10 and the second camera 11 may include the image after the distortion correction. The focal length, the optical axis coordinates, and the distortion correction coefficient may be changed not as fixed values but as variable values.

  The PC 20 receives image data from the first camera 10 and the second camera 11 through the cables 18A and 18B, and performs various image processing (for example, feature point extraction, corresponding point extraction, camera parameter estimation, three-dimensional coordinates, etc. described later) Calculation, accuracy evaluation of 3D coordinate calculation).

  FIG. 2 is a block diagram showing a configuration example of the PC 20 in the stereo camera system 5. The PC 20 has a processor 30, an input device 26, a memory 27, a port 28 and a display 31.

  The input device 26 inputs the designation of the corresponding point by the user and the depth information (the distance from the first camera 10 to the specified corresponding point). The input device 26 is configured to include a mouse, a keyboard and the like. The depth information input through the input device 26 is stored in the memory 27 and input to the accuracy evaluation unit 25.

  The memory 27 holds various data, information, and programs. The memory 27 includes, for example, coordinates of designated corresponding points (hereinafter referred to as designated corresponding points), distance information to designated corresponding points, peripheral images of designated corresponding points on the left and The first image, the second image) are stored. In addition, since corresponding points exist in pairs corresponding to each other in the left and right camera images, they can be said to be corresponding point pairs. Further, since the designated corresponding points are designated in pairs correspondingly in the left and right camera images, they can be said to be designated paired point pairs.

  The memory 27 is configured to include, for example, a memory such as a random access memory (RAM) or a read only memory (ROM), and a storage such as a hard disk drive (HDD) or a solid state drive (SSD).

  The port 28 is communicably connected to the first camera 10 and the second camera 11, transmits an image acquisition request to the first camera 10 and the second camera 11, and transmits a request from the first camera 10. The image data of one image and the image data of the second image transmitted from the second camera 11 are received. The port 28 includes, for example, a communication port for communicating with an external device, and an external device connection port for connecting an external device.

  The display 31 displays the first image captured by the first camera 10 and the second image captured by the second camera 11, and superimposes these images on them to display corresponding points, error messages, guidance, etc. Also, a three-dimensional restored image is displayed. The display 31 is configured to include a display device such as a liquid crystal display.

  The input device 26 and the display 31 may be configured as separate devices, or may be configured as a touch panel in which these are integrated.

  The processor 30 executes the program stored in the memory 27 to perform each function of the feature point extraction unit 21, the corresponding point detection unit 22, the camera parameter estimation unit 23, the three-dimensional coordinate calculation unit 24, and the accuracy evaluation unit 25. To achieve.

  The processor 30 is configured to include, for example, a central processing unit (CPU), a digital signal processor (DSP), or a graphical processing unit (GPU).

  The feature point extraction unit 21 sequentially acquires and analyzes the first image captured by the first camera 10 input from the port 28 and the second image captured by the second camera 11.

  The first camera 10 is, for example, a camera disposed on the left side of FIG. 1 for capturing a left camera image of an object. The second camera 11 is, for example, a camera disposed on the right side of FIG. 1 for capturing a right camera image for an object.

  The feature point extraction unit 21 has a function as an image acquisition unit, and sequentially detects feature points (for example, points in an area where the edge is strong) from the acquired left camera image and right camera image. For detection of feature points, for example, an algorithm that extracts local feature points invariant to enlargement, reduction, or rotation of an image is used. This algorithm includes, for example, Scale-Invariant Feature Transform (SIFT) and Speed-Up Robust Features (SURF).

  FIGS. 3A, 3B, and 3C are schematic diagrams for explaining an example of parameters for deriving three-dimensional coordinates. The three-dimensional coordinates indicate coordinates in a three-dimensional space when the position of the first camera 10 is the origin (0, 0, 0). The three-dimensional coordinates indicate the coordinates of the target point 41 on which a predetermined point included in the first image or the second image is back-projected (three-dimensional restoration). Parameters (also referred to simply as parameters) for deriving three-dimensional coordinates include, for example, feature points, pairs of corresponding points, and the position and orientation of a camera.

  The feature point extraction unit 21 detects feature points a1 to a7 from the first image GZ1 and detects feature points b1 to b7 from the second image GZ2, as shown in FIG. 3A, for example. When it is not necessary to distinguish the feature points a1 to a7 in particular, they are simply referred to as the feature point a. Similarly, the feature points b1 to b7 are simply referred to as a feature point b when it is not necessary to distinguish them in particular. The number of feature points is taken into account, for example, when estimating the position and orientation of the camera. As the number of feature points increases, estimation accuracy for estimating the position and orientation of the second camera 11 with respect to the first camera 10 increases.

  For example, as illustrated in FIG. 3B, the corresponding point detection unit 22 sequentially sets feature points with high similarity included in the first image GZ1 and the second image GZ2 as a corresponding point pair (an example of the corresponding points). It detects and outputs corresponding point pair information (image coordinate pair information).

  The corresponding point pair information includes, for example, information in which a feature point in the first image GZ1 as a corresponding point and a feature point in the second image GZ2 are associated (paired). If the similarity is high, for example, the window angles are similar at the feature point in the first image GZ1 and the feature point in the second image GZ2 included in the corresponding points, and the window angle difference is less than a predetermined angle Including a certain thing.

  The corresponding point detection unit 22 detects corresponding points by, for example, a known technique (for example, the technique described in Patent Document 1).

  In FIG. 3B, the feature point a1 included in the first image GZ1 and the feature point b1 included in the second image GZ2 are detected as corresponding points, and corresponding point pair information is output. Further, the feature point a2 included in the first image GZ1 and the feature point b2 included in the second image GZ2 are detected as corresponding points, and are output as corresponding point pair information. Similarly, the feature points a3 to a7 and the feature points b3 to b7 are detected as the corresponding points, and the corresponding point pair information is output.

  The corresponding point pair information is associated in FIG. 3B with lines (corresponding lines) c1 to c7 connecting the feature points a1 to a7 and the feature points b1 to b7, respectively. The number of corresponding point pairs is taken into account, for example, when estimating the position and orientation of the camera. As the number of corresponding point pairs increases, the estimation accuracy of estimating the position and orientation of the second camera 11 with respect to the first camera 10 increases.

  As shown in FIG. 3C, the camera parameter estimation unit 23 has, for example, corresponding point pair information, the focal length of the first camera 10, the focal length of the second camera 11, the optical axis coordinates of the first camera 10, The position and posture of the second camera 11 with respect to the first camera 10 are sequentially estimated based on the optical axis coordinates of the second camera 11.

  The optical axis coordinates indicate coordinates corresponding to the center of the lens in the captured image. The position of the second camera 11 with respect to the first camera 10 is indicated by, for example, a translation vector p. The attitude of the second camera 11 with respect to the first camera 10 is indicated by, for example, a rotation matrix R.

  The camera parameter estimation unit 23 estimates camera parameters (for example, translation vector p, rotation matrix R) by a known technique (for example, the technique described in Patent Document 1). The translation vector p corresponds to the three-dimensional translation vector described in Patent Document 1. The rotation matrix R corresponds to the three-dimensional rotation matrix described in Patent Document 1. The translation vector p is represented, for example, by (Expression 1).

ｐｘ，ｐｙ，ｐｚは、それぞれＸ軸，Ｙ軸，Ｚ軸方向の並進ベクトル成分を表す。

px, py and pz respectively represent translational vector components in the X-axis, Y-axis and Z-axis directions.

The rotation matrix R is expressed, for example, by (Expression 2).
R = R (θz) · R (θy) · R (θx) (Equation 2)
θz, θy, and θx represent rotation angles (radians) around the Z axis, Y axis, and X axis, respectively, and represent components (rotation components) of the rotation angles of the respective axes. R (θz), R (θy), and R (θx) represent components of the rotation matrix R of the Z axis, the Y axis, and the X axis, respectively. The rotation angle θ = 0 indicates that the first camera 10 and the second camera 11 are parallel to each other.

  The three-dimensional coordinate calculation unit 24 determines the three-dimensional shape of the target point 41 based on the corresponding point pair information, the internal parameters and external parameters of the first camera 10, the internal parameters and external parameters of the second camera 11, and the baseline length 62 Coordinates (X, Y, Z) are sequentially calculated (see FIG. 3C). The corresponding point pair information includes position coordinates (x, y) on the screen designated in the first image GZ1 and the second image GZ2 displayed on the screen of the display 31. The target point 41 is a point obtained by back-projecting the feature points of the first image GZ1 included in the corresponding point pair. The three-dimensional coordinate calculation unit 24 calculates three-dimensional coordinates of the target point 41 by, for example, a known technique (for example, the technique described in Patent Document 1).

  The internal parameters include, for example, focal lengths of the first camera 10 and the second camera 11, optical axis coordinates, aspect ratio, and skew distortion. The external parameters include, for example, the position (three components of the X axis, Y axis and Z axis) and the posture (three rotation components along the X axis, Y axis and Z axis) of the second camera 11 with respect to the first camera 10 . The baseline length 62 is the distance between the first camera 10 and the second camera 11. The internal parameters and external parameters are determined, for example, for each camera.

  The internal parameters are held in advance by a memory (not shown) of each camera. The external parameters are sequentially derived and held in the memory 27. The baseline length 62 is previously held by, for example, a memory (not shown) of at least one of the first camera 10 and the second camera 11. Information on internal parameters and baseline length 62 is obtained from the camera via port 28 and held in memory 27.

  The accuracy evaluation unit 25 evaluates the accuracy of the three-dimensional coordinates calculated by the three-dimensional coordinate calculation unit 24. This accuracy evaluation is performed using the Z coordinate of the corresponding point (specified corresponding point) designated by the user via the input device 26.

  For example, the accuracy evaluation unit 25 inputs, from the input device 26, the Z coordinate of the designated corresponding point, that is, the distance (depth value) from the first camera 10 to the designated corresponding point am. The accuracy evaluation unit 25 compares the input distance with the Z coordinate (distance) of the designated corresponding point am calculated by the three-dimensional coordinate calculation unit 24. The accuracy evaluation unit 25 determines that the calculation result of the three-dimensional coordinates is successful if the error (difference) is equal to or less than a designated value (for example, 10%) as a result of comparison, and if it exceeds the designated value, three-dimensional It is determined that the calculation result of the coordinates is a failure. The specified value can be set to any value by the user.

  In addition, here, as an example of the distance information from the first camera 10 to the designated corresponding point am, the value of the Z coordinate (depth value) is used. Instead of this, for example, when the first camera 10 and the designated corresponding point am are not on the same Z-axis, an actual distance between the first camera 10 and the designated corresponding point am may be used as distance information.

  If the calculation result of the three-dimensional coordinates is a failure, the camera parameter estimation unit 23 again randomly selects a predetermined number of corresponding points different from the previous one among the many corresponding points detected by the corresponding point detection unit 22. Select to The three-dimensional coordinate calculation unit 24 recalculates the Z coordinate of the designated corresponding point using the predetermined number of corresponding points selected. The accuracy evaluation unit 25 performs the same determination as described above using the Z coordinate value of the designated corresponding point calculated again. On the other hand, when the determination result is unsuccessful a predetermined number of times, the user may designate another corresponding point and perform an input operation of depth information.

[Operation example]
Next, an operation example of the stereo camera system 5 will be described.

  FIG. 4 is a diagram for explaining the outline of the operation of the stereo camera system 5. The display 31 displays a first image GZ1 captured by the first camera 10 and a second image GZ2 captured by the second camera 11. For the first image GZ1 and the second image GZ2 displayed on the screen of the display 31, when the user operates the input device 26 to move the cursor SL to the vicinity of the corresponding point and then clicks it, the frames fr1 and fr2 are displayed. The enclosed area is enlarged on the screen.

  When the images (hereinafter referred to as peripheral images) GZ11 and GZ12 near the corresponding points are enlarged and displayed on the screen, the user can easily specify the corresponding points included in the images. For example, the user specifies the corresponding point am included in the enlarged peripheral image GZ11 via the input device 26, and specifies the corresponding point bm included in the enlarged peripheral image GZ12. Furthermore, the user inputs depth information (the distance from the first camera 10 to the corresponding point am) via the input device 26. The depth information may be the distance from the second camera 11 to the corresponding point bm. This depth information is, for example, an actual value measured using a laser range finder or a measure. In FIG. 4, “102 m” is input as the actual measurement value, and displayed on the screen of the display 31 so as to overlap the enlarged peripheral image GZ11.

  When the user finishes designation and input, the processor 30 performs three-dimensional reconstruction processing based on these pieces of information. As a result of this processing, the display 31 displays a three-dimensional restored image.

  FIG. 5 is a flowchart showing an example of the initial calibration performed at the time of initial setting.

  When the initial calibration starts, first, the feature point extraction unit 21 sends an image acquisition request to the first camera 10 and the second camera 11 through the port 28, and the feature point extraction unit 21 receives the request from the first camera 10 and the second camera 11 respectively. The image data of the first image GZ1 and the second image GZ2 are captured (S1). Note that the image data may be taken in periodically without an image acquisition request being made.

  The input device 26 receives a point on the image designated by the user as a corresponding point for the first image GZ1 and the second image GZ2 captured in S1 (S2).

  When the user clicks on the image, the processor 30 enlarges and displays the peripheral image around the point on the image displayed on the display 31 designated by the cursor SL (S3).

  The input device 26 receives corresponding points (designated corresponding points) am and bm designated by the user with respect to the enlarged peripheral images GZ11 and GZ12 (S4). By specifying the corresponding points am and bm, information (an example of position information) of the coordinates of the corresponding points am and bm can be obtained. The designated corresponding point may be a point included in a plurality of corresponding points detected by the corresponding point detection unit 22 or a point not included.

  The input device 26 receives the distance information (depth value) from the first camera 10 to the designated corresponding point am input by the user (S5).

  The processor 30 inputs the coordinates of the designated corresponding point, the distance information, the peripheral images GZ11 and GZ12, and the first image GZ1 and the second image GZ2 (left and right camera images) when the corresponding points are designated, which are input from the input device 26. Are stored (saved) in the memory 27 (S6).

  On the other hand, in parallel with the processes of S2 to S6, the feature point extraction unit 21 extracts a plurality of feature points from the first image GZ1 and the second image GZ2, respectively. The corresponding point detection unit 22 detects a plurality of (for example, about 100) corresponding points representing the correspondence between the first image GZ1 and the second image GZ2 from the similarity of the feature points extracted by the feature point extraction unit 21. Then, corresponding point pair information (image coordinate pair information) representing this correspondence is output (S7).

  In FIG. 6, a plurality of corresponding points a11 to a15 and b11 to b15 which become candidates for specifying the corresponding points am and bm, and corresponding lines c11 to c15 connecting these corresponding points are left and right images (first image GZ1, It is a figure which shows the screen drawn in 2nd image GZ2).

  The display 31 displays a first image GZ1, a second image GZ2, corresponding points a11 to a15, b11 to b15, and corresponding lines c11 to c15.

  The corresponding point detection unit 22 randomly selects a predetermined number (for example, approximately 5) of corresponding points from among a plurality of detected (for example, approximately 100) corresponding points (S8). In this selection of the corresponding points, for example, in the evaluation of the accuracy by the accuracy evaluation unit 25 described later, the corresponding points used when the error is determined to exceed the designated value are excluded.

  The camera parameter estimation unit 23 includes corresponding point pair information of corresponding points selected from among a plurality of corresponding points, the focal length of the first camera 10, the focal length of the second camera 11, the optical axis coordinates of the first camera 10, And the position and attitude of the second camera 11 with respect to the first camera 10 based on the optical axis coordinates of the second camera 11 (S9).

  The three-dimensional coordinate calculation unit 24 calculates corresponding point pair information, internal parameters and external parameters of the first camera 10, internal parameters and external parameters of the second camera 11 at one designated corresponding point (designated corresponding points) am and bm. The three-dimensional coordinates (X, Y, Z) of the target point corresponding to the designated corresponding point am are calculated based on the parameters and the baseline length 62, and the depth value (Z coordinate value) to the target point is determined. (S10). As described above, this target point is a point obtained by back-projecting the feature points of the first image GZ1 included in the corresponding point pair.

  The accuracy evaluation unit 25 uses the depth value calculated in S10 and the depth value input in S5, and determines whether an error that is the difference between them is equal to or less than a specified value (for example, 10%) S11). If the error is equal to or less than the designated value, the accuracy evaluation unit 25 ends the present processing, assuming that the accuracy evaluation is appropriate and the calculation result of the three-dimensional coordinate is successful.

  On the other hand, when the error exceeds the designated value (is equal to or greater than the predetermined value) in S11, the accuracy evaluation unit 25 determines whether the number of accuracy evaluations exceeds the threshold (S12). If the number of accuracy evaluations does not exceed the threshold value, the processor 30 returns to the process of S8, selects a predetermined number of new corresponding points, and repeats the same process.

  On the other hand, when the number of accuracy evaluations exceeds the threshold value, the processor 30 causes the screen of the display 31 to display an error message ms1, an error message ms1 prompting the user to input the position of another corresponding point, and the position of the corresponding point. It is displayed (S13).

  FIG. 7 is a view showing a screen to which the error message ms1 of the corresponding point and the mark MK are added in the screen of FIG. On the screen of display 31, for example, the error message ms1 (Warning "The measured value of the specified corresponding point has exceeded the specified error. Please check the corresponding point position or specify another corresponding point." Information is displayed. Further, in FIG. 7, the corresponding points a11 and b11 are set to the specified corresponding points am and bm, and on the corresponding line cm connecting the corresponding points am and bm, a mark MK (for example, x mark) indicating an error Is displayed.

  Here, it has already been confirmed by the present inventors that the accuracy of the restoration of the three-dimensional information is improved by designating a distant corresponding point corresponding to a target point far from the first camera 10. Based on this fact, the processor 30 displays on the screen of the display 31 a guidance ms2 prompting to designate a distant point (S14).

  FIG. 8 is a diagram showing a screen to which a guidance ms2 is added so as to designate a distant point on the screen of FIG. 6 as a corresponding point. On the screen of the display 31, for example, guidance ms2 (guidance information) of "Please specify a point as far as possible" is displayed. Furthermore, on the screen of the display 31, rectangular frames pf1 and pf2 surrounding the distant corresponding point are displayed.

  In this manner, it is prompted to specify a distant corresponding point included in the rectangular frames pf1 and pf2 which is expected to improve the accuracy of the three-dimensional information restoration. Therefore, the user can easily designate an appropriate corresponding point as a candidate in the rectangular frames pf1 and pf2. The process of S14 is an optional process and can be omitted. After that, the processor 30 returns to the processes of S2 and S7 and performs the same process.

  When it is determined that the accuracy evaluation is appropriate, the three-dimensional coordinate calculation unit 24 determines corresponding point pair information, internal parameters and external parameters of the first camera 10, internal parameters and external parameters of the second camera 11, and Based on the baseline length 62, three-dimensional reconstruction processing is performed in which three-dimensional coordinates (X, Y, Z) of the target point are sequentially calculated. The processor 30 displays the target point subjected to the three-dimensional restoration processing on the screen of the display 31.

  According to the process of FIG. 5, in S8, a predetermined number of corresponding points are randomly selected from the detected corresponding points, and the information of the selected corresponding points is used to select the second camera 11 for the first camera 10. Since the position and orientation are estimated, the estimation results of the position and orientation are different every time the corresponding points are selected. Therefore, the estimation accuracy of the position and orientation will be different. Therefore, depending on the selection of the corresponding points, the accuracy of the three-dimensional reconstruction may be low or high.

  On the other hand, the PC 20 uses the corresponding points when the estimation accuracy of the position and orientation is equal to or higher than the predetermined reference, that is, when the error between the measured value and the calculated value of the distance information is equal to or less than the predetermined The dimensional reconstruction can stably improve the reconstruction accuracy of three-dimensional coordinates.

[Effects, etc.]
Temporarily, when not inputting the position information of arbitrary corresponding points among a plurality of corresponding points or the distance information from the camera to the arbitrary corresponding points, that is, all the corresponding points are automatically detected, and all the corresponding points When the distance information related to is calculated by calculation, the estimation accuracy of the three-dimensional restoration may be low. For example, even if the corresponding point can be detected, the derivation accuracy of the distance information of the corresponding point may be low.

  On the other hand, in the PC 20 of the stereo camera system 5 according to the first embodiment, the port 28 includes the first image GZ1 captured by the first camera 10 and the second image GZ2 captured by the second camera 11. get. The processor 30 detects a plurality of corresponding points including corresponding points am and bm in which the feature point a in the first image GZ1 and the feature point b in the second image GZ2 correspond to each other. The processor 30 restores the three-dimensional coordinates in which the feature point a is back-projected based on the plurality of corresponding points am and bm. The input device 26 is a coordinate (an example of position information) of the corresponding point am designated by the user from among the plurality of corresponding points, and a depth value (distance information from the first camera 10 to the corresponding point am input by the user Example) and. The processor 30 recalculates restoration of three-dimensional coordinates after inputting coordinates and depth values of the corresponding points am.

  The corresponding points am designated by the input device 26 may or may not be included in the plurality of detected corresponding points. That is, the input device 26 sets any corresponding point pair in the first image GZ1 and the second image GZ2 as the specified corresponding point pair, and the position information in the first image GZ1 and the second image of the specified corresponding point pair, and the first And distance information indicating a distance from the camera 10 to the designated corresponding point pair.

  Thus, based on the position coordinates of the corresponding point (an example of the second corresponding point) designated by the user and the depth value from the first camera 10 to the designated corresponding point input by the user, the designation correspondence is provided. Recalculate the reconstruction of the 3D coordinates of the point. Therefore, the PC 20 can evaluate the accuracy of the restored three-dimensional coordinates, and can improve the restoration accuracy of the three-dimensional coordinates restored from the two captured images. In addition, since the number of corresponding points to be specified is small (for example, one point), the restoration accuracy of three-dimensional coordinates can be improved without making the user operation complicated.

  Further, the display 31 may display candidates of a plurality of corresponding points a11 to a15 and b11 to b15 by overlapping with at least one of the first image GZ1 and the second image GZ2 under the control of the processor 30.

  As a result, the user can easily designate corresponding points from among the plurality of corresponding points displayed on the display 31. Therefore, the operability of the PC 20 is improved.

  Further, the display 31 is a control under the control of the processor 30 to prompt the user to designate a corresponding point prior to at least one of the first image GZ1 and the second image GZ2 at a position where the distance from the first camera 10 is long. (An example of guidance information) ms2 may be displayed.

  As a result, the user is prompted to specify a distant corresponding point that is expected to improve the accuracy of three-dimensional coordinate recovery, and it becomes easy to specify an appropriate corresponding point from among a plurality of candidates.

  Further, the processor 30 is based on the depth value input by the input device 26 and the information (Z coordinate value) of the distance from the first camera 10 to the designated corresponding point am obtained by the restoration of the three-dimensional coordinates. The necessity of recalculation of restoration of three-dimensional coordinates may be determined.

  Thereby, for example, when the depth value to the input target point and the depth value to the restored target point are largely different, the PC 20 determines that the accuracy of the three-dimensional coordinate restoration is low, and the three-dimensional coordinate The processing load of recalculation can be reduced without recalculation of restoration of.

  Further, the display 31 is a difference between the depth value input by the input device 26 and the information of the distance from the first camera 10 to the designated corresponding point am obtained by the restoration of the three-dimensional coordinates under the control of the processor 30. If is greater than or equal to a predetermined value (specified value), an error message ms1 related to the input of the specified corresponding point is displayed. The error message ms1 is an example of the warning information.

  Thereby, the user can easily confirm the decrease in the restoration of the three-dimensional coordinates, and can smoothly input, for example, another corresponding point.

  In the present embodiment, FIG. 5 illustrates that the PC 20 performs the processes of S2 to S6 and the processes of S7 to S10 in parallel. The processes of S7 to S10 may be performed after the processes of S2 to S6, or the processes of S2 to S6 may be performed after the processes of S7 to S10.

Second Embodiment
The first embodiment shows the initial calibration performed at the time of initial setting of the stereo camera system. The second embodiment shows calibration performed at the time of operation, that is, recalibration performed after installing the stereo camera system. The recalibration is performed, for example, periodically, when the three-dimensional image to be restored becomes disturbed or when a natural disaster such as a typhoon or an earthquake occurs. Also in the second embodiment, it is assumed that the same initial calibration as in the first embodiment has already been performed.

  In the stereo camera system of the second embodiment, the description of the same components as in the first embodiment will be omitted or simplified by using the same reference numerals.

[Configuration etc.]
FIG. 9 is a block diagram showing a configuration example of the PC 20A in the second embodiment. The PC 20A includes the configuration of the PC 20 of the first embodiment, and further includes a reading unit 35 and a position searching unit 36.

  The processor 30A implements the functions of the reading unit 35 and the position searching unit 36 by executing the program stored in the memory 27.

  The reading unit 35 reads designation information (for example, designated corresponding point coordinates, distance information) and image information (for example, first image GZ1, second image GZ2, peripheral images GZ11 and GZ12) stored in the memory 27.

  The position search unit 36 searches the first image GZ3 and the second image GZ4 captured at the time of recalibration for whether or not the peripheral images GZ11 and GZ12 at the time of initial setting (or at the time of previous calibration) are present. . This search is performed, for example, by template matching in which the peripheral images GZ11 and GZ12 are used as templates and the first image GZ3 and the second image GZ4 are respectively moved to detect regions having high similarity.

[Operation etc.]
10 and 11 are flowcharts showing an example of recalibration performed at the time of operation.

  Similar to the process of S1 in the initial calibration, the port 28 takes in the image data of the first image GZ3 and the second image GZ4 from the first camera 10 and the second camera 11, respectively (S21).

  The reading unit 35 reads the designation information and the image information stored in the memory 27 (S22).

  The position search unit 36 performs template matching on the first image GZ3 and the second image GZ4 captured in S21, using the peripheral images GZ11 and GZ12 read by the reading unit 35 as templates. Then, the position search unit 36 searches for the area images PGZ3 and PGZ4 (see FIG. 12) that match the peripheral images GZ11 and GZ12 in the first image GZ3 and the second image GZ4 (S23).

  The position search unit 36 determines whether the template matching has succeeded (S24).

  FIG. 12 shows a screen on which template matching is performed at the time of recalibration. The processor 30A displays the template matching operation on the screen of the display 31. In the template matching, it is searched whether or not the peripheral image GZ11 and the peripheral image GZ12 are included in the first image GZ3 and the second image GZ4, respectively, as indicated by dotted frames g and h in the drawing. It is determined whether the peripheral image GZ11 and the peripheral image GZ12 match the area image PGZ3 included in the first image GZ3 and the area image PGZ4 included in the second image GZ2, respectively. If only one matches or both do not match, it is determined that the template matching has failed.

  If the template matching is successful, the position search unit 36 stores the matched area image PGZ3 and area image PGZ4 in the memory 27 as new peripheral image GZ11 and peripheral image GZ12, respectively, and the center coordinates of the area image PGZ3 and area image The central coordinates of PGZ 4 are stored as the coordinates of designated corresponding points am and bm, respectively (S 25).

  On the other hand, when the template matching fails in S24, the processor 30A displays on the screen of the display 31 a prompt for corresponding point search error and corresponding point re-input (S26).

  FIG. 13 is a diagram showing a screen to which guidance ms3 and the like for corresponding point re-search are added. On the screen of the display 31, for example, guidance ms3 (guidance information) of "corresponding point re-search error: please input the corresponding point position again with reference to the previous corresponding point designation image" is displayed.

  Further, the processor 30A reduces and displays, for example, the previous first image GZ1 and the second image GZ2 and the peripheral image GZ11 and the peripheral image GZ12 respectively including the corresponding points am and bm in the lower right corner of the screen of the display 31. You may do it (S27).

  The input device 26 receives corresponding points designated by the cursor SL by the user operation with respect to the first image GZ3 and the second image GZ4 (S28). When the user clicks on the image, the processor 30A magnifies and displays the peripheral image around the point on the image displayed on the display 31, which is designated by the cursor SL (S29).

  The input device 26 receives a new corresponding point designated by the user for the enlarged peripheral images GZ11 and GZ12 (S30). Since the distance information (depth value) is input at the time of the initial calibration and stored in the memory 27, the input operation of the depth value may not be performed.

  After this, since the processes of S31 to S38 are the same as the processes of S7 to S14 in the first embodiment, the description of the processes thereafter will be omitted.

  In the case where an error message prompting the user to input another corresponding point position is displayed in S37, the same display as that in FIG. 13 may be performed. That is, the processor 30A displays, for example, guidance ms3 (guidance information) on the screen of the display 31, and in the lower right corner of the screen of the display 31, the previous first image GZ3 and the second image GZ4 and corresponding points am, The peripheral image GZ11 and the peripheral image GZ12 including bm may be displayed in a reduced size.

[Effects, etc.]
As described above, in the PC 20A of the stereo camera system 5A of the second embodiment, the processor 30A performs calibration (an example of the restoration process) including restoration of three-dimensional coordinates multiple times. In the initial calibration (an example of the first restoration processing), the processor 30A captures the position information of the designated corresponding points am and bm input by the input device 26, the depth value, and the first camera 10 in the initial calibration. The memory 27 holds the first image GZ1 and the peripheral image GZ11 including the designated corresponding point am.

  In recalibration (an example of a second restoration process) performed at the time of operation (following the initial calibration), the processor 30A outputs the peripheral image GZ11 in the first image GZ3 captured by the first camera 10 in the recalibration. An area image PGZ3 corresponding to is detected. The processor 30A sets the designated corresponding point obtained in the initial calibration, which is included in the region image PGZ3, as the designated corresponding point am in the recalibration. After setting the designated corresponding point am, the processor 30A recalculates restoration of three-dimensional coordinates.

  As described above, the PC 20A can use the coordinates of the designated corresponding point at the time of initial calibration, the depth value, and the template image at the time of recalibration. Therefore, in the recalibration, it is not necessary to perform the same operation as the initial calibration, and the user operation and the processing of the processor at the time of the recalibration can be reduced. Therefore, the PC 20A can effectively use the coordinates of the designated corresponding point obtained once, the depth value, and the template image, and can easily improve the restoration accuracy of the three-dimensional coordinates restored from the two captured images. The same applies to the case where the current recalibration is performed for the previous recalibration as well as the initial calibration.

  In addition, the display 31 does not detect the area image PGZ3 corresponding to the peripheral image GZ11 in the first image GZ1 in recalibration under the control of the processor 30A, or the depth value held in the memory 27 When the difference between the information of the distance from the first camera 10 to the designated corresponding point am obtained by calibration is equal to or greater than the designated value, the coordinates of the designated corresponding point am held in the memory 27 and the peripheral image GZ11 are displayed You may

  As a result, when recalibration fails, the user can specify the corresponding point again while looking at the position information of the designated corresponding point am and the peripheral image GZ11 displayed on the screen of the display 31. Therefore, the PC 20A can perform recalibration with high accuracy of restoration of three-dimensional coordinates.

  Further, the display 31 refers to the position information of the designated corresponding point am stored in the memory 27 and the peripheral image GZ11, and prompts the user to input the position of the designated corresponding point am in the first image GZ3 in recalibration. The guidance ms3 may be displayed. The guidance ms3 is an example of guidance information.

  Thus, the user can easily specify the corresponding point again according to the guidance ms3 prompting the specification of the corresponding point. Therefore, the operability of the PC 20A is improved.

(Other embodiments)
As mentioned above, the 1st, 2nd embodiment was described as an illustration of the art in this indication. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, and the like have been made. Also, the embodiments may be combined.

  In the first and second embodiments, an example of calibration is shown using FIGS. 5 and 10, but even if the calibration is performed including processes other than the processes shown in FIGS. 5 and 10 Good.

  FIG. 14 is a flow chart showing another example of the initial calibration performed at the time of initial setting. FIG. 15 is a flowchart showing another example of recalibration performed at the time of operation.

  In FIG. 14, when the number of accuracy evaluations does not exceed the threshold in S12, the processor 30 proceeds to S1 and recaptures image data of the first image GZ1 and the second image GZ2. In addition, after the process of S14, the processor 30 proceeds to S1 and recaptures the image data of the first image GZ1 and the second image GZ2.

  In FIG. 15, when the number of accuracy evaluations does not exceed the threshold in S36, the processor 30A proceeds to S21 and recaptures the image data of the first image GZ1 and the second image GZ2. Further, after the process of S38, the processor 30A proceeds to S21 and recaptures the image data of the first image GZ1 and the second image GZ2.

  As described above, the PCs 20 and 20A can acquire a plurality of image data having a difference in imaging time by capturing the image data again. Due to the time difference between the imaging times, for example, bright areas and dark areas in the captured image change, and feature points in each captured image change. By changing the feature points, the corresponding points also change, and the result of the posture estimation also changes.

  Therefore, due to the difference in the imaging times of the plurality of acquired image data, the variation in the time direction is appropriately taken into consideration in the imaged image. Thus, the PCs 20 and 20A can improve the estimation accuracy of estimating the position and orientation of the second camera 11 with respect to the first camera 10. In FIGS. 14 and 15, the designated corresponding point may not be changed even when the image data is fetched again.

  In the first and second embodiments, the processor 30 or 30A exemplifies that the accuracy evaluation is performed based on one designated corresponding point and the calculated corresponding point. Alternatively, the processors 30, 30A may perform the accuracy evaluation based on a plurality of (for example, 2 or 3) designated corresponding points and a plurality of calculated corresponding points. In this case, by using a plurality of corresponding points, the accuracy evaluation becomes more accurate.

  In the first and second embodiments, it is exemplified that the depth value (that is, the Z coordinate value) is used as the distance information to the target point corresponding to the designated corresponding point. Instead of this, evaluation may be performed using, as distance information from the camera to the target point, a distance represented by a square root of a sum of squares of values of X coordinate, Y coordinate, and Z coordinate. As a result, for example, it is possible to cope with the case where it is difficult to obtain the actual measurement value of the Z coordinate of the target point, such as when the target point is largely deviated from the camera in the X coordinate and Y coordinate directions.

  In the second embodiment, at the time of recalibration, the PC 20A exemplifies determining designated correspondence points by template matching for both the first image and the second image. Instead of this, for example, when it is determined that either one of the first image and the second image is obviously deviated, the processor 30A uses the image judged to be deviated to designate the designated corresponding point. You may ask. Then, the processors 30, 30A may perform three-dimensional reconstruction processing using the determined designated corresponding point and the designated corresponding point of the other image in the initial calibration (or the previous recalibration). The PC 20A may also obtain the designated corresponding point by image processing other than template matching.

  In the first and second embodiments, the first camera 10 and the second camera 11 are directly connected to the PCs 20 and 20A via the cables 18A and 18B. Alternatively, a transmitter and a receiver may be provided between the first and second cameras 10 and 11 and the PCs 20 and 20A, and data and signals may be communicated by communication using the transmitter and the receiver. Good. Thereby, the first camera and the second camera are installed at a remote place, and a three-dimensional restoration process can be performed by the PC installed at a distant place.

  In the first embodiment, the processor may be physically configured in any way. In addition, since the processing content can be changed by changing the program by using a programmable processor, the degree of freedom in processor design can be increased. The processor may be configured of one semiconductor chip or may be physically configured of a plurality of semiconductor chips. When it comprises a plurality of semiconductor chips, each control of the first and second embodiments may be realized by different semiconductor chips. In this case, it can be considered that one processor is configured by the plurality of semiconductor chips. Further, the processor may be configured by a member (such as a capacitor) having a function different from that of the semiconductor chip. Further, one semiconductor chip may be configured to realize the function of the processor and the other functions.

5, 5A Stereo Camera System 10 First Camera 11 Second Camera 13 First Case 14 Second Case 18A, 18B, 18C Cable 20, 20A PC
21 feature point extraction unit 22 corresponding point detection unit 23 camera parameter estimation unit 24 three-dimensional coordinate calculation unit 25 accuracy evaluation unit 26 input device 27 memory 28 port 30, 30A processor 31 display 35 readout unit 36 position search unit 41 target point 62 base Line length a1 to a7, b1 to b7 feature points a11 to a15, b11 to b15 corresponding points am, bm specified corresponding points (specified corresponding points)
c1 to c7, cm Corresponding line fr1, fr2 Frame g, h Dotted frame GZ1, GZ3 First image GZ2, GZ4 Second image GZ11, GZ12 Peripheral image MK Mark ms1 Error message ms2, ms3 Guidance pf1, pf2 Rectangular frame PGZ3, PGZ4 Region image SL cursor

Claims (9)

A three-dimensional information restoration apparatus comprising a port, an input device and a processor, wherein
The port acquires a first image captured by a first imaging device and a second image captured by a second imaging device,
The processor is
Detecting a plurality of first corresponding point pairs in which the first feature points in the first image and the second feature points in the second image correspond to each other;
Restoring three-dimensional coordinates in which the first feature point is back-projected based on the plurality of first corresponding point pairs;
The input device is
The position in the first image and the second image of the second corresponding point pair is set as an arbitrary corresponding point pair in the first image and the second image as a second corresponding point pair. And position information indicating the distance from the first imaging device to the second corresponding point pair.
The processor is
The three-dimensional coordinates of the second corresponding point pair are restored based on the input position information of the second corresponding point pair,
The accuracy of the three-dimensional coordinates is evaluated based on the three-dimensional coordinates of the second corresponding point pair restored and the input distance information;
If the accuracy is equal to or less than a predetermined value, the first correspondence point pair is changed to recalculate parameters for deriving three-dimensional coordinates;
Recalculate restoration of three-dimensional coordinates of the input second corresponding point pair based on the recalculated parameter
Three-dimensional information restoration device. The three-dimensional information restoration apparatus according to claim 1, further comprising:
A display that displays, under control of the processor, at least one of the first image and the second image, candidates for the second corresponding point pair included in the plurality of first corresponding point pairs; Three-dimensional information restoration device. The three-dimensional information restoration apparatus according to claim 2, further comprising:
Under control of the processor, the display is a position where the distance from the first imaging device is long in at least one of the first image and the second image as the position of the second corresponding point pair. A three-dimensional information restoration apparatus that displays guidance information prompting to input the position information with priority. The three-dimensional information restoration apparatus according to claim 2 or 3, wherein
The processor is based on distance information input by the input device and information on a distance from the first imaging device to the second corresponding point pair obtained by restoring the three-dimensional coordinates. A three-dimensional information restoration apparatus that determines the necessity of recalculation of restoration of the three-dimensional coordinates. The three-dimensional information restoration apparatus according to claim 4, further comprising:
The display is, under the control of the processor, distance information input by the input device, and information of a distance from the first imaging device to the second corresponding point pair obtained by restoring the three-dimensional coordinates. The three-dimensional information restoration apparatus displays warning information related to the input of the second corresponding point pair when the difference between the two is equal to or more than a predetermined value. The three-dimensional information restoration apparatus according to claim 1, wherein
The processor performs restoration processing including restoration of the three-dimensional coordinates multiple times,
In the first restoration process,
In the first image captured by the first imaging device in the first restoration processing, position information of the second corresponding point pair input by the input device, the distance information, and the second image. Store in memory a third image containing the corresponding point pairs of
In a second restoration process subsequent to the first restoration process,
Detecting a region corresponding to the third image in the first image captured by the first imaging device in the second restoration process;
A corresponding point pair corresponding to the second corresponding point pair obtained in the first restoration process and included in the area is set as a second corresponding point pair in the second restoration process,
Recalculate the reconstruction of the three-dimensional coordinates after setting the second corresponding point pair,
Three-dimensional information restoration device. The three-dimensional information restoration apparatus according to claim 6, further comprising:
Equipped with a display
The display is configured such that, under the control of the processor, an area corresponding to the third image in the first image is not detected in the second restoration process, or the distance information held in the memory And the second information stored in the memory, when the difference between the information of the distance from the first imaging device to the second corresponding point pair obtained by the second restoration processing is equal to or greater than a predetermined value. A three-dimensional information restoration apparatus for displaying position information of a corresponding point pair and the third image. The three-dimensional information restoration apparatus according to claim 7, further comprising:
The display refers to the position information of the second corresponding point pair held in the memory and the third image, and the second correspondence in the first image in the second restoration process A three-dimensional information restoration apparatus that displays guidance information prompting to input position information of point pairs. A three-dimensional information restoration method in a three-dimensional information restoration device comprising a port, an input device and a processor, comprising:
The port acquires a first image captured by a first imaging device and a second image captured by a second imaging device,
The processor is
Detecting a plurality of first corresponding point pairs in which the first feature points in the first image and the second feature points in the second image correspond to each other;
Restoring three-dimensional coordinates in which the first feature point is back-projected based on the plurality of first corresponding point pairs;
The input device is
The position in the first image and the second image of the second corresponding point pair is set as an arbitrary corresponding point pair in the first image and the second image as a second corresponding point pair. And position information indicating the distance from the first imaging device to the second corresponding point pair.
The three-dimensional coordinates of the second corresponding point pair are restored based on the input position information of the second corresponding point pair,
The accuracy of the three-dimensional coordinates is evaluated based on the three-dimensional coordinates of the second corresponding point pair restored and the input distance information;
If the accuracy is equal to or less than a predetermined value, the first correspondence point pair is changed to recalculate parameters for deriving three-dimensional coordinates;
Recalculate restoration of three-dimensional coordinates of the input second corresponding point pair based on the recalculated parameter
3D information restoration method.

Images