JP2015052986A - Three-dimensional restoration device, program, and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional restoration device, program and method capable of easily performing the three-dimensional restoration of a curve from a plurality of two-dimensional images by the intuitive operation of a user.SOLUTION: A display control part 24A allows a display part to display first and second two-dimensional images. A first plotting part 24B allows a user to freely plot a first two-dimensional curve overlapped with the first two-dimensional image on the display part, and a second plotting part 24C allows the user to freely plot a second two-dimensional curve corresponding to the first two-dimensional curve overlapped with the second two-dimensional image on the display part. A first specification part 24D specifies a first control point group as the control point group of the first two-dimensional curve or its approximation curve, and a second specification part 24E specifies a second control point group as the control point group of the second two-dimensional curve or its approximation curve. A three dimensional restoration part 24F calculates the three-dimensional positions of the first and second control point groups on the basis of a correspondence relationship between the first control point group and the second control point group, and restores a three-dimensional curve on the basis of the result.

Description

  The present invention relates to a three-dimensional restoration apparatus, program, and method for three-dimensional restoration of a curve from a plurality of two-dimensional images, and more particularly to three-dimensional restoration of virtual lines that are difficult to recognize on a two-dimensional image. The present invention relates to a suitable three-dimensional restoration apparatus, program, and method.

  Conventionally, based on the principle of stereo vision, a three-dimensional shape of a subject is restored using left and right two-dimensional images taken from different angles. In stereo viewing, if a corresponding point on the left and right two-dimensional images can be identified, the three-dimensional coordinates of this point can be obtained. In addition, three-dimensional reconstruction by factorization is also known, and even in factorization, corresponding points on a plurality of two-dimensional images taken from different angles are specified, and the three-dimensional coordinates of the points are obtained. be able to. Corresponding points can be automatically extracted as feature points by a computer by various known methods such as edge detection and corner detection. Further, Patent Document 1 proposes that a user input several points on curves corresponding to each other on the left and right two-dimensional images. According to this method, the three-dimensional curve can be restored while reducing the calculation load.

Japanese Patent Laid-Open No. 11-213162

  However, it may be difficult for the user to designate the above points. For example, in the case of a two-dimensional image capturing a subject having a smooth surface with little or no pattern, the user cannot understand where to specify. Of course, with such a two-dimensional image, automatic extraction by a computer is also difficult. As a specific example, when the body length of a dolphin is to be measured, a method of three-dimensionally restoring the virtual line L1 as shown in FIG. 1 and measuring the length of the virtual line L1 can be considered. However, even if it is attempted to specify some points on the virtual line L1 on the two-dimensional image in order to restore such a virtual line L1 three-dimensionally, it is almost impossible to recognize other than the start point and the end point. As a result, it becomes difficult to restore the virtual line L1 correctly in three dimensions, and the body length of the dolphin cannot be measured.

  The present invention provides a three-dimensional restoration apparatus, program, and method capable of easily performing three-dimensional restoration of a virtual line that is difficult to recognize from a plurality of two-dimensional images by a user's intuitive operation. Objective.

  A three-dimensional restoration apparatus according to a first aspect of the present invention includes a display control unit, a first drawing unit, a second drawing unit, a first specifying unit, a second specifying unit, and a three-dimensional restoring unit. . The display control unit displays a first 2D image and a second 2D image obtained by photographing the subject from different angles on the display unit. The first drawing unit allows the user to freely draw the first two-dimensional curve so as to overlap the first two-dimensional image on the display unit. The second drawing unit allows a user to freely draw a second 2D curve corresponding to the first 2D curve by overlapping the second 2D image on the display unit. The first specifying unit specifies a first control point group that is a control point group of the first two-dimensional curve or an approximate curve of the first two-dimensional curve. The second specifying unit specifies a second control point group that is a control point group of the second two-dimensional curve or an approximate curve of the second two-dimensional curve. The three-dimensional restoration unit obtains a three-dimensional position of the first control point group and the second control point group based on a correspondence relationship between the first control point group and the second control point group, and the result To restore the 3D curve.

  Here, a two-dimensional image can be displayed on the display unit, and a two-dimensional curve can be freely drawn on the user by superimposing the two-dimensional image on the display unit. That is, the user can specify the entire curve to be restored three-dimensionally with reference to the background image on the two-dimensional image. In addition, the user can visually confirm the two-dimensional curve displayed superimposed on the two-dimensional image while designating the two-dimensional curve in this way. Therefore, the user can specify a curve that correctly reflects his / her intention by an intuitive operation. Note that the above drawing operations are accepted for a plurality of two-dimensional images taken from different angles.

  Then, when two-dimensional curves corresponding to each other are drawn on a plurality of two-dimensional images, control point groups for each two-dimensional curve are specified, and based on the correspondence between these control point groups, the three-dimensional curve is Restored. When the user designates a two-dimensional curve by freehand or the like, since the two-dimensional curve drawn by the user does not have a control point in a strict sense, the two-dimensional curve is changed to a curve having a control point. After the approximation, control points are determined. The curve having control points is, for example, a Bezier curve or a B-spline curve.

  As described above, here, the user can freely draw the entire curve desired to be three-dimensionally restored on the two-dimensional image on the display unit that displays the two-dimensional image. Therefore, the user can freely specify a curve to be restored three-dimensionally by intuitive operation, and can easily perform the three-dimensional restoration of a curve from a plurality of two-dimensional images. In particular, even if the curve to be restored three-dimensionally does not originally exist on the two-dimensional image and is a virtual line that is difficult to recognize, here, such a curve is intuitively displayed on the two-dimensional image. Can be specified. In the example of the dolphin in FIG. 1 described above, it is difficult to specify points other than the start point and the end point on the virtual line L1, but if the entire virtual line L1, the background image and the line drawn by itself are used. Can be grasped while visually checking, and the virtual line L1 can be designated. In addition, since the part to be subjected to the three-dimensional reconstruction is designated by a curve having control points, a smoothing effect can be obtained, and an improvement in safety and an increase in tolerance for errors are expected.

  A three-dimensional restoration device according to a second aspect of the present invention is the three-dimensional restoration device according to the first aspect, further comprising a calculation unit that calculates the length of the three-dimensional curve. Here, by allowing a user to freely draw a curve whose size is to be measured on a plurality of two-dimensional images, it is possible to easily specify and calculate a portion where the size of the subject is to be known. In particular, when calculating the curve length on a smooth surface as in the case of dolphins, in this device in which the part to be three-dimensionally restored is specified by a curve having control points, a calculation error occurs during three-dimensional restoration. Can be absorbed and accuracy can be improved.

  A three-dimensional restoration program according to a third aspect of the present invention includes a step of displaying a first 2D image and a second 2D image obtained by photographing a subject from different angles on a display unit, and the display unit A step of allowing a user to freely draw a first 2D curve superimposed on the first 2D image, and a second corresponding to the first 2D curve superimposed on the second 2D image on the display unit. A step of allowing a user to freely draw a two-dimensional curve; a step of specifying a first control point group which is a control point group of the first two-dimensional curve or an approximate curve of the first two-dimensional curve; A step of specifying a second control point group that is a control point group of a two-dimensional curve or an approximate curve of the second two-dimensional curve, and a correspondence relationship between the first control point group and the second control point group , The first control point group and the second control point group It determined the three-dimensional position, on the basis of the result, to perform the steps in the computer to restore the three-dimensional curve. Here, the same effect as the first aspect can be achieved.

  A three-dimensional restoration program according to a fourth aspect of the present invention is a three-dimensional restoration program according to a third aspect, further causing a computer to execute a step of calculating the length of the three-dimensional curve. Here, the same effect as the second aspect can be achieved.

  A three-dimensional restoration method according to a fifth aspect of the present invention is a method for restoring a three-dimensional shape of a subject along a virtual line that is difficult to recognize, and is obtained by photographing the subject from different angles. A step of capturing one two-dimensional image and the second two-dimensional image into a computer; a step of causing the computer to display the first two-dimensional image and the second two-dimensional image on a display unit; A step of freely drawing a first two-dimensional curve that traces the virtual line over the first two-dimensional image on a section, and inputting the first two-dimensional curve to the computer; A step of freely drawing a second 2D curve that traces the virtual line on the second 2D image on the display unit, and inputting the second 2D curve to the computer; Specifying a first control point group that is a control point group of the first two-dimensional curve or an approximate curve of the first two-dimensional curve; and the computer, the second two-dimensional curve or the second A step of identifying a second control point group which is a control point group of an approximate curve of a two-dimensional curve, and the computer based on a correspondence relationship between the first control point group and the second control point group. Obtaining three-dimensional positions of one control point group and the second control point group, and restoring a three-dimensional curve based on the result. Here, the same effect as the first aspect can be achieved.

  In addition, the user freely draws the first 2D curve and the second 2D curve, the computer specifies the first control point and the second control point, and the computer restores the 3D curve. By repeating a plurality of times, that is, by dividing and processing a curve to be subjected to three-dimensional restoration into a plurality of lines, a curve that more accurately reflects the user's intention can be reproduced.

  A three-dimensional restoration method according to a sixth aspect of the present invention is the three-dimensional restoration method according to the fifth aspect, wherein the computer further includes a step of calculating a length of the three-dimensional curve. Here, the same effect as the second aspect can be achieved.

  ADVANTAGE OF THE INVENTION According to this invention, on the display part which displays a two-dimensional image, a user can freely draw the whole curve which he wants to restore | restore three-dimensionally on a two-dimensional image. Therefore, the user can freely specify a curve to be restored three-dimensionally by intuitive operation, and can easily perform the three-dimensional restoration of a curve from a plurality of two-dimensional images. In particular, even if the curve to be restored three-dimensionally does not originally exist on the two-dimensional image and is a virtual line that is difficult to recognize, here, such a curve is intuitively displayed on the two-dimensional image. Can be specified. In addition, since the part to be subjected to the three-dimensional reconstruction is designated by a curve having control points, a smoothing effect can be obtained, and an improvement in safety and an increase in tolerance for errors are expected.

The figure which shows the dolphin used as the object of a measurement. 1 is a schematic configuration diagram of a three-dimensional restoration apparatus (three-dimensional restoration system) according to an embodiment of the present invention. The schematic block diagram of a computer. The flowchart which shows the flow of a three-dimensional measurement process. The figure which shows the drawing screen for making a user draw a two-dimensional curve. The figure which shows the drawing screen on which the two-dimensional curve was drawn. The figure which shows the principle of three-dimensional reconstruction. The figure which shows the experimental equipment used for performance evaluation. The figure explaining the other use of a three-dimensional decompression | restoration apparatus.

  Hereinafter, a three-dimensional restoration apparatus, program, and method according to an embodiment of the present invention will be described with reference to the drawings.

<1. Schematic configuration>
FIG. 2 is a schematic configuration diagram of a three-dimensional restoration system 1 which is a three-dimensional restoration apparatus according to the present embodiment. The three-dimensional restoration system 1 includes a photographing system 10 and a computer 20, and a plurality of two-dimensional images (hereinafter referred to as 2D images) photographed from different angles by the photographing system 10 are processed by the computer 20, whereby the subject Restore the three-dimensional shape of. Specifically, the computer 20 can perform a three-dimensional restoration of a curve captured on a 2D image so as to be recognizable or difficult to recognize. The computer 20 interactively grasps the user's intention by causing the user to draw the entire curve to be three-dimensionally restored.

  The three-dimensional restoration system 1 of the present embodiment is a system suitable for measuring the length of a dolphin that freely moves around in water. In order to measure the body length of a dolphin, for example, it is necessary to measure the length of a line L1 as shown in FIG. 1, but the dolphin is an animal having a smooth body surface with little or no pattern. Accordingly, the line L1 is a virtual image that is difficult to automatically extract by image processing on a 2D image obtained by photographing a dolphin, is not originally present, and is difficult for a human to visually recognize. Is a line. However, it is not practical to attach a marker to the body surface of the dolphin so that the virtual line L1 can be recognized on the 2D image. Also, taking a 2D image while irradiating a colored laser beam on a dolphin is not recommended because the influence of the laser beam on the dolphin is unknown. In this respect, the three-dimensional restoration system 1 of the present embodiment is not such an invasive method, and the body length of the dolphin, that is, the length of the virtual line L1 is not determined from a plurality of 2D images obtained by simply photographing the dolphin. It is possible to measure invasively. Hereinafter, the configuration and operation of the three-dimensional restoration system 1 will be described in detail.

<2. Shooting system>
As shown in FIG. 2, the photographing system 10 includes a plurality of waterproof cameras 11A, 11B,... Installed in the water so as to photograph a dolphin that moves freely in the water. Therefore, the photographing system 10 can photograph the same scene from various angles. The relative positional relationship between the cameras 11A, 11B,... Is known, and the focal lengths of the cameras 11A, 11B,. The cameras 11A, 11B,... Are connected to a nonvolatile storage device 13 such as a hard disk or a flash memory.

  The cameras 11 </ b> A, 11 </ b> B,... Can take moving images, and these moving images are synchronized with reference to an external trigger signal and are sent to the storage device 13 via the communication line 12. On the other hand, the storage device 13 accumulates frames included in a plurality of moving images sent from a plurality of cameras 11A, 11B,. That is, in the storage device 13, frames taken at the same time by a plurality of cameras 11A, 11B,... Are associated with each other. Note that the method of associating frames shot at the same time is not limited to this. For example, the setting times of the cameras 11A, 11B,... You may make it do.

<3. Computer>
The computer 20 is a general-purpose personal computer as hardware, and is installed with a 3D restoration program 2A which is a 3D restoration program according to the present embodiment. The three-dimensional restoration program 2A is a program for causing the computer 20 to execute processing to be described later, read from a computer-readable recording medium 70 such as a CD-ROM or USB memory, or a communication network such as the Internet or a LAN. Downloaded from another device and installed. The computer 20 is not limited to hardware as long as it can execute image processing to be described later, and may be a device such as a smartphone or a PDA.

  As shown in FIG. 3, the computer 20 includes a display unit 21, an input unit 22, a storage unit 23, a control unit 24, and a communication unit 25. These units 21 to 25 are connected to each other via the bus line 8 and can communicate with each other as appropriate.

  The display unit 21 is a display device such as a liquid crystal display, and displays a screen or the like to be described later to the user. The input unit 22 includes a mouse, a keyboard, a touch panel, a touch pad, and the like, and is an input device that receives an operation from the user on the computer 20.

  The storage unit 23 is a non-volatile storage device such as a hard disk or a flash memory, and stores a three-dimensional restoration program 2A. The control unit 24 includes a CPU, a ROM, a RAM, and the like, and reads and executes the three-dimensional restoration program 2A stored in the storage unit 23, thereby virtually displaying the display control unit 24A and the first drawing. The unit 24B, the second drawing unit 24C, the first specifying unit 24D, the second specifying unit 24E, the three-dimensional restoration unit 24F, the calculation unit 24G, and the calibration unit 24H operate. Details of the operation of each of the units 24A to 24H will be described later.

  The communication unit 25 functions as a communication interface for data communication with an external storage device such as a hard disk or a flash memory, in addition to connecting the computer 20 to a communication network such as the Internet or a LAN. Therefore, the communication unit 25 can acquire the above-described moving image data stored in the storage device 13 and take it into the storage unit 23.

<4. Three-dimensional measurement method>
Hereinafter, a method for three-dimensionally measuring the length of a dolphin using the three-dimensional restoration system 1 will be described in detail.

  First, the camera 11 </ b> A, 11 </ b> B,... Is photographed as a moving image of the dolphin swimming freely in the water and stored in the storage device 13. Thereafter, the moving image data in the storage device 13 is captured by the computer 20 and stored in the storage unit 23.

  When the above preparation is completed, the computer 20 executes the three-dimensional measurement process according to the flowchart of FIG. First, the user reproduces the moving image data in the storage unit 23 on the display unit 21, and selects a pair of left and right 2D images suitable for measuring the body length of the dolphin while watching this (step S1). A pair of left and right 2D images is a pair of frames taken from different angles with different cameras at the same timing. As described above, frames taken at the same time by the cameras 11A, 11B,... Are associated with each other. Accordingly, in order to facilitate the user's selection, the computer 20 can indicate to the user which are the frames taken at the same timing. For example, this can be shown by causing the display control unit 24A to simultaneously reproduce the moving images taken by the cameras 11A, 11B,... On the display unit 21 while synchronizing them. In addition, the 2D image to be selected here is typically an image as shown in FIG. 5, that is, at the timing when the dolphin is in a posture where the center line of the body is U-shaped. Is an image that is generally captured from the ventral side. The dolphin may take a posture in which the tail fin is folded back and the center line of the body is S-shaped, but it is preferable that a scene image of a simpler U-shaped posture is selected. However, even from a 2D image of a dolphin with an S-shaped posture, it is possible to measure the body length as in the case of the U-shape.

  When the user finishes selecting the pair of left and right 2D images, the display control unit 24A displays a drawing screen W1 as shown in FIG. 5 on the display unit 21 (step S2). The drawing screen W1 displays a pair of left and right 2D images (hereinafter referred to as 2D images 41 and 42) selected in step S1 in the vertical and horizontal directions and displays them simultaneously.

  The first drawing unit 24B accepts an operation for freely drawing a two-dimensional straight line (hereinafter referred to as 2D straight line) and a two-dimensional curve (hereinafter referred to as 2D curved line) on the 2D image 41 of the drawing screen W1. Similarly, the second drawing unit 24C also receives an operation for drawing a 2D straight line and a 2D curve on the 2D image 42 on the drawing screen W1 from the user. This drawing operation can be realized by various methods. For example, a straight line and a curve are drawn along the operation direction instructed by the user, such as the moving direction of the mouse or the moving direction of the finger on the touch pad. It is intuitive and preferable. In the present embodiment, the 2D straight line and the 2D curve are configured to be drawn freehand.

  Using this function, the user combines the 2D straight lines 45 and 46 and the 2D curves 43 and 44 on the 2D images 41 and 42, respectively, as shown in FIG. Drawing is performed (step S3). That is, on the head side of the dolphin that hardly deforms, the virtual line L1 is drawn as straight lines 45 and 46, and on the tail fin side that flexibly deforms, the virtual line L1 is drawn as curves 43 and 44. At this time, the user inputs to the computer 20 that the 2D straight lines 45 and 46 correspond to each other and that the 2D curves 43 and 44 correspond to each other. For example, as shown in FIG. 6, the correspondence between the lines 43 to 46 between the 2D images 41 and 42 can be specified by drawing the corresponding lines with the same line type. As described above, in step S3, the user draws a projection line on the 2D images 41 and 42 of the same virtual line L1 in the real world. At this time, the user determines where the switching points between the straight lines 45 and 46 and the curves 43 and 44 are other feature points (for example, the position of the chest fin) reflected in the background 2D images 41 and 42. Judge with reference to.

  Although the virtual line L1 is not shown on the 2D images 41 and 42, the user can draw the entire virtual line L1 while referring to the background image. Specifically, the user looks at the overall outline of the dolphin and the positions of the chest fin, the tip of the nose, the tail fin, etc., and imagines the position of the virtual line L1, and the 2D straight lines 45 and 46 and the 2D curve 43 Specify 44. In addition, the drawing units 24B and 24C display the 2D straight lines 45 and 46 and 2D curves 43 and 44 specified by the user on the 2D images 41 and 42 on the drawing screen W1 in real time. Accordingly, the user can visually confirm the drawn lines 43 to 46 on the 2D images 41 and 42 in real time while drawing the 2D straight lines 45 and 46 and the 2D curves 43 and 44. Furthermore, the drawing units 24B and 24C accept operations from the user for correcting the 2D straight lines 45 and 46 and 2D curves 43 and 44 drawn by the user, and this correction is also reflected on the 2D images 41 and 42 in real time. Is done. Accordingly, the user can intuitively specify the appropriate 2D straight lines 45 and 46 and 2D curves 43 and 44 corresponding to the virtual line L1 through trial and error.

When the drawing of the virtual line L1 ends, the process proceeds to step S4. Note that the end timing of drawing of the virtual line L1 is recognized when the user presses the enter button B1 on the drawing screen W1. In step S4, the first specifying unit 24D approximates the 2D curve 43 drawn freehand by the user to a two-dimensional cubic Bezier curve. Further, the first specifying unit 24D has two-dimensional coordinates P _L0 = (x _L0 , y _L0 ), P _L1 = (x _L1 , y _L1 ) of the four control points of the cubic Bezier curve in the 2D image 41, P _L2 = (x _L2 , y _L2 ) and P _L3 = (x _L3 , y _L3 ) are derived. Similarly, the second specifying unit 24E approximates the 2D curve 44 drawn freehand by the user to a two-dimensional cubic Bezier curve. In addition, the second specifying unit 24E has two-dimensional coordinates P _R0 = (x _R0 , y _R0 ) P _R1 = (x _R1 , y _R1 ), P of the four control points of this cubic Bezier curve in the 2D image 42. _R2 = ( _xR2 , _yR2 ) and _PR3 = ( _xR3 , _yR3 ) are derived. Note that (P _L0 , P _R0 ), (P _L1 , P _R1 ), (P _L2 , P _R2 ), and (P _L3 , P _R3 ) correspond to each other on different 2D images 41 and 42 according to this combination. (P _L0 , P _R0 ) is the start point, and (P _L3 , P _R3 ) is the end point. This correspondence is automatically recognized by the computer 20.

Specifically, in step S4, first, the start points P _L0 and P _R0 and the end points P _L3 and P _{R3 of} the Bezier curve are detected as control points. In the Bezier curve, other control points P _L1 , P _R1 , P _L2 , and P _R2 exist on the tangent lines at the start points P _L0 and P _R0 and the end points P _L3 and P _R3 . The initial values of the coordinates of the control points P _L1 , P _R1 , P _L2 , P _R2 are set and learning proceeds. Therefore, in step S4, the control points P _{L0 to} P _L3 and P _{R0 to} P _R3 can be obtained at high speed and with high accuracy.

In the present embodiment, the initial values of the coordinates of the control points P _L1 , P _R1 , P _L2 , P _R2 are set by interaction with the user. Specifically, the specifying units 24D and 24E derive temporary approximate curves of the 2D curves 43 and 44 drawn by the user freehand and control points of the temporary approximate curves, and the display control unit 24A The temporary approximate curve and the temporary control point are displayed on the 2D images 41 and 42. The specifying units 24D and 24E receive an operation for freely moving a temporary control point on the 2D images 41 and 42 from the user, and when the user actually moves the point, the temporary unit 24D and 24E are temporarily moved according to the moved position of the point. Recalculate the approximate curve of. The recalculated temporary approximate curve is displayed on the 2D images 41 and 42 in real time by the display control unit 24A. Then, when the user determines that the 2D curves 43 and 44 drawn by himself / herself are visually approximated while moving the temporary control points in various directions, the fact is close. Is input to the computer 20. Then, the specifying units 24D and 24E set the position of the temporary control point at the timing when the input is performed as an initial value for learning, and search for the position of the final control point.

  When a curve length along a smooth surface such as the body surface of a dolphin is to be measured, if the approximation to a higher order Bezier curve is performed, the accuracy may be reduced. Accordingly, in step S4, approximation to a cubic Bezier curve is performed. Using a low-order Bezier curve models a subject with a smooth curve, and is suitable for measuring a curve length along a smooth surface such as the body surface of a dolphin.

In subsequent step S5, the three-dimensional reconstruction unit 24F converts a two-dimensional cubic Bezier curve that is an approximate curve of the 2D curve 43 and a two-dimensional cubic Bezier curve that is an approximate curve of the 2D curve 44 into a three-dimensional cubic. Convert to a Bezier curve (hereinafter referred to as 3D reconstruction curve). Specifically, the three-dimensional reconstruction unit 24F has a three-dimensional coordinate P ₀ = (x ₀ , y ₀ ) in the world coordinate system (XYZ space) of the four control points of the 3D reconstruction curve based on the principle of stereo vision. , Z ₀ ), P ₁ = (x ₁ , y ₁ , z ₁ ), P ₂ = (x ₂ , y ₂ , z ₂ ), P ₃ = (x ₃ , y ₃ , z ₃ ). This three-dimensional restoration calculation is based on the correspondence between the control points P _L0 , P _L1 , P _L2 , and P _L3 on the 2D image 41 and the control points P _R0 , P _R1 , P _R2 , and P _R3 on the 2D image 42. Based on. That is, the three-dimensional coordinates of P ₀ from two control points the two-dimensional coordinates of P _L0, P _R0, three-dimensional coordinates of P ₁ from the two dimensional coordinates of the control points P _L1, P _R1, P ₂ Are derived from the two-dimensional coordinates of the two control points P _L2 and P _R2 , and the three-dimensional coordinate of P ₃ is derived from the two-dimensional coordinates of the two control points P _L3 and P _R3 . In addition, the relative positional relationship and the focal length of the cameras 11A, 11B,... Are used for the calculation of the three-dimensional restoration, and these values are stored in the storage unit 23 in advance. To do.

A detailed algorithm of the three-dimensional restoration in step S5 will be described with reference to FIG. In the following, for simplicity, a case will be described in which P ₀ = (x ₀ , y ₀ , z ₀ ) is calculated from P _L0 = (x _L0 , y _L0 ) and P _R0 = (x _R0 , y _R0 ). The same applies to the remaining P _L1 , P _L2 , and P _L3 . Now, it is assumed that the 2D image 41 is taken by the camera 11A, and the focal length of the camera 11A is f1. Further, the 2D image 42 is taken by the camera 11B, and the focal length of the camera 11B is f2. Further, it is assumed that the positions of the cameras 11A and 11B (more precisely, the position of the center of the image sensor) are relatively (Dx, Dy, Dz) apart in the XYZ space. The origin in the XYZ space may be determined anywhere, but here, the position of the camera A is the origin (0, 0, 0). In this case, in the XYZ space, the coordinates of the position of the camera 11B are (Dx, Dy, Dz). The coordinates of the position of the camera 11B can be set as (Dx, 0, 0) to simplify the calculation.

The point P _L0 is projected at a position of coordinates (x _L0 , y _L0 ) in the virtual screen surface SC1 existing in front of the camera 11A. Further, the distance D1 from the position (0, 0, 0) of the camera 11A to the virtual screen surface SC1 is converted from the focal length f1. Therefore, the three-dimensional reconstruction unit 24F can calculate the three-dimensional direction vector (X _L0 , Y _L0 , Z _L0 ) of the point P _L0 in the XYZ space based on the values of x _L0 , y _L0 and f1. . Similarly, the point P _R0 is projected at the position of the coordinates (x _R0 , y _R0 ) in the virtual screen surface SC2 existing in front of the camera 11B. Further, the distance D2 from the position (Dx, Dy, Dz) of the camera 11B to the virtual screen surface SC2 is converted from the focal length f2. Therefore, the three-dimensional reconstruction unit 24F uses the three-dimensional direction vector (X _R0 , Y _R0 , Z _R0 ) of the point P _R0 in the XYZ space based on the values of x _R0 , y _R0 , f2, Dx, Dy, and Dz. Can be calculated.

Subsequently, the three-dimensional reconstruction unit 24F includes a straight line U1 having a three-dimensional direction vector P _L0 = (X _L0 , Y _L0 , Z _L0 ) starting from the position (0, 0, 0) of the camera 11A and the camera 11B. A straight line U2 having a three-dimensional direction vector P _R1 = (X _R0 , Y _R0 , Z _R0 ) starting from the position (Dx, Dy, Dz) is calculated. Theoretically, the intersection of these two straight lines U1 and U2 is the point P ₀ = (x ₀ , y ₀ , z ₀ ). However, normally, the straight lines U1 and U2 calculated by the three-dimensional restoration unit 24F are in a twisted position due to an error and do not intersect. Accordingly, here, the three-dimensional reconstruction unit 24F calculates the three-dimensional coordinates of the midpoint between the two points that are the shortest distances on the straight lines U1 and U2, and this is calculated as P ₀ = (x ₀ , y ₀ , z _0). ).

In subsequent step S6, the calculation unit 24G calculates the length of the 3D restoration curve derived in step S5. Specifically, the 3D reconstruction curve (x (t), y (t), z (t)) is obtained by using the three-dimensional coordinates P ₀ = (x ₀ , y ₀ , 4) of the four control points derived in step S5. z ₀ ), P ₁ = (x ₁ , y ₁ , z ₁ ), P ₂ = (x ₂ , y ₂ , z ₂ ), P ₃ = (x ₃ , y ₃ , z ₃ ) It is expressed as follows. t is a parameter that satisfies 0 ≦ t ≦ 1.

The length T1 of the 3D restoration curve is expressed by the following equation. In this embodiment, the Simpson method is used to calculate the following integral value.

In subsequent step S7, the first specifying unit 24D performs two-dimensional coordinates (x _L4 , y _L4 ) of the start point P _L4 and 2D coordinates (x _L5 , y of the end point P _L5 ) of the 2D straight line 45 drawn freehand by the user. _L5 ) is derived in the 2D image 41. Similarly, the second specific portion 24E is, two-dimensional coordinates of the starting point P _R4 of 2D linear 46 drawn by the user freehand (x _R4, y _R4) and two-dimensional coordinates of the end point _{P R5 (x R5, y R5} ) Are derived in the 2D image 42.

In subsequent step S8, the three-dimensional restoration unit 24F converts the 2D straight lines 45 and 46 into three-dimensional straight lines (hereinafter referred to as 3D restored straight lines). Specifically, the three-dimensional reconstruction unit 24F, based on the principle of stereo vision, 3D coordinates P ₄ = (x ₄ , y ₄ , 3D) in the world coordinate system (XYZ space) of the start point and end point of the 3D reconstruction curve. z ₄ ), P ₅ = (x ₅ , y ₅ , z ₅ ) is derived. Three-dimensional coordinates of P ₄ from the corresponding two dimensional coordinates of the starting point P _L4, P _R4 together, the three-dimensional coordinates of P _5, respectively from the corresponding two dimensional coordinate of the end point P _L5, P _R5 together Derived. In addition, the relative positional relationship and the focal length of the cameras 11A, 11B,. Since the basic principle is the same as that in step S5, detailed description is omitted here. If the user has specified that the end point P ₅ of the straight line and the start point P _{0 of the} curve overlap, the calculation for deriving the three-dimensional coordinates of the end point P ₅ may be omitted.

In subsequent step S9, the calculation unit 24G calculates the length of the 3D restoration straight line derived in step S8. Specifically, the length T2 of the 3D restoration straight, three-dimensional coordinates of the start point _{P 4 (x 4, y 4} , z 4) and three-dimensional coordinates of the end point _{P 5 (x 5, y 5} , z 5) And is represented as follows:

  In the subsequent step S10, the calculating unit 24G adds the length T1 of the 3D restoration curve calculated in step S6 and the length T2 of the 3D restoration line calculated in step S9, so that the length of the virtual line L1 is obtained. Is calculated. Thereafter, the length of the virtual line L1 is stored in the storage unit 23 as a calculation result and displayed on the display unit 21 by the display control unit 24A (step S11).

<5. Calibration>
As described above, the computer 20 can measure the length in the three-dimensional space of the line arbitrarily designated by the user on the drawing screen W1. The computer 20 is equipped with a calibration function for improving the accuracy of the three-dimensional measurement. In this calibration, the relative positional relationship between the cameras 11A, 11B,..., Which are parameters used for the calculation of the three-dimensional measurement, is corrected.

  Specifically, a rectangular flat plate is photographed by the cameras 11A, 11B,... And the image data is taken into the storage unit 23 of the computer 20. Then, the display control unit 24A displays a plurality of images taken by the cameras 11A, 11B,. The calibration unit 24H causes the user to specify the coordinates of the four corners of the flat plate on each image displayed on the display unit 21. Subsequently, the calibration unit 24H uses the relative positional relationship between the two cameras that have captured these two images as variables for two images selected from these images. Optimization is performed so that the specified four corners are on the same plane. The calibration unit 24H stores the optimal solution thus obtained in the storage unit 23 as a correction value for the relative positional relationship of the cameras.

  Although the relative positional relationship between the cameras 11A, 11B,... Is expected to be gradually changed, the accuracy of three-dimensional measurement can be maintained by performing the above calibration periodically. .

<6. Performance evaluation>
In order to evaluate the effects of the above three-dimensional measurement method and calibration method, the following experiment was performed using the experimental equipment shown in FIG. First, the two cameras 15 and 16 were placed on a horizontal desk so that the optical axes A1 and A2 were parallel to each other. At this time, the distance between the centers of the two cameras 15 and 16 was measured with a measure and set to 50.00 cm. Further, a bar member made of a flexible material having a total length of 42.00 cm was prepared, and this was bent into a U-shape to create a U-shaped member 50. Then, this U-shaped member 50 was placed on a flat plate and arranged in front of the cameras 15 and 16 with an interval of about 165 cm from the cameras 15 and 16. At this time, the flat plate on which the U-shaped member 50 was placed was disposed so as to be slightly inclined with respect to the desk surface on which the cameras 15 and 16 were placed.

  Under the above environment, the U-shaped member 50 is photographed by the two cameras 15 and 16, and the length of the U-shaped member 50 is three-dimensionally measured by the three-dimensional measuring method of the present embodiment. As a result, the result is 42.85 cm. was gotten. That is, the error was 42.85 cm-42.00 cm = 0.85 cm (2.0%). In this three-dimensional measurement, the relative positional relationship between the cameras 15 and 16 is assumed to be 50.00 cm in the left and right direction and 0.00 cm in the vertical direction.

  Thereafter, the relative positional relationship between the cameras 15 and 16 was corrected by the calibration method of the present embodiment. Specifically, the whiteboard was photographed with the cameras 15 and 16, and the coordinates of the four corners of the whiteboard were designated manually on the two obtained images. As a result, the relative positional relationship between these cameras was corrected as being 49.06 cm in the left-right direction and 0.043 cm in the vertical direction. And while diverting the 2D image used in the 3D measurement performed immediately before and the 2D curve drawn by the user, only the parameters of the relative positional relationship of the cameras 15 and 16 are changed to the corrected values, When the length of the U-shaped member 50 was recalculated, a result of 42.75 cm was obtained. That is, the error was 42.75 cm-42.00 cm = 0.75 cm (1.8%), and the calibration effect was confirmed.

<7. Features>
In the above-described embodiment, the computer 20 overlaps the 2D images 41 and 42 displayed on the display unit 21 and allows the user to freely draw the entire curve L1 to be three-dimensionally restored. Therefore, the user can specify the curve L1 to be restored three-dimensionally by intuitive operation, and can easily perform the three-dimensional restoration of the curve L1 from the plurality of 2D images 41 and 42. In particular, in the above embodiment, the object of measurement is the body length of the dolphin, and the curve L1 to be three-dimensionally restored is a virtual line that is difficult to recognize on the 2D images 41 and 42. It can be specified intuitively on the 2D images 41 and 42.

<8. Modification>
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said embodiment, A various change is possible unless it deviates from the meaning. For example, the following changes can be made. Moreover, the gist of the following modifications can be combined as appropriate.

<8-1>
In the above-described embodiment, a method for performing three-dimensional restoration from two 2D images has been described. However, two or more sets of 2D images are created from 3 or more 2D images, and 2D straight lines and 2D curves are three-dimensionally restored for each set, and the respective results are averaged as the final result. Also good. Further, a 2D curve may be restored three-dimensionally by a factorization method using a plurality of 2D images simultaneously. When the factorization method is used, the relative positional relationship between the cameras 11A, 11B,... Used for photographing may be unknown. The same applies to other parameters such as focal length. In the present invention, the method for performing the three-dimensional restoration using the corresponding points after obtaining the corresponding points on a plurality of 2D images is not particularly limited, and the above-described stereo vision and factorization method are examples.

<8-2>
The 2D curve may be approximated to a quadratic Bezier curve instead of a cubic Bezier curve, or may be approximated to a quadratic or higher order Bezier curve. However, if the length of a curve along a smooth surface such as the body surface of a dolphin is measured, it is preferable to use a low-order Bezier curve because the effect of smoothing is obtained. The third order is optimal. Further, it may be approximated to various curves (parametric free curves) having control points such as B-spline curves instead of Bezier curves.

<8-3>
In the above embodiment, the three-dimensional reconstruction is performed by combining a straight line and a curve. However, the three-dimensional reconstruction may be performed by combining the curve and the curve, or only one curve may be restored three-dimensionally. The process shown in the flowchart of FIG. 4 is a process particularly suitable for measuring the body length of a dolphin, and can be appropriately changed depending on the measurement target. Similarly, even when measuring the body length of a dolphin, it is not always necessary to perform a three-dimensional reconstruction by combining a straight line and a curve, and a three-dimensional reconstruction may be performed by combining a curve and a curve. Only the curve may be restored three-dimensionally. Further, the user may designate a straight line and / or curve to be restored three-dimensionally, or the computer 20 may automatically divide the straight line and / or curve input by the user into a plurality of lines. May be. In the latter example, for example, the user designates a single curve, and further manually designates feature points as a reference for division in the computer 20, and the computer 20 designates the user based on the positions of these points. The curved line may be automatically divided into a plurality of lines.

<8-4>
In the above embodiment, the 2D curve is configured to be drawn freehand by the user, but the 2D curve drawing mode is not limited to this. For example, the drawing units 24B and 24C may be configured to draw a Bezier curve in accordance with a user operation. In this case, since the 2D curve drawn by the user is always a Bezier curve, the processing in step S4 for approximating the 2D curve drawn by the user to the Bezier curve can be omitted. In this case, since the drawing units 24B and 24C calculate the two-dimensional coordinates of the control points at the time of drawing the Bezier curve, the specifying units 24D and 24E obtain the calculation results of the drawing units 24B and 24C. Can be used to specify the two-dimensional coordinates of the control points.

<8-5>
The three-dimensional restoration system 1 of the above embodiment is used for measuring the body length of a dolphin, but can be applied to various scenes in which the size and shape of a curve are measured. For example, the present invention can be applied to a fish ecology survey, and the size and shape of various parts such as the fish body length, girth, and dorsal fin can be measured three-dimensionally as shown in the example of FIG. In addition, the dorsal fin has a jagged outline, but the user can draw a virtual line along the length of the dorsal fin, without strictly following the jagged outline, thereby providing a dorsal fin. The approximate size and shape can be measured. It is also possible to calculate the length of the thorns and the spacing between the thorns by three-dimensionally restoring the virtual curve connecting the tip of the spine thorn and the virtual curve passing through the root of the thorn and comparing them. FIG. 9B shows a leaf having a jagged outline, but in this example as well, the size and shape of the leaf can be measured. In the example of FIG. 9C, the virtual curve passing through the lower end of the upper tooth and the virtual curve passing through the upper end of the lower tooth are three-dimensionally restored, and the meshing can be evaluated by comparing them. . In addition, it is possible to evaluate the tooth arrangement from the curvature of such a curve, and it is also possible to evaluate the size of the entire gum by drawing a curve along the gum. As described above, the three-dimensional restoration method of the above embodiment that can three-dimensionally restore virtual lines that do not actually exist on the 2D image can be applied to various scenes.

  In addition, the computer 20 derives a plurality of 3D restoration curves by the three-dimensional restoration method of the above-described embodiment, and compares them, so that the subjects captured in the 2D image are not two-dimensionally but three-dimensionally. It can be determined whether or not they are similar. Therefore, it is possible to search for an image including a similar three-dimensional object from a large number of images. According to this application, it is possible to find similar products or counterfeits of copyrighted works, and to check the design of industrial products.

  In addition, the three-dimensional restoration method of the above embodiment can be applied not only to the measurement of the length of a curve, but also the measurement of the area of a curved surface that is an aggregate of curves.

<8-6>
In the above embodiment, the relative positional relationship between the cameras 11A, 11B,... Is corrected by the calibration unit 24H, but other parameters used for the calculation of the three-dimensional measurement are corrected by the same method. Also good. For example, instead of or in addition to the relative positional relationship between the cameras 11A, 11B,..., The focal lengths, lens distortions, and the like of the cameras 11A, 11B,. That is, the calibration unit 24H makes the four corners designated by the user on the same plane with respect to a plurality of 2D images, using the focal lengths of the cameras 11A, 11B,. Can be optimized. And the calibration part 24H should just preserve | save the optimal solution obtained in this way in the memory | storage part 23 as correction values, such as a focal distance of a camera 11A, 11B, ..., and lens distortion.

<8-7>
In the above embodiment, the moving images are shot by the cameras 11A, 11B,..., But a plurality of still images may be shot while synchronizing. However, in order to extract a set of a plurality of 2D images suitable for three-dimensional reconstruction for dolphins and the like that move freely in water, moving image shooting is preferable.

1 Three-dimensional restoration system (three-dimensional restoration device)
2A Three-dimensional restoration program 11A, 11B, ... Camera 20 Computer 21 Display unit

Claims (6)

A display control unit that displays on the display unit the first two-dimensional image and the second two-dimensional image obtained by photographing the subject from different angles;
A first drawing unit that allows a user to freely draw a first two-dimensional curve superimposed on the first two-dimensional image on the display unit;
A second drawing unit that allows a user to freely draw a second two-dimensional curve corresponding to the first two-dimensional curve superimposed on the second two-dimensional image on the display unit;
A first specifying unit that specifies a first control point group that is a control point group of the first two-dimensional curve or an approximate curve of the first two-dimensional curve;
A second specifying unit that specifies a second control point group that is a control point group of the second two-dimensional curve or an approximate curve of the second two-dimensional curve;
Based on the correspondence between the first control point group and the second control point group, a three-dimensional position of the first control point group and the second control point group is obtained, and based on the result, a three-dimensional curve is obtained. A three-dimensional restoration device comprising: a three-dimensional restoration unit that restores. The three-dimensional restoration apparatus according to claim 1, further comprising a calculation unit that calculates a length of the three-dimensional curve. Displaying a first two-dimensional image and a second two-dimensional image obtained by photographing a subject from different angles on a display unit;
Allowing the user to freely draw a first 2D curve overlaid on the first 2D image on the display;
Allowing the user to freely draw a second 2D curve corresponding to the first 2D curve superimposed on the second 2D image on the display unit;
Identifying a first control point group that is a control point group of the first two-dimensional curve or an approximate curve of the first two-dimensional curve;
Identifying a second control point group which is a control point group of the second two-dimensional curve or an approximate curve of the second two-dimensional curve;
Based on the correspondence between the first control point group and the second control point group, a three-dimensional position of the first control point group and the second control point group is obtained, and based on the result, a three-dimensional curve is obtained. A three-dimensional restoration program for causing a computer to execute the step of restoring. The three-dimensional restoration program according to claim 3, further causing the computer to execute a step of calculating the length of the three-dimensional curve. A method for restoring a three-dimensional shape of a subject along a virtual line that is difficult to recognize,
Capturing the first two-dimensional image and the second two-dimensional image obtained by photographing the subject from different angles into a computer;
The computer displaying the first two-dimensional image and the second two-dimensional image on a display unit;
A user freely draws a first two-dimensional curve that traces the virtual line on the first two-dimensional image on the display unit, and inputs the first two-dimensional curve to the computer; ,
A user freely draws a second 2D curve that traces the virtual line on the second 2D image on the display unit, and inputs the second 2D curve to the computer; ,
The computer specifying a first control point group which is a control point group of the first two-dimensional curve or an approximate curve of the first two-dimensional curve;
The computer specifying a second control point group which is a control point group of the second two-dimensional curve or an approximate curve of the second two-dimensional curve;
The computer obtains a three-dimensional position of the first control point group and the second control point group based on a correspondence relationship between the first control point group and the second control point group, and based on the result. And a step of restoring a three-dimensional curve. The computer further comprises calculating a length of the three-dimensional curve;
The three-dimensional restoration method according to claim 5.