JP5363878B2 - Stereo image photographing apparatus and method thereof - Google Patents

Abstract

Provided is a stereo image photographing device by which an overlapping area of a previously photographed image and a currently photographed image can be appropriately determined. The stereo image photographing device (1) sequentially photographs a measurement target (2) by a single camera so that the photographs overlap; wherein the stereo image photographing device (1) is provided with a photographing unit (3) which continually acquires live images (25) of the measurement target (2) corresponding to the positional movement of the camera, and photographs the measurement target (2) at an appropriate time to acquire a photographed image which is a still image; an overlapping area setting unit (6) which sets an overlapping area (23) between a previously photographed image (21) and the live image (25) on the previously photographed image (21) which is a photographed image previously photographed by the photographing unit (3); and a display unit (4) which displays the previously photographed image (21) and the live image (25) by overlapping the images at the overlapping area (23) set by the overlapping area setting unit (6), displays the previously photographed image (21) so as to extend from the overlapping area (23) in the direction opposite to the moving direction of the photographing unit (3), and displays the live image (25) so as to extend from the overlapping area (23) in the same direction as the moving direction of the photographing unit (3).

Description

  The present invention relates to a stereo image photographing apparatus and a method thereof. More specifically, the present invention relates to a stereo image capturing apparatus and method for appropriately determining an overlapping area between a previously captured image and a current captured image.

  In order to grasp the whole image of the measurement object and reproduce it as a three-dimensional model image, it is necessary to link the photographed images taken from a plurality of photographing positions. In order to measure the three-dimensional coordinates of the photographing apparatus or the object from the plurality of photographed images taken by the photographer in this way, the feature points (on the object) corresponding to each other on each of the two or more photographed images. Represents the same point) and needs to be tracked. In this case, since feature points inappropriate for three-dimensional measurement can be mixed in the captured image, it is possible to accurately determine the capturing position, orientation, or position coordinates of the target object of the capturing apparatus while determining the suitability of the feature points in the captured image. An image processing apparatus capable of measurement has been proposed. (See Patent Document 1)

JP 2007-183256 A (FIGS. 1 to 11, paragraphs 0021 to 0065)

  When taking a picture while moving using a single camera and performing three-dimensional measurement of the measurement object, or when creating an all-around image in which the photographed image is linked over 360 degrees around the measurement object, It is necessary to shoot while overlapping images and connect the images continuously. However, if the degree of overlap between the captured images is reduced, the efficiency of the image deteriorates due to the fact that the overlapping state of the images necessary for 3D measurement and creation of all-around images is unclear, or shooting is lost. was there. On the other hand, if the degree of overlap between the captured images is increased, there is a problem that the amount of time and labor for image processing and work increases, and as a result, the efficiency is deteriorated. For this reason, when shooting using a single camera while moving, the overlapping area between the previous shot image that is the previous shot image and the current shot image that is scheduled to be shot this time is appropriately determined. There has been a need for a stereo image capturing apparatus and a method thereof.

  The present invention has been made in view of the above problems, and a stereo image photographing apparatus capable of appropriately determining an overlapping region between a previous photographed image and a current photographed image when photographing while moving using a single camera. And an object thereof.

  In order to solve the above-described problem, the stereo image capturing apparatus 1 according to the first aspect of the present invention, for example, as shown in FIG. 1 (see FIG. 2 for images), overlaps the measurement object 2 with a single camera. In the stereo image capturing apparatus 1 that sequentially captures, a capturing unit that acquires the live image 25 of the measurement object 2 as needed with the movement of the camera, captures the measurement object 2 in a timely manner, and acquires a captured image that is a still image. 3, the overlap area setting section 6 that sets the overlap area 23 with the live image 25 in the previous captured image 21 that is the captured image previously captured by the capturing section 3, and the previously captured image 21 and the live image 25 are overlapped. The overlapping area 23 set by the area setting unit 6 is displayed in an overlapping manner, the previous captured image 21 is displayed by extending from the overlapping area 23 in the direction opposite to the moving direction of the imaging unit 3, and the live image 25 is displayed from the overlapping area 23. Shoot Extend on the same side as the direction of movement of the part 3 and a display unit 4 for displaying.

  Here, the stereo image photographing device refers to a photographing device that normally includes a pair of cameras and shoots a measurement object in an overlapping manner. However, in the present invention, since a single camera is photographed while moving, the measurement object is overlapped. An imaging apparatus that sequentially captures images while performing the above. The live image 25 is an image acquired at the current position using a camera such as a video camera or a digital camera that can acquire and display images continuously, and dynamically changes as the camera moves. Typically, the live image 25 is overwritten in the temporary memory and transferred to the display unit 4. The live image 25 is sampled at an appropriate frame interval or time interval, stored in a memory for a predetermined period (temporarily), and used for image processing. Further, the live image 25 at the current position is acquired as the current captured image 22 which is a captured image scheduled to be captured this time by the shutter operation. Therefore, in the display area 26 of the display unit 4, if the image of the overlapping area (live image overlapping area 25 a) in the live image 25 is approximately coincident with the image of the overlapping area 23 in the previous captured image 21, it is planned. The captured image 22 can be obtained this time. The overlap area 23 is set by setting the overlap ratio and the overlap direction for the previously captured image 21. The overlap ratio is set, for example, in% and can be set to an arbitrary value. The overlapping direction is typically set to one of left side, right side, upper side, and lower side. Although the upper left side, the lower left side, the upper right side, and the lower right side may be set, in this case, the overlapping ratio is set as a horizontal ratio and a vertical ratio. In addition, the overlap typically overlaps a semi-transparent image, but the transparency (density) of the semi-transparent image can be arbitrarily selected. For example, when three images are overlapped, if the density is set low, the overlapping portion of the three images becomes dark and easy to distinguish. For example, if the moving direction of the photographing unit 3 is on the right side, displaying the previous captured image 21 from the overlapping region 23 and the overlapping region 23 and displaying the previous captured image 21 extending from the overlapping region 23 to the opposite side to the moving direction of the photographing unit 3. This means that the live image 25 is displayed on the left side and extended from the overlap region 23 to the same side as the moving direction of the photographing unit 3 to display the live image 25 on the overlap region 23 and the right side thereof. means. The term “approximately coincidence” means that it is sufficient that they coincide within the margin even if they do not coincide completely. This is because if the overlapping area is set with a margin, there is no problem if the overlapping areas match. For example, since the overlap ratio may be set to a little over 50% for three-dimensional measurement, if the overlap ratio is set to 60%, a deviation of ± 5% is not a problem, and a tolerance of slightly less than ± 10% is acceptable. Even if a deviation occurs within the margin, it can be sufficiently recovered by intentionally shifting the image in the reverse direction in the next image capturing. Of course, it is preferable to match as much as possible in order to maintain the uniformity of the image processing, for example, within ± 3%.

  With this configuration, the live image 25 can be displayed in the display area 26 of the display unit 4 so as to overlap the overlap region 23 of the previous captured image 21, and the appropriate shooting position for the camera to capture the current captured image 22. You can visually determine whether or not Thereby, in the case of shooting while moving using a single camera, it is possible to provide the stereo image shooting device 1 in which the overlapping area 23 between the previous shot image 21 and the current shot image 22 can be appropriately determined.

In the first aspect, the stereo image photographing device 1 according to the second aspect of the present invention displays the photographed image and the live image 25 with a frame in the first aspect, for example, as shown in FIG. .
In this case, the captured image includes, in addition to the previous captured image 21, a previously captured image, an initial captured image, and the like. If comprised like this aspect, the range of a picked-up image and the live image 25 can be displayed clearly.

In the stereo image capturing device 1 according to the third aspect of the present invention, in the first or second aspect, the overlapping area setting unit 6 sets the overlapping area 23 to 50 to 70% of the previous captured image 21.
Here, the overlap ratio is preferably, for example, 10% or more when creating a continuous image such as a panoramic image or an all-around image, but for measuring the three-dimensional shape of the measurement object 2, for example, 50 to 70%. Is preferable, and 55 to 65% is more preferable. Moreover, it can be set to any value depending on the situation. If comprised like this aspect, all the imaging | photography areas of a measurement target object can be imaged redundantly, and the measurement of the three-dimensional shape of the measurement target object 2 will be attained.

Moreover, the stereo image photographing device 1 of the fourth aspect of the present invention is the measurement object 2 in the previous photographed image 21 and the live image 25 in any of the first to third aspects, for example, as shown in FIG. The feature point extracting unit 7 for extracting the feature points of the camera, and the camera moving direction determination for obtaining the moving direction of the photographing unit 3 from the screen positions of the feature points extracted by the feature point extracting unit 7 in the previous captured image 21 and the live image 25 The overlapping region setting unit 6 sets the overlapping region 23 in the previous captured image 21 on the same side as the moving direction determined by the camera moving direction determining unit 8.
If comprised in this way, the direction of an overlap area can be set correctly, without adding equipment, such as a sensor.

Further, the stereo image photographing device 1B according to the fifth aspect of the present invention has a GPS or an inertial sensor as shown in FIG. 19 in any of the first to third aspects, and moves the photographing unit 3. A moving direction detecting unit 15 that detects a direction is provided, and the overlapping region setting unit 6 sets the overlapping region 23 in the previous captured image 21 on the same side as the moving direction detected by the moving direction detecting unit 15.
If comprised in this way, the direction of an overlap area | region can be set rapidly and correctly using GPS or an inertial sensor.

Further, the stereo image photographing device 1C according to the sixth aspect of the present invention can input in advance the moving direction of the photographing unit 3 in any one of the first to third aspects, for example, as shown in FIG. The overlapping area setting unit 6 sets the overlapping area 23 in the previous captured image 21 on the same side as the input moving direction.
If comprised in this way, an overlapping area | region can be set flexibly according to a photographer's judgment according to a condition.

  Further, the stereo image photographing device 1A according to the seventh aspect of the present invention is the measurement object 2 in the previous photographed image 21 and the live image 25 as shown in FIG. 10, for example, in any of the first to sixth aspects. Obtained by the feature point extracting unit 7 for extracting the feature points of the camera, the orientation unit 12 for obtaining the position and inclination of the camera of the photographing unit 3 using the feature points extracted by the feature point extracting unit 7, and the orientation unit 12. The display unit 4 includes a displacement correction image forming unit 13 that performs a displacement correction process on the previously captured image 21 and the live image 25 using the position and tilt of the camera to form a displacement correction image. The displacement correction image formed by the displacement correction image forming unit 13 is displayed.

  Here, the deviation correction processing is performed on the subject so that the stereo image geometry is established with respect to the measurement object 2 using the photographing position and inclination of the camera obtained by the orientation. This is a process of correcting the image so that the epipolar line coincides with the left and right horizontal lines in parallel (see FIG. 12). Therefore, distortions such as magnification and inclination of the left and right images are corrected, and the positions of the magnification and height are the same. When configured as in this aspect, by creating a deviation correction image, the previously captured image 21 and the live image 25 can be displayed in a superimposed manner in a stereoscopically viewable manner, and an image that is almost the same can be obtained when overlapping. .

  In addition, the stereo image photographing device 1 according to the eighth aspect of the present invention is a photographed image obtained by the overlapping region setting unit 6 previously photographed as shown in FIG. 8, for example, in any of the first to seventh aspects. The overlap area set in the previous shot image 27 is the previous image overlap area 27a, the overlap area set in the previous shot image 21 is the previous image overlap area 21a, and the display unit 4 rotates the previous shot image 27 and the previous shot image 21 in advance. The image overlap area 27a is displayed in an overlapping manner, the previous image captured image 27 is displayed extending from the image overlap area 27a to the side opposite to the moving direction of the image capturing unit 3, and the previous image image 21 is displayed from the image overlap area 27a before the image capture area. 3 is extended and displayed on the same side as the moving direction 3, the previous captured image 21 and the live image 25 are superimposed on the previous image overlap area 21 a, and the previous captured image 21 is displayed on the previous image overlap. Displays to extend from the region 21a on the side opposite to the moving direction of the imaging unit 3, and displays the extend on the same side as the moving direction of the imaging unit 3 live images 25 from the previous image overlap region 21a.

  In this case, the previous-second-time image overlap area 27a is already set in the previous-time-shot image 27 by the overlap area setting unit 6 at the time of the previous shooting. The moving direction of the photographing unit 3 refers to the moving direction from the photographing position of the previous photographed image 21 to the photographing position of the live image 25, and typically the previous photographed image from the photographing position of the last photographed image 21 and the last photographed image 27. This corresponds to the moving direction to the photographing position 21. If they do not match, it is sufficient to shoot after correcting the movement direction of the photographer. For example, if the moving direction of the photographing unit 3 is on the right side, displaying the previous captured image 21 from the previous image overlapping area 21a to the opposite side to the moving direction of the photographing unit 3 is displayed as the previous captured image 21. This means that the live image 25 is displayed on the overlap area 21a and the left side thereof, and the live image 25 is extended from the previous image overlap area 21a to the same side as the moving direction of the photographing unit 3 and displayed. And on the right side. When configured in this manner, the previously captured image 27, the previous captured image 21, and the live image 25 can be displayed in an overlapped manner on the display area 26, and the tie region 29 that is tripled can be confirmed. Also, “overlapping” means that they do not have to be completely matched, but need only be matched within the margin. For example, if the image overlap area 27a is set with a margin for overlapping with the live image 25, the image overlap area 27a is not problematic as long as it matches within the margin. For example, since it is sufficient if there is an overlap area for three-dimensional measurement, if the overlap ratio is set to 20%, a deviation of ± 5% is not a problem, and a tolerance of ± 20% is acceptable. Even if a deviation occurs within the margin, it can be sufficiently recovered by intentionally shifting the image in the reverse direction in the next image capturing. Of course, it is preferable to match as much as possible in order to maintain the uniformity of the image processing, for example, within ± 3%.

Further, in the eighth aspect of the stereo image capturing device 1 of the ninth aspect of the present invention, as shown in FIG. 8, for example, the overlap area setting unit 6 includes the image overlap area 27a and the previous image overlap area 21a. The image overlap area 27a and the previous image overlap area 21a are set the previous time so as to overlap at least partially.
If comprised in this way, by this, all the imaging regions of the measuring object 2 can be imaged, and the three-dimensional shape of the measuring object 2 can be measured.

Further, the stereo image photographing device 1A according to the tenth aspect of the present invention, in any of the first to ninth aspects, measures the previous photographed image 21 and the live image 25 as shown in FIGS. 10 and 16, for example. The feature point extraction unit 7 that extracts the feature points of the target object 2, and the feature point extraction unit 7 extracts the feature points of the measurement target object 2 extracted from the live image 25 and the first shot image 28 that is a first shot image. The corresponding point search unit 8a that performs stereo matching with the feature point of the measurement object 2 extracted from the image, and when an overlapping region is found between the live image 25 and the first captured image 28 by stereo matching, The area setting unit 6 sets the overlapping area in the first shot image 28 as the one-round image overlapping area 28c, and the display unit 4 sets the first shot image 28 and the live image 25 to the one-round image overlapping area 28c. Cascade to display.
With this configuration, it can be seen that when the first shot image 28 is displayed, shooting is completed in the current or next shooting.

Further, the stereo image photographing device 1A according to the eleventh aspect of the present invention, in any one of the first to ninth aspects, measures the previous photographed image 21 and the live image 25 as shown in FIGS. 10 and 16, for example. The feature point extraction unit 7 that extracts the feature points of the target object 2, the live image 25 extracted by the feature point extraction unit 7, and the feature points of the measurement target object 2 in the first captured image 28 that is the first captured image The three-dimensional coordinate calculation unit 14 for obtaining the three-dimensional coordinates of the respective photographing positions from the screen positions of the live image 25 and the first-captured image 28 based on the three-dimensional coordinates of the photographing position in the first-captured image 28. When the overlapping area is found in the first captured image 28, the overlapping area setting 6 sets the overlapping area as the one-round image overlapping area 28c, and the display unit 4 And 28 and live images 25 displayed superimposed in round image overlap region 28c.
With this configuration, it can be seen that when the first shot image 28 is displayed, shooting is completed in the current or next shooting.

Further, the stereo image photographing device 1B of the twelfth aspect of the present invention, in the fifth aspect, as shown in FIG. 19, for example, a live image 25 detected by a GPS or an inertial sensor and a photographed image photographed for the first time. When an overlap area is found between the live image 25 and the first shot image 28 from the shooting position in the first shot image 28, the overlap area setting unit 6 overlaps the first shot image 28 with the overlap area. The area 28c is set, and the display unit 4 displays the first shot image 28 and the live image 25 so as to overlap each other in the round image overlap area 28c.
Here, “turning once” means moving to the same direction and reaching a position where shooting from all directions planned by the current shooting is completed. For example, when shooting from 12 directions is scheduled, this means that the 12th shooting position has been reached. When configured in this manner, it can be seen that when the first shot image 28 is displayed, shooting is completed in the current or next shooting.

Further, in the sixth aspect, the stereo image photographing device 1C according to the thirteenth aspect of the present invention can input the fact that the photographer has made one round as shown in FIG. 21, for example, as shown in FIG. Is input, the overlap region setting unit 6 sets the one-round image overlap region 28c in the first shot image 28 which is the first shot image, and the display unit 4 The live image 25 and the live image 25 are displayed so as to overlap each other in the round image overlap area 28c.
With this configuration, it can be seen that when the first shot image 28 is displayed, shooting is completed in the current or next shooting.

  In addition, the stereo image capturing device 1A according to the fourteenth aspect of the present invention, in any one of the first to ninth aspects, measures the previous captured image 21 and the live image 25 as shown in FIGS. 10 and 18, for example. A feature point extraction unit 7 that extracts feature points of the object 2, a location unit 12 that obtains the position and inclination of the camera of the photographing unit 3 using the feature points extracted by the feature point extraction unit 7, and a location unit 12 A three-dimensional coordinate calculation unit 14 for obtaining the three-dimensional coordinates of the feature points using the obtained camera position and tilt, and the overlapping area setting unit 6 includes the feature points obtained by the three-dimensional coordinate calculation unit 14. Using the three-dimensional coordinates, the overlap area between the previous captured image 21 and the live image 25 is set as the previous overlap area 21c in the previous captured image 21 and the live overlap area 25c in the live image 25, and the display unit 4 , Last time The image 21 and the live image 25 are included in the display area 26 so that the same portion of the measurement object 2 substantially overlaps in the previous overlap region 21c and the live overlap region 25c, and both are attached with a frame. To display.

  Here, for example, the previous overlap region 21c and the live overlap region 25c may be represented by a quadrangle that approximates and overlaps the overlapping outermost feature points. If comprised like this aspect, the image of the measuring object 2 in the live side overlapping area 25c can be displayed more accurately. Further, by comparing the positions of the overlapping area 23 and the live side overlapping area 25c, it can be determined whether or not to shoot. When the deviation is large, the live side overlapping area 25c may be moved closer to the overlapping area 23, and both areas may be photographed together. Further, according to the position of the live-side overlap area 25c, that is, according to the shooting direction, the previous shot image 21, the live image 25, and the positions of these frames are changed and displayed.

  In order to solve the above-described problem, a stereo image capturing method according to a fifteenth aspect of the present invention is a stereo image capturing method for sequentially capturing an object to be measured 2 with a single camera as shown in FIG. A photographing step (S110) for photographing the measurement object 2 in a timely manner to obtain a photographed image that is a still image, and a live image obtaining step (S140) for obtaining a live image of the measurement object 2 at any time as the camera moves. And an overlapping area setting step (S130 and S180) for setting an overlapping area 23 with the live image 25 on the previous captured image 21 that is the previously captured image, and an overlapping area between the previously captured image 21 and the live image 25. The overlapping area 23 set in the setting step (S130 and S180) is displayed in an overlapping manner, and the previous captured image 21 is displayed on the side opposite to the moving direction of the imaging unit 3 from the overlapping area 23. Extended to display, and a display step (S190) of displaying extend on the same side as the moving direction of the imaging unit 3 live images 25 from the overlap region 23.

  When configured in this manner, the live image 25 can be displayed in the display area 26 of the display unit 4 so as to be superimposed on the overlapping area 23 of the previous captured image 21, so that the camera can capture images suitable for capturing the current scheduled image 22. You can visually determine whether or not you are in position. Thus, when shooting is performed while moving using a single camera, it is possible to provide a stereo image shooting method in which an overlapping region 23 between the previous shot image 21 and the current shot image 22 can be appropriately determined.

  ADVANTAGE OF THE INVENTION According to this invention, when image | photographing, moving using a single camera, the stereo image imaging device and its method which can define appropriately the duplication area | region of the last captured image and this captured image can be provided.

1 is a block diagram illustrating a configuration example of a stereo image capturing device according to Embodiment 1. FIG. It is a figure for demonstrating the duplication area | region in a display area. It is a figure which shows the example (the 1) which overlaps and displays a live image on the last picked-up image. It is a figure which shows the example (the 2) which overlaps and displays a live image on the last picked-up image. It is a figure which shows the example of the feature point and corresponding point in a stereo image. It is a figure for demonstrating corresponding point search. FIG. 5 is a diagram illustrating an example of a processing flow of image capturing according to the first embodiment. It is a figure which shows the example which displays a live image overlappingly on the last time photographed image and the last photographed image. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing according to a second embodiment. FIG. 10 is a block diagram illustrating a configuration example of a stereo image capturing apparatus according to a third embodiment. It is a figure for demonstrating mutual orientation. It is a figure for demonstrating a deviation correction image. It is a figure for demonstrating the magnification of a picked-up image and a display image. It is a figure for demonstrating the stereo method. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing according to a third embodiment. It is a figure which shows the example which overlaps and displays a live image on a first picked-up image. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing according to a fourth embodiment. It is a figure which shows the example displayed by changing the position of the last picked-up image, a live image, and these frames according to the imaging direction. FIG. 10 is a block diagram illustrating a configuration example of a stereo image photographing device according to Embodiment 6. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing according to a sixth embodiment. FIG. 10 is a block diagram illustrating a configuration example of a stereo image capturing device according to Embodiment 7. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing according to a seventh embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

  The first embodiment is an example in which the previously captured image and the live image are overlapped and displayed in the display area of the display unit. The camera moving direction determining unit obtains the moving direction of the photographing unit, and the overlapping region setting unit determines the camera moving direction. An example in which an overlapping area is set on the same side as the movement direction determined by the section will be described.

[Device configuration]
FIG. 1 is a block diagram showing a configuration example of a stereo image photographing apparatus 1 according to the first embodiment of the present invention. The stereo image photographing device 1 obtains a live image of the measurement object 2 as needed as the camera moves, captures the measurement object 2 in a timely manner, and obtains a still image as a still image. And the display unit 4 for displaying the image processed image, the input unit 9 for the operator to input data and instructions, the storage unit 5 for storing the captured image, the live image, the image processed image, etc. An overlapping area setting unit 6 that sets an overlapping area with a live image on a previous captured image that is a captured image, a feature point extracting unit 7 that extracts feature points from the captured image and the live image, and a feature point extracting unit 7 From the screen position of the extracted feature point in the previous captured image and the live image, the camera movement direction determination unit 8 that determines the moving direction of the photographing unit 3, the stereo image photographing device 1, and the control unit 10 that controls each of the components. Provided. Further, the overlapping area setting unit 6, the feature point extraction unit 7, the camera movement direction determination unit 8, and the control unit 10 are configured in a personal computer (PC) 16. Each of these units may be configured as a processing unit in the digital camera, but in the present embodiment, it is configured in the PC 16.

The stereo image photographing apparatus 1 according to the present embodiment is suitable for obtaining a whole image of the measurement object 2 by connecting a large number of photographed images, for example, photographing the entire circumference of the measurement object 2. In addition, it is suitable for obtaining a photographed image for performing three-dimensional measurement by performing duplicate photographing over the entire measurement object 2. Refer to FIG. 2 for images.
The photographing unit 3 is composed of a single camera such as a video camera or a digital camera. The photographer uses a single camera as the photographing unit 3 to sequentially capture images while providing an overlapping region while moving around the measurement object 2, for example. In order to obtain an all-around image connected 360 degrees around the object 2 to be measured, it is only necessary to provide an overlap region between adjacent photographed images, but for three-dimensional measurement, the overlap region is shared. In order to extract the feature points from the overlapping areas and to obtain the three-dimensional coordinates of these feature points, it is necessary to capture the entire shooting area of the measurement object 2 in duplicate. is there.

The display unit 4 includes a display such as a liquid crystal display, for example. At least the previously captured image 21 and the live image 25 of the measurement object 2 are displayed in the display area 26 so as to overlap each other. The display area 26 can display a range wider than the shooting range, and the range shot by the shutter operation is the range displayed as the live image 25. In addition, the display unit 4 may display an image that has been subjected to image processing, such as a previously captured image and a displacement corrected image. In addition, a speaker is provided for voice display.
The input unit 9 includes, for example, a mouse and a keyboard, and is used for an operator to input data and instructions.

  The storage unit 5 is composed of, for example, a hard disk and is used as a database. A captured image storage unit 51 that stores the captured image, a live image storage unit 52 that stores the live image 25, a processed image storage unit 53 that stores an image-processed image such as a displacement corrected image, and a feature point of the measurement object 2 A position coordinate storage unit 54 that stores the position coordinates of the photographing position is provided. In a video camera, a photographed image is usually stored in an external memory by a shutter operation, but if there is an internal memory, it is stored there. In a digital camera, a photographed image is stored in an internal memory by a shutter operation. In addition, live images taken by the video camera or digital camera while the photographer moves are always overwritten in the temporary memory of the live image storage unit 52 and transferred to the display unit 4. Further, image processing is required to determine the moving direction from the live image. Therefore, the live image is sampled at an appropriate frame interval or time interval, and stored in the memory of the live image storage unit 52 for a predetermined period (temporarily). The predetermined period may be a time sufficient for determining the moving direction, for example. Therefore, the live image storage unit 52 has a temporary memory and a memory for storing the sampled live image. Although the internal memory of the camera may be used as the storage unit 5, it is preferable to use the hard disk of the PC 16 because it is suitable for various processes at high speed. In this embodiment, a camera using an external memory stores a captured image directly in the storage unit 5, and a camera using an internal memory transfers the captured image to the storage unit 5.

[Overlapping area setting section]
The overlapping area setting unit 6 sets the overlapping area 23 with the live image 25 in the previous captured image 21 that is the captured image previously captured by the capturing unit 3. This overlapping area 23 is also an area in which the previously captured image 21 and the current captured image 22 which is the captured image scheduled to be captured this time are overlapped and captured. In the display area 26 of the display unit 4, if shooting is performed when the image of the live image overlap area 25 a that is an overlap area in the live image 25 substantially coincides with the image of the overlap area 23 in the previous shot image 21, the scheduled current shooting is performed. An image 22 can be obtained. The overlapping area setting unit 6 sets the overlapping ratio and the overlapping direction for the overlapping area 23. With respect to the overlapping direction, in this embodiment, the overlapping area 23 is set on the same side as the moving direction determined by the camera moving direction determination unit 8. For example, when the photographer has the photographing unit 3 and moves clockwise around the measurement object 2, the overlapping region 23 is set and displayed on the right side of the previous photographed image 21. The display area 26 can display a range wider than the shooting range, and the range shot by the shutter operation is the range displayed as the live image 25. Therefore, if shooting is performed when the image of the live image overlapping area 25a, which is the overlapping area in the live image 25, does not match the image of the overlapping area 23 in the previous captured image 21, a captured image that deviates from the scheduled current captured image 22 is obtained. can get. By the way, “matching roughly” means that it is sufficient that they match within the margin even if they do not match completely. This is because if the overlapping area is set with a margin, there is no problem if the overlapping areas match. For example, since the overlap ratio may be set to a little over 50% for three-dimensional measurement, if the overlap ratio is set to 60%, a deviation of ± 5% is not a problem, and a tolerance of slightly less than ± 10% is acceptable. Even if a deviation occurs within the margin, it can be sufficiently recovered by intentionally shifting the image in the reverse direction in the next image capturing. Of course, it is preferable to match as much as possible in order to maintain the uniformity of the image processing, for example, within ± 3%. In addition, the overlap ratio can be arbitrarily set according to the situation. For example, when it is desired to finely observe the measurement object 2 or to finely track the feature point, it is set to 90%. When the measurement object 2 may be coarsely observed, the feature point is coarsely tracked. For example, it may be set to 55%.

  FIG. 2 is a diagram for explaining an overlapping area in the display area. FIG. 2A shows the overlapping state in the display area 26, and FIG. A car as the measurement object 2 is displayed in a display area 26 (a region surrounded by a thick solid line frame on the outside in FIG. 2A) of the display unit 4 attached to the digital camera as the photographing unit 3. The inside solid line frame is the previous photographed image 21, which is arranged on the left side of the display area 26, here near the left end, and the broken line frame is the live image 25, on the right side of the display area 26, here near the right end. Has been placed. That is, this is a case where the moving direction of the photographing unit 3 is the right direction. In the center of the display area 26, there is a portion within the inner solid line frame and within the broken line frame, which is an overlapping region (overlap region) 23 between the previous captured image 21 and the live image 25. This overlapping area 23 is also an area in which the previously captured image 21 and the current captured image 22 which is the captured image scheduled to be captured this time are overlapped and captured. The overlap ratio is about 60%. The previously captured image 21 and the live image 25 are displayed so as to overlap each other in the overlapping area 23, and the previous captured image 21 is displayed extending from the overlapping area 23 to the left (opposite to the moving direction of the capturing unit 3). Is displayed extending from the overlapping area 23 to the right side (the same side as the moving direction of the photographing unit 3). In the previous captured image 21, the extended portion is the extended region 24, and in the live image 25, the portion in the overlapping region 23 is the live image overlapping region 25a, and the extended portion is the live image extended region. 25b. If shooting is performed when the image of the measurement object 2 in the live image overlap area 25a substantially coincides with the image of the measurement object 2 in the overlap area 23, the scheduled current captured image 22 can be obtained.

  FIG. 3 shows an example (part 1) in which a live image is displayed in an overlapping manner with a previously captured image. The measurement object 2 is an automobile. FIG. 3A shows the previously captured image 21, FIG. 3B shows the current captured image 22 scheduled to be captured this time, FIG. 3C shows the live image 25, and FIG. 3D shows the previously captured image 21. Shows an image in which the live image 25 is overlapped. Refer to FIG. 2 for image codes. In FIG. 3A, the overlap region 23 is set to the right half of the broken line in the previous photographed image 21, and the left half is the extension region 24. In the current captured image 22 in FIG. 3B and the live image 25 in FIG. 3C, the left half is a portion that overlaps the overlapping region 23 set in the previous captured image 21. That is, in the live image 25, the left half is the live image overlap area 25a, and the right half is the live image extension area 25b. In FIG. 3D, the previous captured image is arranged on the left 2/3 slightly, the live image is disposed on the right 2/3, and the previously captured image 21 and the live image 25 are displayed in the overlap area 23 of the center 1/3 slightly. Duplicate display. In the figure, the live image 25 is surrounded by a dotted line and is displayed slightly shifted upward from the previously captured image 21. If the position and angle of the camera are adjusted and shooting is performed when the same portion of the measurement object 2 in the previous captured image 21 and the live image 25 substantially overlaps in the overlapping region 23, the scheduled current captured image 22 can be obtained. it can.

  FIG. 4 shows an example (part 2) in which a live image is displayed in an overlapping manner with a previously captured image. The measurement object 2 is an automobile. FIG. 3 shows an example in which the previous captured image 21 and the live image 25 are displayed side by side, but FIG. 4 shows an example in which the previous captured image 21 and the live image 25 are displayed side by side. In the figure, the live image 25 is shown on the lower side in a direction surrounded by a dotted line. The range of the center arrow is the overlapping region 23. The live image 25 is displayed slightly shifted to the right side. As in the case of FIG. 3, if the position and angle of the camera are adjusted and shooting is performed when the same portion of the measurement object 2 in the previous captured image 21 and the live image 25 substantially overlaps in the overlapping region 23, the scheduled operation is performed. The captured image 22 can be obtained this time.

[Feature point extraction unit]
The feature point extraction unit 7 extracts feature points from the captured image. The feature points include, for example, the center position, the center of gravity position, the corner position of the measurement object 2, a position having a different characteristic from others, and a sign affixed or projected on the measurement object 2. A feature extraction operator is used for feature point extraction. Here, an example in which a Moravec operator is used will be described.

  The MORAVEC operator has long been used as a general-purpose feature extractor. The MORAVEC operator uses, for example, a 3 × 3 pixel around a certain target pixel as a mask, and sets the minimum value of the density difference (direction density difference) when the mask moves one pixel in each of the four directions around the target pixel. The feature value. It is characterized by simple and high-speed processing and relatively easy hardware. In order to perform high-speed processing, a memory several times as large as an image is required. Although the feature point extraction by the Moravec operator has been described here, other operators, for example, the Harris operator and others, and anything that can detect feature points may be used.

[Camera moving direction determination unit]
The camera movement direction determination unit 8 moves the photographing unit 3 from the screen position of the feature point extracted by the feature point extraction unit 7 in the previous captured image 21 and the live image 25 or from the change in the screen position between the live images 25. Find the direction. That is, using the previously captured image 21 and the live image 25 as a stereo image, the feature points of the previously captured image are associated with the feature points of the live image, and the movement vector of the feature point of the live image 25 with respect to the feature point of the previously captured image 21 is obtained. It can be obtained and averaged in the overlapping area, and the direction opposite to the average movement vector can be obtained as the moving direction of the photographing unit 3. Correspondence of feature points (corresponding point search) is performed by the corresponding point search unit 8a. It is also possible to track a feature point of a live image, obtain an average movement vector from changes in feature points between live images, and obtain a moving direction when movement in the same direction has continued a predetermined number of times (for example, three times or more). good. Note that the predetermined number of times may be one time, and tracking of feature points may be omitted. When a live image is shot by operating the shutter at a position that is too far from the planned shooting position, it is preferable that the photographed image can be discarded by inputting a discard signal by the photographer. After that, it is preferable for the photographer to move back to the shooting position of the previous captured image in order to ensure the determination by the camera movement direction determination unit 8.

  FIG. 5 is a diagram illustrating an example of feature points and corresponding points in a stereo image. FIG. 5A shows feature points extracted from the left image, FIG. 5B shows feature points extracted from the right image, and FIG. 5C shows associated feature points. The left image and the right image form a stereo pair, and are photographed at a position slightly shifted left and right. In the left image and the right image, the positions of many points attached to the automobile as the measurement object 2 are the positions of the points extracted as feature points. Most of these feature points are associated by the corresponding point search, and are displayed in a three-dimensional coordinate space (here, a perspective view is used) in FIG. These feature points are assigned feature point numbers, and the three-dimensional position coordinates and the position coordinates in the left and right images are stored in the position coordinate storage unit 54.

FIG. 6 is a diagram for explaining a corresponding point search for associating feature points between stereo images. Cross-correlation processing is used for the association. As shown in FIG. 6, the template image of N ₁ × N ₁ pixel is moved on the search range (M ₁ −N ₁ +1) ² in the input image of M ₁ × M ₁ pixel larger than that, and the following C The upper left position (a, b) of the template image that maximizes (a, b) is obtained, and it is considered that the template image has been searched. In the case of left and right images, for example, a template image of N ₁ × N ₁ pixel is set on the left image, a search region of M ₁ × M ₁ pixel is set on the right image, and this operation is performed on each image. What is necessary is just to do about a position.

[Correlation processing]
Here, the normalized cross-correlation processing has been described with respect to the corresponding point search, but other methods, for example, a residual sequential test method (SSDA) may be used.

  The control unit 10 has a control program in the built-in memory, controls the stereo image capturing device 1 and each unit constituting the same, controls the flow of signals and data, and executes functions as a stereo image capturing device.

FIG. 7 shows an example of a processing flow of image capturing according to the present embodiment. An example in which the previously captured image and the live image are displayed in the display area in an overlapping manner will be described.
First, the measurement object 2 is imaged by the shutter operation of the imaging unit 3 (imaging process: S110). The captured image is stored in the captured image storage unit 51. Next, the captured image is displayed in the display area 26 of the display unit 4 (captured image display step: S120), and the overlapped region setting unit 6 captures the previous captured image that was captured in the capturing step and displayed in the captured image display step. An overlap ratio is set in 21 (overlap ratio setting step: S130). The overlap area 23 is set for the overlap ratio and the overlap direction. The overlapping area setting process includes an overlapping ratio setting process (S130) and an overlapping direction setting process (S180) described later. Here, the overlap ratio is set first, but the overlap direction is set temporarily. For example, when “right (side)” and “60 (%)” are input from the input unit 9 using the keyboard, the overlap area 23 in the previous captured image 21 displayed in the display area 26 in the captured image display step (S120). Is displayed surrounded by a frame. When the photographer clicks a confirmation button on the display screen with a mouse, for example, the overlap ratio is set, and the overlap direction is temporarily set. The overlapping direction is set in the subsequent overlapping direction setting step (S180). Next, a live image is acquired (live image acquisition step: S140). The live image is always overwritten in the temporary memory of the live image storage unit 52 and transferred to the display unit 4. In addition, sampling is performed at an appropriate frame interval or time interval and stored in the live image storage unit 52 for a predetermined period (temporarily). For example, when the overlapping ratio is set to “60 (%)” and the overlapping direction is temporarily set to “right (side)”, 60% of the previous captured image 21 becomes the overlapping area 23 and is displayed in the display area 26. The previous captured image 21 is displayed extending from the overlapping area 23 to the left side (ie, the overlapping area and the left side thereof), and the live image is displayed extending from the overlapping area to the right side (ie, the overlapping area and its area). On the right). Further, for example, the temporary setting of the overlapping direction may be omitted, and the live image 25 may be displayed in the center of the display area 26 until the overlapping direction is finally set.

Next, feature points are extracted from the previous captured image 21 and live image 25 (feature point extraction step: S150). For the live image 25, for example, an image after a predetermined time (for example, 3 sec) has elapsed since the last captured image 21 was captured from images sampled at an appropriate frame interval or time interval, and feature points are extracted. Using the previously captured image 21 and the live image 25 as a stereo image, the corresponding point search unit 8a associates the overlapping feature points. The camera movement direction determination unit 8 captures a direction opposite to the average movement vector from the average movement vector of the corresponding point of the live image 25 with respect to the feature point of the previously captured image 21 (the feature point associated with the corresponding point search unit 8a). It can be obtained as the moving direction of the part 3. Next, feature points are tracked for the live image 25 (feature point tracking step: S160). For example, a live image after 6 sec and a live image after 9 sec are selected in addition to the live image after 3 sec since the last captured image 21 was captured. The corresponding point of the live image after 6 sec is searched for the feature point of the live image after 3 sec, and the average movement vector is obtained. Further, the corresponding point of the live image after 9 sec is searched for the feature point of the live image after 6 sec, and the average movement vector is obtained. In this way, the feature points of the live image can be tracked. The tracking time interval can be arbitrarily set according to the situation. For example, 1 sec is set when it is desired to track feature points finely, and 5 sec is set when the feature points may be tracked coarsely. Next, the camera movement direction determination unit 8 moves the photographing unit 3 from the screen position of the feature point extracted by the feature point extraction unit 7 in the previous captured image 21 and the live image 25 or from the change in the screen position between live images. A direction is obtained (moving direction determination step: S170). For example, a direction opposite to the direction of the average movement vector obtained from the previous photographed image 21 and live image 25 can be obtained as the movement direction of the photographing unit 3. Further, if the feature points between the live images are tracked and the direction of the average movement vector continues in the same direction (for example, the left direction) for example three times or more, the direction opposite to the direction of the average movement vector is set as the movement direction of the photographing unit 3. You may ask.

  Next, the overlapping area setting unit 6 performs the main setting of the overlapping direction (overlapping direction setting step: S180). The overlapping area setting unit 6 sets an overlapping area 23 with the live image 25 in the previous captured image 21 on the same side as the movement direction determined by the camera movement direction determination unit 8. For example, when it is determined that the moving direction of the photographing unit 3 is the right direction, the overlapping region 23 is set on the right side of the previous photographed image 21. Next, the display unit 4 displays the previously captured image 21 and the live image 25 in the display area 26 in an overlapping manner (display process: S190). For example, when the overlap area is set to “right (side)” and “60 (%)”, 60% of the previous captured image becomes the overlap area 23, and the previous captured image 21 in the display area 26 is the overlap area 23. The live image 25 is displayed on the right side which is the same side as the moving direction of the photographing unit 3 from the overlapping area 23.

  Next, the photographer monitors whether or not the previously captured image 21 and the live image 25 overlap in the display area 26, and determines whether or not the shooting position and the camera angle are appropriate (camera position determination step: S210). If an image is taken when the same portion of the measurement object 2 in the previous captured image 21 and the live image 25 substantially overlaps in the overlapping area 23, the scheduled current captured image 22 can be obtained. If there is a difference between the previously captured image 21 and the live image 25 in the overlapping area 23 (NO in S210), the shooting position and the camera angle are adjusted, and the live image acquisition process (S140) to the image display process (S190) are performed. repeat. When the same portion of the measurement object 2 in the previous captured image 21 and the live image 25 substantially overlaps (YES in S210), it is determined whether or not the number of captured images has reached a predetermined number (photographing end determination step: S220). For example, the scheduled number of shots is registered in the control unit 10 in advance, and the number of shots is counted for each shot, and the control unit 10 compares the counted number of shots with the registered scheduled number of shots. When the number of shots <the number of shots to be shot (NO in S220), the shooting step (S110) to the camera position determination step (S210) are repeated. When the counted number of shots ≧ the scheduled number of shots (YES in S220), the image shooting process flow ends. If the scheduled number of shots is not registered, it is determined in the shooting end determination step (S220) whether the photographer ends.

  As described above, according to the present embodiment, in the case of shooting while moving using a single camera, a stereo image shooting apparatus and method for appropriately determining an overlapping area between a previous shot image and a current shot image Can provide.

  In the first embodiment, an example in which the previously captured image and the live image are displayed in the display area of the display unit in an overlapping manner has been described. However, in the second embodiment, the previously captured image, the previously captured image and the live image are displayed in the display area of the display unit. An example in which images are displayed in duplicate will be described. The configuration of the stereo image photographing device 1 is the same as that of the first embodiment. Differences from the first embodiment will be mainly described.

  FIG. 8 shows an example in which a live image is displayed in an overlapped manner on the last-time photographed image and the previous photographed image. 8A shows the display unit 4 of the digital camera 3, and FIG. 8B shows the camera display area 26 on a plane. A car as the measurement object 2 is displayed in the display area 26 of the display unit 4 of the digital camera 3 (a region surrounded by an outer solid line frame in FIG. 8A). The inner solid line frame is the image 27 taken two times before and is arranged on the left side of the display area 26, here, near the left end, and the one-dot chain line frame is the previous photographed image 21, arranged in the center of the display area 26, and broken line The inside of the frame is a live image 25, which is arranged on the right side of the display area 26, here near the right end. That is, this is a case where the moving direction of the photographing unit 3 is the right direction. The inner solid line frame and the portion within the alternate long and short dash line frame form an overlapping area (overlap area) between the previous captured image 27 and the previous captured image 21, and the previous image repeated area 27 a set as the previous captured image 27. is there. The portion within the one-dot chain line frame and within the broken line frame is an overlapping region (overlap region) between the previous captured image 21 and the live image 25, and is the previous image overlapping region 21 a set to the previous captured image 21. The overlapping ratio of these overlapping areas is about 60%. The portion extending from the previous image overlap area 27a to the left side (opposite to the moving direction of the image capturing unit 3) of the previous image shot image 27 is the previous image extension area 27b and the left side of the previous image image 21 from the previous image overlap area 21a (shooting). The part extending in the direction opposite to the moving direction of the part 3) is the previous image extension area 21b, the part overlapping the previous image overlap area 21a of the live image 25 is the live image overlap area 25a, and the part extending rightward from the live image overlap area 25a. Is the live image extension area 25b.

  Further, a portion within the inner solid line frame and the broken line frame is a triple overlap region (overlap region) of the previously captured image 27, the previous captured image 21 and the live image 25, and is a tie region 29. The overlapping ratio of the tie area 29 is about 20%. In this way, by overlapping the live image 25 and the previous captured image 27, it is possible to overlap the entire imaging region of the measurement object 2, and to obtain the three-dimensional coordinates of the entire imaging region of the measurement object 2. If shooting is performed when the image of the live image overlapping area 25a in the live image 25 substantially matches the image of the previous image overlapping area 21a in the previous captured image 21, the scheduled current captured image 22 can be obtained. Alternatively, if the photographing is performed when the same portion of the measurement object 2 in the tie region 29 and the measurement object 2 in the live image 25 substantially overlap each other, the planned current photographing image 22 can be obtained. The term “substantially match” means that it is only necessary to match within the margin even if they do not match completely. For example, if the image overlap area 27a is set with a margin for overlapping with the live image 25, the image overlap area 27a is not problematic as long as it matches within the margin. For example, since it is sufficient for the three-dimensional measurement to have an overlap area between the previously captured image 27 and the live image 25, if the overlap ratio is set to 20%, there is no problem with a deviation of ± 5%, and a little less than ± 20% Is acceptable. Even if a deviation occurs within the margin, it can be sufficiently recovered by intentionally shifting the image in the reverse direction in the next image capturing. Of course, it is preferable to match as much as possible in order to maintain the uniformity of the image processing, for example, within ± 3%. Note that it is easy to distinguish each overlapping area by changing the color and transparency of each overlapping area. In particular, it is preferable that the tie region 29 is easily distinguished by hatching or the like. Further, when the transparency of each image is reduced, the tie area 29 becomes darker than the other portions, and it becomes easy to distinguish.

  FIG. 9 shows an example of a processing flow of image capturing according to the second embodiment. In the present embodiment, an example will be described in which the previously captured image 27, the previously captured image 21 and the live image 25 are displayed in the display area 26 in an overlapping manner. In the image display process (190) of the first embodiment (see FIG. 7), the previous captured image 21 and the live image 25 are displayed. In contrast, in the second embodiment, the previously captured image 27 is displayed in the image display process (192). The difference is that the previously captured image 21 and the live image 25 are displayed. Therefore, the live image 25 can be viewed overlapping the previous captured image 27 and the previous captured image 21. In particular, it is possible to confirm the overlap in the tie area 29 and obtain a photographed image. Other processing flows are the same as those in the first embodiment, and according to this embodiment, when shooting is performed using a single camera as in the first embodiment, the previous captured image and the current captured image overlap. It is possible to provide a stereo image capturing apparatus and a method thereof in which a region can be appropriately determined.

  In the third embodiment, a three-dimensional position measurement function is added to the first embodiment, and a displacement correction image can be created, the three-dimensional position coordinates of the feature point of the measurement object, and the shooting position of the camera can be measured. An example of correcting the magnification of an image displayed using an image will be described. Differences from the first embodiment will be mainly described.

  FIG. 10 is a block diagram illustrating a configuration example of a stereo image photographing device 1A according to the third embodiment of the present invention. In the stereo image photographing device 1A, a three-dimensional position measurement unit 11 is added to the configuration of the first embodiment (see FIG. 1). The three-dimensional position measurement unit 11 uses the feature points extracted by the feature point extraction unit 7 to determine the camera position and inclination of the photographing unit 3, and the camera position and inclination obtained by the orientation unit 12. Using the position and inclination of the camera obtained by the displacement correction image forming unit 13 and the orientation unit 12 for performing the displacement correction processing on the previously captured image and the live image to form the displacement correction image. It has a three-dimensional coordinate calculation unit 14 that calculates the three-dimensional position coordinates of the feature point of the object 2 and the shooting position.

[Calculation of external orientation elements: mutual orientation]
Next, a method for obtaining the position and tilt of the camera by the relative orientation method from the points where the left and right images are associated will be described. The relative orientation is performed by the orientation unit 12.
A model image refers to a three-dimensional image obtained when a subject is reproduced from two or more photographed images. Forming relatively similar model images is called relative orientation. That is, the relative orientation determines the position and inclination of the projection center of each of the left and right cameras so that the two corresponding light beams of the captured image meet.

FIG. 11 is a diagram for explaining relative orientation. Next, the details of the orientation calculation of each model image will be described. By this calculation, the positions of the left and right cameras (three-dimensional coordinates and three-axis tilt) are obtained.
The parameters relating to the positions of these cameras are obtained by the following coplanar conditional expression.

The origin of the model coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. For the scale, the base line length is taken as the unit length. The parameters to be obtained at this time are the rotation angle κ ₁ of the left camera, the rotation angle φ _{1 of} the Y axis, the rotation angle κ _{2 of the} right camera, the rotation angle φ ₂ of the Y axis, and the rotation of the X axis. the five of the rotation angle of the corner ω _2. In this case, since the rotation angle ω ₁ of the X axis of the left camera is 0, it is not necessary to consider.

Under such a condition, the coplanar conditional expression of (Expression 1) becomes as shown in (Expression 2), and each parameter can be obtained by solving this expression.

Here, between the model coordinate system XYZ and the camera coordinate system xyz, the following coordinate transformation relational expressions (formula 3) and (formula 4) hold.

Using these equations, unknown parameters are obtained by the following procedure.
(A) The initial approximate value of the unknown parameter is normally 0.
(B) The coplanar conditional expression (Expression 2) is Taylor-expanded around the approximate value, and the value of the differential coefficient when linearized is obtained by (Expression 3) and (Expression 4), and an observation equation is established.
(C) A least square method is applied to obtain a correction amount for the approximate value.
(D) Correct the approximate value.
(E) Using the corrected approximate value, the operations (b) to (d) are repeated until convergence.
By obtaining unknown parameters (κ ₁ , φ ₁ , κ ₂ , φ ₂ , ω ₂ ), the position and tilt of the camera can be obtained.
If the position of the camera is obtained by the relative orientation method, it is possible to obtain the three-dimensional coordinates on the object space point, that is, the three-dimensional coordinates of the feature points by the stereo method.

[DLT method]
Further, instead of the orientation, a target can be arranged around the measurement object 2 and the coordinates of the photographing position can be obtained using the DLT method. For example, the calculation is performed by the three-dimensional coordinate calculation unit 14.
The DLT method approximates the relationship between the photographic coordinates and the three-dimensional coordinates (target point coordinates) of the subject using a cubic projective transformation equation.

The basic formula of the DLT method is (Formula 5).
If the denominator is deleted from (Equation 5), the following line form can be derived.
Further, when (Expression 6) is modified, the following expression is obtained.

If (Expression 7) is solved directly using the least square method, 11 unknown variables L _{1 to} L ₁₁ that determine the relationship between the photograph coordinates and the object point coordinates can be acquired. Therefore, by solving (Equation 7) for six points whose position coordinates are known, the relationship between the photographic coordinates (x, y) of the subject and the three-dimensional coordinates (target point coordinates) (X, Y, Z) of the subject is obtained. Thus, the position coordinates (0, 0, 0) of the photographing position corresponding to the convergence point of the straight line connecting the photograph coordinates (x, y) and the target point coordinates (X, Y, Z) can be obtained. In this case, the shooting position is a position (−X, −Y, −Z) with respect to the target point coordinates.

FIG. 12 is a diagram for explaining a deviation corrected image. FIG. 12A is a diagram for explaining the displacement correction image three-dimensionally, and FIG. 12B is a diagram for explaining the plan. The displacement correction process is performed by the displacement correction image forming unit 13. The deviation correction process is performed in parallel to the subject so that the stereo image geometry is established for the measurement object 2 using the shooting position and tilt of the camera obtained by the orientation. This is a process of correcting the image so that the epipolar line matches the left and right horizontal lines. Therefore, distortions such as magnification and inclination of the left and right images are corrected, and the positions of the magnification and height are the same. In FIG. 12A, the shooting positions of the camera 3 for shooting the left image and the right image of the stereo image are O ₁ and O ₂ , and the position of the measurement point of the measurement object 2 is P, respectively. The origin of the three-dimensional position coordinates and O _1, the three-dimensional coordinate axes X, Y, and Z axis. Since the shot images taken at the shooting positions O ₁ and O ₂ vary in the shooting distance and the camera angle, the magnification and inclination of the image are distorted as shown by the thin solid squares. These captured images are corrected to an image with the same magnification as viewed from the front of the camera, as shown by a thick solid square, by a deviation correction process. In FIG. 12B, as shown in the upper part, the magnification and the inclination are distorted in the previous captured image that is the left image and the live image that is the right image. As shown in the lower part of this, the epipolar line is corrected by the displacement correction process so that the epipolar line coincides with the left and right horizontal lines for the previously captured image 21 that is the left image and the live image 25 that is the right image. The tilt is aligned.

In the displacement correction process, an internal orientation element (parameter) obtained by mutual orientation is used, and a photographed image tilted from the relational expression of the projective transformation is displayed on the X and Y planes of the model coordinate (deviation corrected image coordinate) system (vertical). Project to the photo. Next, a relational expression related to deviation correction is shown. For the left image, the model coordinates are (X1, Y1), the position coordinates in the captured image plane are (X _C1 , Y _C1 ), for the right image, the model coordinates are (X2, Y2), and the position coordinates in the captured image plane are (X _C2 , Y _C2 ), C1 and C2 are the focal lengths of the cameras in the captured images at the capturing positions O ₁ and O ₂ , m11 to m33 are conversion parameters related to the left image, and n11 to n33 are conversion parameters related to the right image. Then, the conversion expressions are expressed by (Expression 8) and (Expression 9). Thereby, the displacement correction process is performed.

  FIG. 13 is a diagram for explaining the magnification of the captured image and the display image. FIG. 13A is a diagram showing that the magnifications of the last-time photographed image 27, the previous-time photographed image 21, and the live image 25 are different, and FIG. It is a figure which shows that the image 25 is displayed on the range of the same magnitude | size. Since the measurement object 2 has a difference in magnification and inclination according to the change of the photographing position, when the three images are represented by images of this magnification, a difference occurs in the dimensions of the images as shown in FIG. (The figure shows an example in which only the magnification is different). However, in the display area 26, as shown in FIG. 13B, the last-time photographed image 27, the last-time photographed image 21, and the live image 25 are displayed in the same size range. In addition, the overlapping area 27o between the last-time captured image 27 and the previous captured image 21 and the overlapping area 21o between the previous captured image 21 and the live image 25 in FIG. 13A are shown in FIG. Thus, the image overlap area 27a and the previous image overlap area 21a are displayed two times before. Accordingly, the magnification of each image is different in each of the overlapping areas 21a and 27a, and the magnification of the three images is displayed differently in the tie area 29. Further, when the angle at which the measurement object 2 is viewed is different, the image is inclined in each of the overlapping regions 21 a and 27 a and the tie region 29. Therefore, if a deviation correction process is performed on the last-time photographed image 27, the previous photographed image 21, and the live image 25 to correct the deviation-corrected image, the images have the same magnification and are viewed from the front. Thereby, in the overlapping areas 21a and 27a and the tie area 29, the same part of the measuring object 2 in the two or three images can be displayed so as to be almost coincident with each other. Note that the reason why they do not completely match is that the positional relationship between the feature points changes somewhat depending on the viewing direction. As a result, the previous captured image 21 and the live image 25 can be matched well in the overlapping region 21a or the tie region 29, and the current captured image 22 can be captured more accurately.

[Stereo method]
FIG. 14 is a diagram for explaining the stereo method. The three-dimensional position coordinates of each feature point of the measurement object 2 can be measured using the principle of the stereo method. The calculation of the three-dimensional position coordinates is performed by the three-dimensional coordinate calculation unit 14. Therefore, the three-dimensional shape can be obtained by overlappingly shooting the entire shooting region of the measurement object 2. For simplicity, the same camera is used to shoot from _two shooting positions (O ₁ , O ₂ ), the optical axes are parallel, the distance c from the camera lens principal point to the CCD surface is equal, and the CCD is on the optical axis. Assume that it is placed at a right angle. Let B be the distance between the two optical axes (baseline length).
Point on the object P (x, y, z) and imaging screen (screen of the previous captured image in this embodiment) of the photographing position _{O 1} point on _{P 1 (x 1, y 1} ), the photographing position _{O 2} There is the following relationship between the coordinates of the point P ₂ (x ₂ , y ₂ ) on the shooting screen (live image screen in this embodiment).
x ₁ = cx / z --- (Equation 10)
y ₁ = y ₂ = cy / z --- (Formula 11)
x ₂ −x ₁ = cB / z −−− (Formula 12)
However, it is assumed to take the entire coordinate system (x, y, z) the origin of the lens principal point of the photographing position O _1.
Z is obtained from (Equation 12), and x and y are obtained from (Equation 10) and (Equation 11) using this.

  Further, when the three-dimensional coordinates of each feature point are obtained, the previous captured image or the last captured image is converted into an image viewed from an arbitrary direction, for example, the current position of the live image 25 or the planned shooting position of the current captured image 22. It can be expressed. In addition, a three-dimensional model image, an ortho image, and an all-around image can be formed. If feature points are arranged and constructed in the three-dimensional image space from the obtained three-dimensional coordinates, a three-dimensional model image is projected. If projected onto a plan view in parallel projection, an ortho image is projected. If it is arranged continuously, it becomes an all-around image. In addition, the three-dimensional model image can be expressed from an arbitrary direction using a perspective view or a projection view.

  FIG. 15 illustrates an example of a processing flow of image capturing according to the third embodiment. It is an example which correct | amends the magnification of the image displayed using a deviation correction image. Compared to the processing flow example of Embodiment 1 (see FIG. 7), the orientation step (S182) and the displacement correction image forming step (S184) are added after the overlapping direction setting step (S180). That is, the orientation unit 12 obtains the position and inclination of the camera of the photographing unit 3 using the feature points extracted by the feature point extraction unit 6, and the displacement-required image forming unit 13 obtains the camera obtained by the orientation unit 12. Using the position and inclination of the above, a displacement correction process is performed on the previously captured image 21 and the live image 25 to form a displacement correction image. In the display unit 4, the previous captured image 21 and the live image 25 are displayed in an overlapping manner using the displacement correction image, so that the same portion of the measurement object 2 in the previous captured image 21 and the live image 25 almost matches. It can be displayed overlaid. The almost coincidence is because the position of the symmetry point on the screen slightly changes depending on the viewing direction. As a result, the current captured image 22 can be captured more accurately. Even in the case where the previously captured image 27, the previously captured image 21 and the live image 25 are displayed in an overlapping manner, the same portion of the measurement object 2 in the three images can be displayed in the tie region 29 so as to almost coincide with each other. The captured image can be captured more accurately. Other processing flows are the same as those in the first or second embodiment.

  As described above, according to the present embodiment, in the case of shooting while moving using a single camera, a stereo image shooting apparatus and method for appropriately determining an overlapping area between a previous shot image and a current shot image Can provide. In addition, by creating a displacement corrected image, the previously captured image 21 and the live image 25 can be displayed in an overlapping manner in a stereoscopically viewable manner, and an image that is almost the same in the overlapping region 23 can be obtained at the planned shooting position. It is.

  In the fourth embodiment, in addition to the third embodiment, an example will be described in which the first shot image that is the first shot image is displayed in the display area when the shooting is completed (when the circumference of the measurement object is made a round). To do. Differences from the third embodiment will be mainly described. The apparatus configuration is the same as that of the third embodiment (see FIG. 10).

  FIG. 16 shows an example in which a live image is displayed in an overlapped manner with the first shot image. FIG. 16A shows the previous display area that makes a round, and FIG. 16B shows the display area that makes a round. The corresponding point search unit 8a performs stereo matching (corresponding to feature points) between the first shot image 28 and the live image 25. When the first captured image 28 and the live image 25 are correlated by stereo matching and an overlapping area is found, the first captured image 21 is displayed in the display area 26 where the previously captured image 21 and the live image 25 are displayed. 28 can be displayed. When an overlapping area is found, as shown in FIG. 16A, together with the previous captured image 21 and the live image, of the first captured image 28, the one-round image overlapping area 28c that is the overlapping area with the live image 25 is displayed. The part is displayed, and the other part is outside the display area 26. In the previous captured image 21, the previous image overlap area 21a and the previous image extension area 21b are displayed separately by the frame of the live image 25. In the live image 25, the live image overlap area 25a and the live image extension area 25b are the frame of the previous captured image 21. Are displayed separately. As shown in FIG. 16 (b), a display area 26 is shown when one round is taken during the next shooting, and a round image overlapping area 28c of the first shot image 28 partially overlaps with the previous shot image 21, and at least the first image overlap. A part of the area 28 a can be displayed in the display area 26. In this way, the rounded image overlap region 28c partially overlaps with the previous captured image 21, and the triple overlap region of the previous captured image 21, the initial captured image 28, and the live image 25 is formed, and thus the shooting has been completed. I understand. In stereo matching, the captured image and the live image can be used as they are, but if the deviation correction image of the captured image and the live image is used, corresponding points can be detected with high accuracy, and stereo matching can be performed more accurately. Can be performed.

  FIG. 17 shows an example of a processing flow of image capturing according to the present embodiment. In this example, the first shot image 28 is displayed in the display area 26 when shooting is completed. Compared to the processing flow example (see FIG. 15) of the third embodiment, a stereo matching process (S186) is added after the displacement correction image forming process (S184), and an image display process is performed instead of the image display process (S190). (S194) is performed. In the image display step (S194), when an overlapping area is found in the first captured image 28 and the live image 25 as compared with the image display step (S190), in addition to the previous captured image 21 and the live image 25, At least a part of the first shot image 28 is displayed in the display area 26. In the feature point extraction step (S150), the feature point extraction unit 6 extracts feature points of the first shot image 28 after the first shooting. In the stereo matching step (S186), stereo matching is performed on the first shot image 28 and the live image 25. Thereby, it is possible to obtain an overlapping area (one-round image overlapping area 28 c) between the first shot image 28 and the live image 25. In the image display step (S194), when an overlapping area is found in the first shot image 28 and the live image 25 in the stereo matching step (S186), at least the first shot image is added to the previous shot image 21 and the live image 25. A part of 28 is displayed in the display area 26. Thereby, at the time of photographing of the current photographed image 22, it is possible to photograph while confirming the overlap with the first photographed image 28. Then, a state in which the previous captured image 21 partially overlaps the overlapping region (initial image overlapping region 28a) of the first captured image 28 (a state in which a triple overlapping region of the previously captured image 21, the first captured image 28, and the live image 25 exists). Thus, if the photographed image 22 is photographed this time, the measurement object 2 can be photographed over the entire circumference, and three-dimensional coordinates can be obtained. Other processing flows are the same as the processing flow example of the third embodiment.

  Further, instead of finding an overlapping area by stereo matching, it is also possible to find the overlapping area by calculating the photographing position. The three-dimensional coordinate calculation unit 14 uses, for example, a DLT method to calculate the tertiary of each shooting position from the screen position of the feature point of the measurement object 2 in the live image 25 and the first shot image 28 extracted by the feature point extraction unit 7. Find original coordinates. When an overlap area is found between the live image 25 and the first shot image 28 from the shooting positions in the live image 25 and the first shot image 28, the overlap area setting unit 6 goes around the overlap area in the first shot image 28. It is set as the image overlap area 28c. For example, if the distance between the shooting position in the live image 25 and the shooting position in the first shooting image 28 is equal to or less than the distance between the shooting position in the live image 25 and the shooting position in the previous shooting image 27, the live image and the first shooting image 28 Corresponds to a case where an overlapping region is found. The overlap ratio is calculated from the distance between the shooting position in the live image 25 and the shooting position in the first shot image 28. The display unit 4 displays the first shot image 28 and the live image 25 so as to overlap each other in the round image overlap area 28c. In the image shooting process flow, the shooting position is calculated instead of the stereo matching step (S186), but the other steps are the same.

  According to the present embodiment, as in the third embodiment, in the case of shooting while moving using a single camera, a stereo image shooting apparatus capable of appropriately determining an overlapping area between the previous shot image and the current shot image and its Can provide a method. Moreover, it can be seen that the shooting is completed by displaying the first shot image.

  In the fifth embodiment, in addition to the third embodiment, an example will be described in which the positions and frames of the previously captured image and the live image are changed and displayed according to the shooting direction. Differences from the third embodiment will be mainly described. The apparatus configuration is the same as that of the third embodiment (see FIG. 10).

  FIG. 18 shows an example in which the previously captured image, the live image, and the positions of these frames are changed and displayed according to the shooting direction. By performing stereo matching between the previously captured image 21 and the live image 25, the image of the overlapping area between the previously captured image 21 and the live image 25 can be expressed with higher accuracy. In this case, the overlapping area setting unit 6 sets the overlapping area 23 between the previous captured image 21 and the live image 25 as the previous overlapping area 21c on the previous captured image 21 side and the live overlapping area 25c on the live image 25 side. In the display unit 4, the same part of the measurement object 2 is displayed so as to substantially overlap in the previous overlap area 21 c and the live overlap area 25 c. When displayed in this way, for example, when the live image 25 is shifted to the lower right side as compared with the position where the live image 25 is to be displayed, the live image 25 is displayed shifted to the lower right side as shown in FIG. In this case, if the magnification of the display image is adjusted, both the previously captured image 21 and the live image 25 can be displayed so as to be included in the display area 26 together with a frame. As a result, it can be seen that the live side overlap area 25c is shifted to the lower right side with respect to the overlap area 23, the position of the live image is moved in the upper left direction, and the live side overlap area 25c is photographed according to the overlap area 23. Just do it. Other processing flows are the same as the processing flow example of the third embodiment. Further, when three images of the last-time photographed image 27, the previous-time photographed image 21, and the live image 25 are displayed, the overlapping area is displayed on the display unit 4 by performing stereo matching between the last-time photographed image 27 and the last-time photographed image 21. It is also possible to display so as to substantially overlap.

  According to the present embodiment, as in the third embodiment, in the case of shooting while moving using a single camera, a stereo image shooting apparatus capable of appropriately determining an overlapping area between the previous shot image and the current shot image and its Can provide a method. In addition, the live image and the positions of these frames are displayed in the previously captured image in accordance with the shooting direction, so that the direction of deviation of the live image can be visually grasped.

  In the sixth embodiment, an example in which the overlapping area setting unit sets the overlapping area on the same side as the movement direction detected by the movement direction detection unit will be described. Differences from the first embodiment will be mainly described.

  FIG. 19 is a block diagram illustrating a configuration example of a stereo image capturing device 1B according to the sixth embodiment. Compared to the configuration of the first embodiment (see FIG. 1), the feature point extraction unit 7 and the camera movement direction determination unit 8 are not provided, and a movement direction detection unit 15 is added instead. The movement direction detection unit 15 includes a GPS (Global Positioning System) or an inertial sensor, and detects the movement direction of the imaging unit 3. GPS is a system that measures the position of a person using radio waves from the US Department of Defense artificial satellite, and can measure latitude and longitude with accuracy of several tens of meters from the difference in arrival time from three or more satellites. It is used for automobile navigation. The relative position change (moving direction) can be measured with higher accuracy. Also, an acceleration sensor such as a piezoelectric type or a capacitance type can be used as the inertial sensor. The position change can be measured by integrating the detected acceleration twice. The overlapping area setting unit 6 sets an overlapping area on the same side as the moving direction detected by the moving direction detecting unit 15. Other configurations are the same as those of the first embodiment.

  FIG. 20 shows an example of a processing flow of image capturing according to the sixth embodiment. Compared to the processing flow of Embodiment 1 (see FIG. 9), there is no feature point extraction step (S150) to camera movement direction determination step (S170), and a movement direction detection step (S152) is added instead. Yes. In the moving direction detection step (S152), the moving direction of the photographing unit 3 is detected using a GPS sensor or an inertial sensor. In the overlapping direction setting step (S180), an overlapping region is set on the same side as the moving direction detected in the moving direction detection step (S152). Other processes are the same as those in the first embodiment.

  In addition, the first shot image 28 can be displayed in the display area 26 when shooting is completed (when the object is circled around). When an overlap area is found between the live image 25 and the first shot image 28 from the shooting position in the live image 25 and the first shot image 28 detected by the GPS or the inertial sensor, the overlap area setting unit 6 An overlapping area is set as a round image overlapping area 28c in the captured image 28. For example, if the distance between the shooting position in the live image 25 and the shooting position in the first shooting image 28 is equal to or less than the distance between the shooting position in the live image 25 and the shooting position in the previous shooting image 27, the live image and the first shooting image 28 Corresponds to a case where an overlapping region is found. The overlap ratio is calculated from the distance between the shooting position in the live image 25 and the shooting position in the first shot image 28. The display unit 4 displays the first shot image 28 and the live image 25 so as to overlap each other in the round image overlap area 28c.

  According to the present embodiment, as in the first embodiment, when shooting while moving using a single camera, a stereo image shooting apparatus capable of appropriately determining an overlapping area between the previous shot image and the current shot image and its Can provide a method.

  In the seventh embodiment, an example in which the overlapping area setting unit sets the overlapping area on the same side as the input moving direction will be described. Differences from the first embodiment will be mainly described.

  FIG. 21 is a block diagram illustrating a configuration example of a stereo image capturing device 1 </ b> C according to the seventh embodiment. Compared to the configuration of the first embodiment (see FIG. 1), the feature point extraction unit 7 and the camera movement direction determination unit 8 are eliminated. The overlapping area setting unit 6 sets an overlapping area by an input from the photographer. In the overlapping area setting unit 6, the overlapping area 23 is set in the previous captured image 21. The overlap area 23 is set for the overlap ratio and the overlap direction. The moving direction of the imaging unit 3 can be input in advance, and the moving direction input by the photographer is set as the overlapping direction. For example, when “right (side)” (moving direction) and “60 (%)” (overlapping ratio) are input from the input unit 9 using a keyboard, the previously captured image 21 is displayed with an overlapping area 23 surrounded by a frame. The For example, when the photographer clicks the confirmation button on the display screen with the mouse, the overlapping ratio and the overlapping direction are determined, and the overlapping area 23 is set in the previous captured image 21. Other configurations are the same as those of the first embodiment.

  FIG. 22 shows an example of a processing flow of image capturing according to the seventh embodiment. Compared to the processing flow of the first embodiment (see FIG. 7), there is no feature point extraction step (S150) to camera movement direction determination step (S170), and an overlap ratio setting step (S130) and an overlap direction setting step ( In place of (S180), there is an overlapping area setting step (S132). In other words, in the overlapping area setting step (S132), the overlapping direction is set together with the overlapping ratio by the photographer's input. Other processes are the same as those in the first embodiment.

  In addition, the first shot image 28 can be displayed in the display area 26 when shooting is completed (when the object is circled around). When the photographer makes a round, it is possible to input that it has made a round, and when the photographer has entered that it has made a round, the overlap area setting unit 6 sets the round image overlap area 28 c in the first shot image 28. For example, the round image overlapping area 28c is set on the side opposite to the moving direction of the photographing unit 3 input by the photographer, and the overlapping ratio is set to be the same as the initial image overlapping area 28a. Here, “turning once” means moving to the same direction and reaching a position where the shooting from all directions planned by the current shooting is completed. For example, when shooting from 12 directions is scheduled, this means that the 12th shooting position has been reached. The display unit 4 displays the first shot image 28 and the live image 25 so as to overlap each other in the round image overlap area 28c.

  According to the present embodiment, as in the first embodiment, when shooting while moving using a single camera, a stereo image shooting apparatus capable of appropriately determining an overlapping area between the previous shot image and the current shot image and its Can provide a method.

  The present invention can also be realized as a stereo image capturing method invention described in the flowcharts of the above-described embodiments, and a program for causing a stereo image capturing apparatus to execute the method invention. The program may be stored and used in a built-in storage unit of the stereo image photographing device, may be stored and used in an external storage device, or may be downloaded from the Internet and used. Moreover, it is realizable also as a recording medium which recorded the said program.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it is obvious that various modifications can be made to the embodiments without departing from the spirit of the present invention. .

  For example, the overlapping display of the last-time photographed image, the previous photographed image, and the live image described in the second embodiment can be applied to the third to seventh embodiments. In addition, the deviation correction image described in the third embodiment, the first captured image described in the fourth embodiment, the previous captured image and the live image described in the fifth embodiment, and the display for changing the positions of these frames are displayed. This can also be applied to the sixth embodiment or the seventh embodiment. The image processing flow can be changed as appropriate. For example, in the first to fifth embodiments, the example in which the overlap ratio is set before the live image acquisition has been described. However, it may be set in advance before shooting. Further, the feature point tracking step in the first to fifth embodiments can be omitted. Alternatively, the position may be moved while photographing a stereo image at a position slightly shifted left and right at each photographing position, and photographing may be performed while overlapping with other stereo images. Further, when the moving direction is both the vertical direction and the horizontal direction, the overlapping area may be set and displayed as the photographing side overlapping area in FIG. In the above embodiment, the live image is temporarily stored. However, the live image may be stored so as to be reproducible like a video image. In addition, the number of shots, the overlapping ratio, and the like can be changed as appropriate.

  The present invention can be used for photographing for three-dimensional measurement of an object to be measured and photographing all-around images.

1, 1A to 1C Stereo image capturing device 2 Measurement object 3 Imaging unit 4 Display unit 5 Storage unit 6 Overlapping region setting unit 7 Feature point extraction unit 8 Camera movement direction determination unit 8a Corresponding point search unit 9 Input unit 10 Control unit 11 Three-dimensional position measurement unit 12 Orientation unit 13 Deviation correction image forming unit 14 Three-dimensional coordinate calculation unit 15 Movement direction detection unit 16 Personal computer (PC)
21 Previous shot image 21a Previous image overlap area 21b Previous image extension area 21c Previous side overlap area 21o Overlap area between previous shot image and live image 22 Current shot image 23 Overlap area 24 Extension area 25 Live image 25a Live image overlap area 25b Live Image extension area 25c Live-side overlap area 26 Display area 27 Previous-time captured image 27a Previous-time image overlap area 27b Previous-time image extension area 27o Overlapping area 28 between the previous-time captured image and the previous captured image 28 First-time captured image 28a Initial image-overlapped area 28c Region 29 Tie region 51 Captured image storage unit 52 Live image storage unit 53 Processed image storage unit 54 Position coordinate storage unit

Claims (18)

In a stereo image capturing device that sequentially captures a measurement object while overlapping with a single camera;
A photographing unit that obtains a live image of the measurement object at any time as the camera moves, and photographs the measurement object in a timely manner to obtain a photographed image that is a still image;
An overlapping area setting unit that sets an overlapping area with the live image in the previous captured image that is a captured image previously captured by the capturing unit;
The previous captured image and the live image are overlapped and displayed in the overlapping area set by the overlapping area setting unit, and the previous captured image is extended and displayed from the overlapping area to the opposite side of the moving direction of the imaging unit. And a display unit for displaying the live image extending from the overlapping region to the same side as the moving direction of the photographing unit;
The overlapping area setting unit sets an overlapping area between the previous captured image and the live image as a previous image overlapping area, and the previous captured image and the previous captured image, which are captured images previously captured by the capturing unit. An overlapping area is set as the previous image overlapping area, and further, the live image and the previous captured image are overlapped, so that any imaging area of the measurement object is overlapped and shot;
The display unit displays the previous image overlap region and the previous image overlap region, and also displays a tie region that is a triple overlap region of the previous image, the previous image, and the live image;
Stereo image capturing device. The overlapping area setting unit sets so that any imaging area of the measurement object in the previous captured image is included in the previous image overlapping area or the previous image overlapping area;
The stereo image photographing device according to claim 1. The overlapping area setting unit sets the overlapping ratio of the overlapping area between the previous captured image and the live image to 55 to 90%, and sets the overlapping ratio of the overlapping area between the previous captured image and the previous captured image to 55 to 90%. Set to%;
The stereo image photographing device according to claim 1 or 2. In a stereo image capturing device that sequentially captures a measurement object while overlapping with a single camera;
A photographing unit that obtains a live image of the measurement object at any time as the camera moves, and photographs the measurement object in a timely manner to obtain a photographed image that is a still image;
An overlapping area setting unit that sets an overlapping area with the live image in the previous captured image that is a captured image previously captured by the capturing unit;
The previous captured image and the live image are overlapped and displayed in the overlapping area set by the overlapping area setting unit, and the previous captured image is extended and displayed from the overlapping area to the opposite side of the moving direction of the imaging unit. And a display unit for displaying the live image extending from the overlapping region to the same side as the moving direction of the photographing unit;
The overlapping area setting unit sets an overlapping ratio of an overlapping area between the previous captured image and the live image to 55 to 90%, and the previously captured image and the previous captured image that are captured images previously captured by the capturing unit. Set the overlap ratio of the overlapping area with the image to 55-90%;
The display unit displays an overlapping region between the previous captured image and the live image, an overlapping region between the previous captured image and the previous captured image, and the previous captured image, the previous captured image, and the live image. Display a tie region that is a triple overlap region of
Stereo image capturing device. The overlapping region setting unit sets the overlapping direction to the left or right side for the previous captured image, or sets the overlapping direction to the upper or lower side for the previous captured image;
The stereo image photographing device according to claim 3 or 4. The display unit displays the captured image and the live image with a frame;
The stereo image photographing device according to claim 1. A feature point extraction unit that extracts feature points of the measurement object in the previous photographed image and the live image;
A camera movement direction determination unit that determines a movement direction of the photographing unit from the screen position of the feature point extracted by the feature point extraction unit in the previous captured image and the live image;
The overlapping area setting unit sets the overlapping area in the previously captured image on the same side as the movement direction determined by the camera movement direction determination unit;
The stereo image photographing device according to any one of claims 1 to 6. A GPS or inertial sensor, and a movement direction detection unit that detects a movement direction of the imaging unit;
The overlapping area setting unit sets the overlapping area in the previous captured image on the same side as the moving direction detected by the moving direction detecting unit;
The stereo image photographing device according to any one of claims 1 to 6. The moving direction of the photographing unit can be input in advance, and the overlapping region setting unit sets the overlapping region in the previous captured image on the same side as the input moving direction;
The stereo image photographing device according to any one of claims 1 to 6. A feature point extraction unit that extracts feature points of the measurement object in the previous photographed image and the live image;
An orientation unit that obtains the position and inclination of the camera using the feature points extracted by the feature point extraction unit;
A displacement correction image forming unit that forms a displacement correction image by performing a displacement correction process on the previously captured image and the live image using the camera position and tilt determined by the orientation unit;
The display unit displays the displacement correction image formed by the displacement correction image forming unit;
The stereo image photographing device according to any one of claims 1 to 9. A feature point extraction unit that extracts feature points of the measurement object in the previous photographed image and the live image;
Correspondence in which the feature point extraction unit performs stereo matching between the feature point of the measurement object extracted from the live image and the feature point of the measurement object extracted from the first shot image that is a first shot image. A point search unit;
When an overlapping area is found between the live image and the first captured image by the stereo matching, the overlapping area setting unit sets the overlapping area as a round image overlapping area in the first captured image, The display unit displays the first shot image and the live image in an overlapping manner in the round image overlap region;
The stereo image photographing device according to any one of claims 1 to 10. A feature point extraction unit that extracts feature points of the measurement object in the previous photographed image and the live image;
Three-dimensional coordinates for obtaining the three-dimensional coordinates of the respective shooting positions from the screen positions of the feature points of the measurement object in the first-shot image that is the first-time shot image and the live image extracted by the feature point extraction unit With an arithmetic unit;
When an overlapping area is found between the live image and the first shot image by the three-dimensional coordinates of the shooting position in the live image and the first shot image, the overlapping area setting unit adds the first shot image to the first shot image. The overlapping area is set as a one-round image overlapping area, and the display unit displays the first-shot image and the live image in an overlapping manner in the one-round image overlapping area;
The stereo image photographing device according to any one of claims 1 to 10. When an overlap area is found between the live image and the first captured image from the shooting position in the first captured image that is the first captured image and the live image detected by the GPS or inertial sensor. The overlapping area setting unit sets the overlapping area as a one-round image overlapping area on the first-shot image, and the display unit displays the first-shot image and the live image so as to overlap with the one-round image overlapping area. ;
The stereo image photographing device according to claim 8. When the photographer has made a round, it is possible to input that it has made a round, and when the photographer has entered that it has made a round, the overlapping area setting unit sets the first shot image that is the first shot image. A round image overlapping area is set, and the display unit displays the first shot image and the live image in an overlapping manner in the round image overlapping area;
The stereo image photographing device according to claim 9. A feature point extraction unit that extracts feature points of the measurement object in the previous photographed image and the live image;
Using the feature points extracted by the feature point extraction unit, an orientation unit that obtains the camera position and tilt of the photographing unit, and using the camera position and tilt obtained by the orientation unit, a tertiary of the feature points A three-dimensional coordinate calculation unit for obtaining original coordinates;
The overlapping region setting unit uses the three-dimensional coordinates of the feature points obtained by the three-dimensional coordinate calculation unit to convert an overlapping region between the previous captured image and the live image into the previous captured image. As a live side overlap area in the live image;
The display unit includes the previous photographed image and the live image so that the same portion of the measurement object substantially overlaps in the previous overlap region and the live overlap region, and both are attached with a frame. Display to be included in the display area;
The stereo image photographing device according to any one of claims 1 to 10. In a stereo image capturing method of sequentially capturing an object to be measured while overlapping with a single camera;
A photographing step of photographing the measurement object in a timely manner to obtain a photographed image that is a still image;
A live image acquisition step of acquiring a live image of the measurement object at any time as the camera moves;
An overlapping area setting step of setting an overlapping area with the live image in the previous captured image that is a previously captured image;
The previous captured image and the live image are displayed in an overlapping manner set in the overlapping region setting step, and the previous captured image is displayed extending from the overlapping region to the opposite side of the moving direction of the photographing unit. A display step of displaying the live image extending from the overlapping region to the same side as the moving direction of the photographing unit;
The overlap region setting step, the the previous captured image overlap region between the live image and set as the previous image overlap region, the second last captured image is the second last captured photographed image and the previous captured image in the imaging step An overlapping area is set as the previous image overlapping area, and further, the live image and the previous captured image are overlapped, so that any imaging area of the measurement object is overlapped and shot;
The display step displays the previous image overlap region and the previous image overlap region, and displays a tie region that is a triple overlap region of the previous image, the previous image, and the live image;
Stereo image shooting method. The overlapping area setting step sets so that any imaging area of the measurement object in the previous captured image is included in the previous image overlapping area or the previous image overlapping area;
The stereo image photographing method according to claim 16. In a stereo image capturing method of sequentially capturing an object to be measured while overlapping with a single camera;
A photographing step of photographing the measurement object in a timely manner to obtain a photographed image that is a still image;
A live image acquisition step of acquiring a live image of the measurement object at any time as the camera moves;
An overlapping area setting step of setting an overlapping area with the live image in the previous captured image that is a previously captured image;
The previous captured image and the live image are displayed in an overlapping manner set in the overlapping region setting step, and the previous captured image is displayed extending from the overlapping region to the opposite side of the moving direction of the photographing unit. A display step of displaying the live image extending from the overlapping region to the same side as the moving direction of the photographing unit;
In the overlapping area setting step, the overlapping ratio of the overlapping area between the previous captured image and the live image is set to 55 to 90%, and the previous captured image and the previous captured image, which are captured images previously captured in the capturing process , are set. Set the overlap ratio of the overlapping area with the image to 55-90%;
The display step displays an overlapping area between the previous captured image and the live image, an overlapping area between the previous captured image and the previous captured image, and the previous captured image, the previous captured image, and the live image. Display a tie region that is a triple overlap region of
Stereo image shooting method.