JP2010256253A - Image capturing device for three-dimensional measurement and method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image capturing device for three-dimensional measurement wherein the progress of image capturing can be recognized and the image capturing can be carried out efficiently without excess and lack. <P>SOLUTION: The image capturing device 1 for three-dimensional measurement that captures an image of an article 2 to be measured by using a single camera, includes: an image capturing part 3 for acquiring a captured image; a feature point extracting part 61 for extracting a feature point from the captured image; a capturing area separating part 81 for separating the captured image into capturing areas each surrounded by three or more of feature points; a capturing area image storing part 54 for storing the separated captured images; a three-dimensional position measuring part 7 for determining three-dimensional coordinates of the feature points; a capturing range image forming part 85 that writes the respective feature points on a capturing range image for representing the capturing range of the article 2 to be measured in three-dimensional space and that sections the capturing range image into capturing areas by connecting the feature points; an insufficiently captured area determining part 86 for determining presence or absence of an insufficiently captured area; and a display part 4 for displaying the capturing range image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image photographing apparatus for three-dimensional measurement and a method thereof. More specifically, the present invention relates to a three-dimensional measurement image photographing apparatus and method for performing photographing while grasping the progress of photographing when sequentially photographing while measuring objects are overlapped by a single camera.

  In order to grasp the whole image of the measurement object and reproduce it as a three-dimensional model image, it is necessary to link the photographed images taken from a plurality of photographing positions. In order to measure the three-dimensional coordinates of the photographing apparatus or the object from the plurality of photographed images taken by the photographer in this way, the feature points (on the object) corresponding to each other on each of the two or more photographed images. Represents the same point) and needs to be tracked. In this case, since feature points inappropriate for three-dimensional measurement can be mixed in the captured image, it is possible to accurately determine the capturing position, orientation, or position coordinates of the target object of the capturing apparatus while determining the suitability of the feature points in the captured image. An image processing apparatus capable of measurement has been proposed. (See Patent Document 1)

JP 2007-183256 A (FIGS. 1 to 11, paragraphs 0021 to 0065)

  When taking a picture while moving using a single camera and performing a three-dimensional measurement of an object to be measured, it is necessary to take pictures from many directions and overlap the images, and connect the images continuously. . However, when taking a full circumference of the measurement object, when the measurement object is large or wide, or when the shape is complicated, the number of images to be taken becomes large, and the image overlap required for 3D measurement As a result, there is a problem in that the efficiency of the image becomes poor. For example, there is a problem in that the image becomes unclear or the shooting is lost. On the other hand, if the number of shots is increased in advance to compensate for the lack of shooting, there has been a problem that the efficiency of image processing and work at the time of analysis increases, resulting in poor efficiency.

  The present invention has been made in view of the above problems, and when shooting is performed while moving using a single camera, if the progress of shooting can be grasped, shooting can be performed efficiently without excess or deficiency. become. In other words, the appearance can be captured if the image is taken once, but since three-dimensional measurement uses feature points imaged twice in two photographed images, it is necessary to photograph two or more times. Further, in order to connect a captured image like a panoramic image, the feature points used for the connection need to be captured three or more times. In addition, the greater the number of times a feature point is shot, the more accurate the three-dimensional measurement of the feature point. Therefore, it has been desired to be able to grasp the progress status such as the overlapping status and the shortage status of the shooting during the shooting.

  An object of the present invention is to provide an image photographing apparatus for three-dimensional measurement capable of grasping the progress of photographing and efficiently performing photographing without excess or deficiency.

  In order to achieve the above object, the three-dimensional measurement image photographing apparatus 1 according to the first aspect of the present invention, for example, as shown in FIG. In the image photographing apparatus, the photographer captures the measurement object 2 from a plurality of photographing positions and obtains the photographed image, and the feature points of the measurement object 2 in the photographed image obtained by the photographing unit 3. A feature point extraction unit 61 to be extracted and a photographing region composed of a region (including the feature point) surrounded by three or more feature points extracted by the feature point extraction unit 61 in the region of the measurement object 2 of the photographed image. An imaging area classifying unit 81 for classifying, a 3D position measuring unit 7 for obtaining 3D coordinates of feature points from screen positions in a plurality of captured images of feature points extracted by the feature point extracting unit 61, Representing the shooting range in 3D space Each feature point for which the three-dimensional position measurement unit 7 obtains the three-dimensional coordinate is written in the coordinate space of the shadow range image, and among the feature points written in the coordinate space of the shooting range image, three or more features surrounding the shooting region are written. A shooting range image forming unit 85 that connects points to divide each shooting region and forms a shooting range image as a wire frame image, and a shooting shortage region determination unit that determines whether or not the shooting range image has a shooting shortage region 86 and the display unit 4 for displaying the shooting range image formed by the shooting range image forming unit 85.

  Here, the imaging range image is an image representing the imaging range of the measurement object 2 in a three-dimensional space. The shooting range image is initially only in the coordinate space, but each feature point is written by the shooting range image forming unit 85, and three or more feature points surrounding the shooting region are connected and partitioned for each shooting region. It is formed as a wire frame image and, if necessary, formed as a texture image by applying (pasting) the captured image to the partitioned imaging region. Further, as shooting progresses, feature points and shooting areas are added, and the range expressed as a wire frame image or texture image increases. In addition, the imaging area is an area surrounded by three or more feature points here. It is not always necessary to match the range of the captured image (the range actually captured). For example, it may be a region that can serve as a target for capturing, for example, focusing on this region. The shooting area may be the largest polygonal area surrounded by the feature points in the shot image, but in this case, if it is found that it partially overlaps with other shot images, the shot is taken with the overlapping part and non-overlapping part. Since there is a difference in the number of times, it is preferable to divide into overlapping parts and non-overlapping parts. A small area surrounded by three feature points may be used. However, since the number of shooting areas increases and the processing amount of the computer increases, it is preferable to integrate a plurality of small areas having the same number of shooting times. In order to see the overlap of the captured images, about 1/2 to 1/10 of the entire captured image is preferable, and about 1/3 to 1/5 is more preferable. In addition, even if it is considered that the area covered by all the shooting areas in the shot image is narrower than the actual shooting range and the part outside these shooting areas has not been shot, If this is counted, there is no problem in performing the necessary overlap photography because it is a safe side in the sense of ensuring duplication. The portion of the shooting area stored in the shooting area image storage unit 54 can be acquired from any shot image in which the shooting area was shot, but the portion of the shooting area shot from the direction perpendicular to the shooting area is acquired as much as possible. This is preferable because deformation at the time of applying (pasting) to the shooting range image can be reduced. In addition, as an image represented in the three-dimensional space, a three-dimensional model image composed of a perspective view and a projection diagram is typically cited, but an ortho image and an all-around image are also represented by viewing the three-dimensional space from a specific direction. It is assumed that the image is included in the image expressed in the three-dimensional space. A wire frame image refers to an image in which a three-dimensional shape of a measurement object is represented by a line, but the line does not necessarily follow the undulation of the measurement object, and includes, for example, an image in which feature points are connected by straight lines. Further, the feature points constituting the wire frame image may be extracted discretely. A texture image is an image in which a photographed image is pasted to represent the texture of the surface of the measurement object, but the surface does not necessarily follow the undulations of the measurement object. For example, the feature points are connected by straight lines. It shall include the finished surface. Further, the surface constituting the texture image may be bent, and the feature points may be extracted discretely.

  With this configuration, when shooting using a single camera while moving, the shooting range is visually indicated using the shooting range image, so the shooting progress can be grasped, and shooting is excessive or insufficient. It is possible to provide an image photographing apparatus for three-dimensional measurement that can be performed efficiently without any problems.

  The three-dimensional measurement image photographing apparatus according to the second aspect of the present invention is the three-dimensional measurement image photographing apparatus according to the first aspect, for example, as shown in FIG. An imaging area image storage unit 54 is provided, and an imaging area image forming unit 85 surrounds the imaging area with images of each imaging area stored in the imaging area image storage unit 54 in the imaging area divided into the imaging area images. A shooting range image as a texture image is formed by applying the above feature points so that the positions match.

  Here, when the image of the shooting area stored in the shooting area image storage unit 54 is applied to the shooting area divided into shooting range images, the feature point numbers of three or more feature points surrounding the shooting area are stored in the shooting area image storage. The image of the shooting area image stored in the shooting area image storage unit 54 is expanded and contracted so that the image of the shooting area stored in the section 54 matches the shooting area divided into the shooting range images. Applying (pasting) to the position of the imaging region. For example, when all of the three or more feature points surrounding the imaging area are on the same plane, they may be stretched and pasted on the two-dimensional plane. If all feature points are not on the same plane, select each of the three feature points, stretch and paste them on the two-dimensional plane, and combine them to form an image of the shooting area. good. If comprised like this aspect, since a picked-up image is affixed to a picking range image, a picking range image can be expressed like a texture image like a real thing, and the overshooting and the progress situation of photographing can be grasped visually.

The three-dimensional measurement image photographing apparatus according to the third aspect of the present invention is the same as the first or second three-dimensional measurement image photographing apparatus, for example, as shown in FIG. Each time the image capturing range is acquired, the shooting range image forming unit 85 forms a new shooting range image, and the display unit 4 displays the newly formed shooting range image.
If comprised in this way, the progress of imaging | photography can be grasped | ascertained online so that it may be said for every imaging | photography during progress of imaging | photography.

The three-dimensional measurement image photographing apparatus 1A according to the fourth aspect of the present invention is a three-dimensional measurement image photographing apparatus according to the first or second aspect. For example, as shown in FIG. After acquiring a predetermined number of images, the shooting range image forming unit 85 forms a new shooting range image, and the display unit 4 displays the newly formed shooting range image.
Here, the predetermined number can be appropriately determined, for example, by appropriately selecting the number of times of photographing the measurement object and depending on the selected number of times of photographing. Further, for example, the planned shooting position may be determined in advance, and the predetermined number may be set to the number of planned shooting positions, 1/2, 1/4, and the like. Constituting as in this aspect is efficient because image processing can be performed collectively for each predetermined number of sheets.

The three-dimensional measurement image photographing apparatus 1 according to the fifth aspect of the present invention is the three-dimensional measurement image photographing apparatus according to any one of the first to fourth aspects. For example, as shown in FIG. , An area photographing number counting unit 83 that counts the number of photographing included in different photographed images is provided, and the insufficient photographing region determining unit 86 performs photographing in the photographing range image based on the number of photographings counted by the region photographing number counting unit 83. It is determined whether or not there is an area.
Here, as the number of times of photographing in the photographing region, the number of times of photographing in which all feature points surrounding the photographing region can be extracted may be used. When configured as in this aspect, the determination is made with certainty because the determination is made based on the actual number of times of photographing.

The three-dimensional measurement image photographing apparatus 1D according to the sixth aspect of the present invention is a three-dimensional measurement image photographing apparatus according to any one of the first to fourth aspects. For example, as shown in FIG. The unit 7 obtains the three-dimensional coordinates of the shooting position from the screen positions of the extracted feature points in the plurality of shot images, and stores the scheduled shooting position storage unit 57 that stores a plurality of predetermined shooting scheduled positions. The next photographing scheduled position selection is performed in which the next photographing scheduled position is selected from the plurality of photographing scheduled positions by comparing the photographing position obtained by the position measuring unit 7 with the plurality of planned photographing positions stored in the planned photographing position storage unit 57. Unit 92, movement information calculation unit 91 for obtaining a relative position of the shooting position with respect to the next shooting scheduled position, and the photographer moves from the shooting position to the next shooting scheduled position based on the relative position obtained by movement information calculation unit 91. The As create guide information, and a movement information notification unit 93 for notifying the display unit 4.
If comprised in this way, since it image | photographs by setting an imaging | photography scheduled position, it can image | photograph systematically. In addition, since the photographer is guided to the planned shooting position by the guide information, the certainty that the shooting can be performed at the planned shooting position is high.

  The three-dimensional measurement image photographing apparatus 1E according to the seventh aspect of the present invention is the three-dimensional measurement image photographing apparatus according to the sixth aspect. For example, as shown in FIG. Captures a live image by photographing the measurement object 2 at the current position, and when the live image is obtained, the three-dimensional position measurement unit 7 obtains the three-dimensional coordinates of the current position and selects a next photographing scheduled position. The unit 92 compares the current position for which the position coordinates have been obtained by the three-dimensional position measurement unit 7 with the plurality of planned shooting positions stored in the planned shooting position storage unit 57, and schedules the next shooting from the plurality of planned shooting positions. Select a position.

  Here, the live image refers to an image obtained by the moving photographer photographing the measurement object 2 with a video camera, a digital camera, or the like at the current position. Usually, a measurement object is displayed as a live image on a display or the like while a photographer moves using a video camera or a digital camera. With this configuration, the live image can be moved to the planned shooting position while comparing the sample image with the sample image, so that the certainty that the shooting can be performed at the planned shooting position is high.

The three-dimensional measurement image photographing device 1D according to the eighth aspect of the present invention is the three-dimensional measurement image photographing device according to the sixth or seventh aspect. For example, as shown in FIG. Then, the shooting position for which the position coordinate is obtained by the three-dimensional position measuring unit 7 is stored as the already-taken position, and the next scheduled shooting position selection unit 92 stores the existing shooting position stored in the scheduled shooting position storage unit 57 from the plurality of scheduled shooting positions. The next scheduled shooting position is selected excluding the shooting position.
If comprised in this way, unnecessary duplication photography can be prevented.

The three-dimensional measurement image photographing apparatus 1D according to the ninth aspect of the present invention is the three-dimensional measurement image photographing apparatus according to the eighth aspect. For example, as shown in FIG. When all the planned shooting positions are stored as the already-captured positions in the position storage unit 57, it is determined that there is no insufficient shooting area.
With this configuration, it is possible to reliably complete shooting at a scheduled shooting position, and the probability of an insufficient shooting region is low.

  The three-dimensional measurement image photographing apparatus 1B according to the tenth aspect of the present invention is the three-dimensional measurement image photographing apparatus according to any one of the first to fourth aspects, for example, as shown in FIG. A sample image storage unit 56 that stores a shooting range image related to the simulation target object that simulates the imaging object, and the shooting shortage area determination unit 86 compares the shooting range image related to the measurement target object 2 with the shooting range image related to the simulation target object. By doing so, it is determined whether or not there is an imaging insufficient region.

  Here, the simulated object refers to, for example, an automobile similar to the measurement object or a model similar to the automobile when the measurement object is an automobile. The more similar the shape, the better. However, in the same type of car or the same car, there may be cases where you want to correct blurred parts or foreign-matter-contaminated parts, obtain higher-quality photographic images, or shoot under different conditions. It shall include type objects and measurement objects themselves. If comprised like this aspect, an imaging | photography insufficient area | region can be automatically detected by comparing the imaging | photography range image which concerns on a measurement object with the imaging | photography range image which concerns on a simulation target object. Further, the photographing range image related to the measurement object and the photographing range image related to the simulation target object may be displayed side by side on the display unit 4 so that the photographer can compare and confirm.

The three-dimensional measurement image photographing apparatus 1 of the eleventh aspect of the present invention is the three-dimensional measurement image photographing apparatus of the ninth or tenth aspect, for example, as shown in FIG. A shooting shortage information notifying unit 87 for displaying the determination result on the display unit 4 is provided.
With this configuration, the photographer can know in a timely manner whether or not shooting is insufficient by notifying the determination result.

  The three-dimensional measurement image photographing apparatus 1C according to the twelfth aspect of the present invention is the three-dimensional measurement image photographing apparatus according to any one of the first to eleventh aspects. For example, as shown in FIG. A code identification unit 63 that includes a mark having an identification code that can be identified, and that identifies the identification code of the mark extracted by the feature point extraction unit 61 is provided.

Here, the mark having an identification code that can be identified by itself and others is arranged in a pattern such as a code, a barcode, a two-dimensional barcode, and a color code using an identification code such as a numeric string, a character string, and a symbol. Contains a code with distinctiveness. Of these, in the embodiment, an example in which a color code is mainly used will be described. For example, the marks are preferably arranged on the surface of the measurement object 2 at substantially equal intervals so that the entire measurement object 2 can be grasped. In addition, it is preferable to arrange the parts at a high density so that the details can be easily grasped in a part having a complicated shape or a part having a large change.
If comprised like this aspect, feature point extraction and corresponding point search will become easy using an identification code, and three-dimensional position measurement will become efficient. In addition, since a mark having an identification code that can identify itself and others is used, it is advantageous for automating image processing such as connection of a photographed image, three-dimensional measurement of a mark position, and formation of a three-dimensional model image.

  The three-dimensional measurement image photographing apparatus 1 according to the thirteenth aspect of the present invention is the three-dimensional measurement image photographing apparatus according to any one of the first to twelfth aspects. For example, as shown in FIG. The measurement object 2 is any one of a three-dimensional model image, an ortho image, and an all-around image in which the measurement object 2 is projected from the entire periphery toward the center point and continuously arranged.

  Here, the three-dimensional model image refers to a three-dimensional image obtained when the subject is reproduced from two or more three-dimensional photographs. If feature points are arranged and constructed in the shooting range image space from the obtained three-dimensional coordinates, a three-dimensional model image is created. If projected onto a plan view in parallel projection, an ortho image is projected. If the images are projected and arranged continuously, an all-around image is obtained. If comprised like this aspect, a 3D model image etc. can be used to express the already imaged range visually in the imaged range image, and the progress status can be grasped easily and accurately.

The three-dimensional measurement image photographing device according to the fourteenth aspect of the present invention is the three-dimensional measurement image photographing device according to any one of the first to thirteenth aspects, wherein the first unit having the photographing unit 3 and the display unit 4 is The second unit having the three-dimensional position measuring unit 7 and the photographing range image forming unit 85 is configured to be movable independently, and the first unit and the second unit are configured to be communicable by wire or wirelessly.
When configured in this manner, the photographer can act quickly with a small burden. Also suitable for high-performance, high-speed computer processing.

  The three-dimensional measurement image photographing method according to the fifteenth aspect of the present invention is a three-dimensional measurement image photographing method in which, for example, as shown in FIGS. A photographing step (S100) in which the photographer photographs the measurement object 2 from a plurality of photographing positions to obtain a photographed image, and a feature point for extracting a feature point of the measurement object 2 in the photographed image obtained in the photographing process An extraction step (S120), an imaging region segmentation step (S220) for segmenting into an imaging region consisting of an area (including the feature points) surrounded by three or more feature points obtained in the feature point extraction step, and a captured image 3D to obtain 3D coordinates of feature points from the screen positions of a plurality of photographed images of feature points extracted in the feature point extraction step (S225). Each feature point obtained in the three-dimensional position calculation step is written in the coordinate space of the photographing range image representing the photographing range of the measurement object 2 in the three-dimensional space, and the photographing region is taken in the photographing range image. A shooting range image forming step (S240) in which each shooting region stored in the image storage step is connected so as to match the positions of three or more feature points surrounding the shooting region, and is divided for each shooting region, and a shooting range image Includes an undershooting region determination step (S170) for determining whether or not there is an undershooting region, and a display step (S160) for displaying the shooting range image formed in the shooting range image forming step.

  With this configuration, when shooting using a single camera while moving, the shooting range is visually indicated using the shooting range image, so the shooting progress can be grasped, and shooting is excessive or insufficient. It is possible to provide an image capturing method for three-dimensional measurement that can be performed efficiently and efficiently.

  ADVANTAGE OF THE INVENTION According to this invention, when imaging | photography while moving using a single camera, the imaging progress can be grasped | ascertained and the imaging | photography apparatus which can perform imaging | photography efficiently without excess and deficiency can be provided.

1 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus according to Embodiment 1. FIG. It is a figure for demonstrating corresponding point search. It is a figure which shows the example of a feature point and a corresponding point. It is a figure for demonstrating mutual orientation. It is a figure for demonstrating the stereo method. It is a figure which shows the example of the picked-up image by which the mark as a feature point was stuck on the earthenware which is a picked-up object. It is a figure which shows the example of the position of the feature point written in the coordinate space of the imaging | photography range image. It is a figure which shows the example of a division of the imaging region in an imaging range image. It is a figure which shows the example which distinguished the imaging | photography area | region in the imaging | photography range image with the frequency | count of imaging | photography. It is a figure which shows the example in which a photography area is added to a photography range image. It is a figure which shows the completion example of an imaging | photography range image. 6 is a diagram illustrating an example in which a captured image is pasted and displayed on an imaging range image in Embodiment 1. FIG. It is a figure which shows the example whose imaging | photography range image is an ortho image. It is a figure which shows the example whose imaging | photography range image is a wire frame image. It is a figure which shows the example whose imaging | photography range image is a perimeter image. FIG. 6 is a diagram illustrating an example of a processing flow of image shooting in the first embodiment. FIG. 6 is a diagram illustrating an example of a processing flow of imaging range image formation in the first embodiment. 6 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus according to Embodiment 2. FIG. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing in the second embodiment. FIG. 9 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus according to a third embodiment. FIG. 10 is a diagram illustrating an example of a processing flow of imaging range image formation in the third embodiment. FIG. 10 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus according to a fourth embodiment. It is a figure which shows the example of a color code target. It is explanatory drawing of the gravity center position detection using a retro target. FIG. 10 is a diagram illustrating an example of a shooting range image in the fourth embodiment. FIG. 10 is a diagram illustrating an example of a processing flow of image shooting in the fourth embodiment. It is a figure which shows the example which uses both a color code target and a retro target as a distinguishability mark. FIG. 10 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus according to a sixth embodiment. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing in the sixth embodiment. It is a figure which shows the positional relationship of a measurement object and an imaging | photography planned position. It is a figure which shows the example (the 1) in which the imaging | photography planned position was written in the imaging | photography range image. It is a figure which shows the example (the 2) which wrote in the imaging | photography range image the imaging | photography planned position. It is a figure which shows the example (the 1) which instruct | indicated the imaging position to the imaging | photography range image. It is a figure which shows the example (the 2) which instruct | indicated the imaging position to the imaging | photography range image. It is a figure which shows the example (the 1) which formed the imaging | photography area | region in the imaging | photography range image. It is a figure which shows the example (the 2) which formed the imaging | photography area | region in the imaging | photography range image. It is a figure which shows the example (the 3) which formed the imaging | photography area | region in the imaging | photography range image. It is a figure which shows the example (the 1) which pasted and displayed the imaging | photography area | region on the imaging | photography range image. It is a figure which shows the example (the 2) which stuck and displayed the picked-up image on the imaging | photography range image. FIG. 10 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus according to a seventh embodiment. It is a figure which shows the example of a display of a live image and a sample image (the 1). It is a figure which shows the example of a display of the live image and a sample image (the 2). It is a figure which shows the example of a display of a live image and a sample image (the 3). FIG. 10 is a diagram illustrating an example of a processing flow of image capturing in the seventh embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted. The following embodiments relate to a three-dimensional measurement image photographing apparatus and a photographing method for performing photographing while grasping the progress of photographing when sequentially photographing while measuring objects are overlapped by a single camera.

  In the first embodiment, the three-dimensional coordinates of the feature point of the measurement object are obtained, the photographed image is divided into photographing regions surrounded by three or more feature points, and the photographing range of the measurement object is expressed in a three-dimensional space. An example of forming a shooting range image will be described. An example in which a new shooting range image is formed and displayed each time shooting is performed will be described. In addition, an example will be described in which the presence / absence of an undershooting area is determined by counting the number of times of shooting in the shooting area and the presence / absence of a shooting area that has not reached the predetermined number of times.

[Device configuration]
FIG. 1 is a block diagram illustrating a configuration example of a three-dimensional measurement image capturing apparatus 1 according to the first embodiment of the present invention. The three-dimensional measurement image photographing apparatus 1 includes a photographing unit 3 for photographing a measurement object 2, a display unit 4 for displaying a photographed image and an image processed image, an input unit 4A for an operator to input data, instructions, and the like. A storage unit 5 that stores a photographed image and an image processed image, a feature point, the position of a photographing region, the number of times of photographing, and the like, a feature extraction unit 6 that extracts a feature point from the photographed image, a feature point of the measurement object 2, and a photographing position Three-dimensional position measurement unit 7 for obtaining three-dimensional coordinates, dividing a photographed image into photographing regions, counting the number of photographing of feature points and photographing regions, forming a photographing range image, determining the presence or absence of a photographing insufficient region, etc. An image processing unit 8 that performs image processing, a three-dimensional measurement image capturing apparatus 1, and a control unit 10 that controls each unit constituting the image processing unit 8 are provided. The storage unit 5 includes a photographed image storage unit 51, a feature point photographing number storage unit 52, a region photographing number storage unit 53, a photographing region image storage unit 54, and a feature point position relationship storage unit 55. The feature extraction unit 6 includes a feature point extraction unit 61. The three-dimensional position measurement unit 7 includes a corresponding point search unit 71, an orientation unit 72, and a three-dimensional position calculation unit 73. The image processing unit 8 includes a shooting area sorting unit 81, a feature point shooting number counting unit 82, a region shooting number counting unit 83, a measurement object boundary extraction unit 84, a shooting range image forming unit 85, a shooting shortage region determination unit 86, and a shooting. A shortage information notification unit 87 is provided. The feature extraction unit 6, the three-dimensional position measurement unit 7, the image processing unit 8, and the control unit 10 are configured in a personal computer (PC) 15.

The three-dimensional measurement image photographing apparatus 1 according to the present embodiment is suitable for photographing the entire circumference of the measurement object 2, when the measurement object 2 is large / wide, or when the shape is complicated. Yes.
The photographing unit 3 is composed of a single camera such as a video camera or a digital camera. The photographer uses a single camera as the photographing unit 3 to sequentially photograph while providing an overlapping photographing region while moving around the measurement object 2, for example. To perform three-dimensional measurement, two captured images that share an overlapping shooting area are used as stereo images, and feature points are extracted from the overlapping shooting area, and the three-dimensional coordinates of these feature points are obtained, so that shooting is performed in duplicate. It is necessary. In a video camera, a photographed image is usually stored in an external memory by a shutter operation, but if there is an internal memory, it is stored there. In a digital camera, a captured image is stored in an internal memory by a shutter operation.
The display unit 4 includes a display such as a liquid crystal display, for example. At least an imaging range image of the measurement object 2 is displayed, and feature points and imaging areas are displayed therein. In addition, a photographed image, an image processed image, an ortho image, an all-around image, and the like may be displayed. In addition, a speaker is provided for voice display.
The input unit 4A includes, for example, a mouse and a keyboard, and is used for an operator to input data and instructions.

  The storage unit 5 is composed of, for example, a hard disk and is used as a database. A photographed image storage unit 51 that stores the photographed image and the image processed image, a feature point photographing number storage unit 52 that stores the number of photographings in association with a feature point number for each feature point, and the number of photographings for each photographing region An area shooting count storage unit 53 that stores the image in association with the feature point numbers of the feature points surrounding the shooting area, a shooting area image storage unit 54 that extracts and stores an image of each shooting area from the shot image, and features about each feature point A feature point position relationship storage unit 55 is provided that stores point numbers in association with three-dimensional position coordinates and position coordinates in each captured image. Although the internal memory of the camera may be used as the captured image storage unit 51, it is preferable to use the hard disk of the PC 15 because it is suitable for various processes at high speed. In this embodiment, a camera using an external memory stores a captured image directly in the storage unit 5, and a camera using an internal memory transfers the captured image to the storage unit 5.

[Feature extraction unit]
The feature extraction unit 6 includes a feature point extraction unit 61 that extracts feature points from the captured image. The feature point extraction unit 61 extracts feature points from a plurality of captured images. The feature points include, for example, the center position, the center of gravity position, the corner position of the measurement object 2, a position having a different characteristic from others, and a sign affixed or projected on the measurement object 2. A feature extraction operator is used for feature point extraction. Here, an example in which a Moravec operator is used will be described.

  The MORAVEC operator has long been used as a general-purpose feature extractor. The MORAVEC operator uses, for example, a 3 × 3 pixel around a certain target pixel as a mask, and sets the minimum value of the density difference (direction density difference) when the mask moves one pixel in each of the four directions around the target pixel. The feature value. It is characterized by simple and high-speed processing and relatively easy hardware. In order to perform high-speed processing, a memory several times as large as an image is required. Although the feature point extraction by the Moravec operator has been described here, other operators, for example, the Harris operator and others, and anything that can detect feature points may be used.

[Three-dimensional position measurement unit]
The three-dimensional position measurement unit 7 obtains the three-dimensional position coordinates of the feature point of the photographing object 2 from the screen position in the photographed image of the feature point extracted by the feature extraction unit 6. The three-dimensional position measurement unit 7 captures the feature points extracted by the feature point extraction unit 61 based on the corresponding points obtained by the corresponding point search unit 71 and the corresponding points obtained by the corresponding point search unit 71. An orientation unit 72 that performs orientation for obtaining the position and inclination of the camera in the unit 3, and a 3D coordinate calculation unit 73 that obtains the 3D position coordinates of the feature points using the camera position and inclination obtained by the orientation unit 72.

  In the three-dimensional measurement, the corresponding points are searched and the relative orientation is performed using the feature points of the overlapping portions in two or more photographed images, and the three-dimensional coordinates of the feature points of the measurement object 2 are obtained. First, two captured images are set as a stereo image (one is a reference image and the other is a search image), and the corresponding point search unit 71 associates feature points (corresponding point search). For example, corresponding point search is performed by cross-correlation processing. In the selection of the stereo image, for example, the last captured image and the newly captured image are taken as a stereo image in the order of shooting. In addition, the existing captured image having the largest overlapping area with the new captured image and the new captured image may be set as a stereo image, and the already captured image and the new captured image having the largest number of feature points shared with the new captured image are stereo. It is good as an image.

[Correlation processing]
Cross correlation coefficient method

FIG. 2 is a diagram for explaining the corresponding point search. As shown in FIG. 2, the template image of N ₁ × N ₁ pixel is moved over the search range (M ₁ −N ₁ +1) ² in the input image of M ₁ × M ₁ pixel which is larger than the template image. The upper left position of the template image in which (a, b) is maximized is obtained, and it is considered that the template image has been searched. In the case of left and right images, for example, a template image of N ₁ × N ₁ pixel is set on the left image, a search region of M ₁ × M ₁ pixel is set on the right image, and this operation is performed on each image. What is necessary is to do about the position.
Here, the normalized cross-correlation processing has been described with respect to the corresponding point search, but other methods, for example, a residual sequential test method (SSDA) may be used.

  FIG. 3 shows examples of feature points and corresponding points. FIG. 3A shows feature points extracted from the left image, FIG. 3B shows feature points extracted from the right image, and FIG. 3C shows associated feature points. The left image and the right image form a stereo pair, and are photographed at a position slightly shifted left and right. In the left image and the right image, the positions of many points attached to the automobile as the measurement object 2 are the positions of the points extracted as feature points. Most of these feature points are associated by the corresponding point search, and are displayed in a three-dimensional coordinate space (here, a perspective view is used) in FIG. These feature points are assigned feature point numbers, and the three-dimensional position coordinates and the position coordinates in the left and right images are stored in the feature point position relationship storage unit 55.

[Calculation of external orientation elements: mutual orientation]
Next, a method for obtaining the position and tilt of the camera by the relative orientation method from the points where the left and right images are associated will be described. The relative orientation is performed by the orientation unit 72.
A model image refers to a three-dimensional image obtained when a subject is reproduced from two or more photographed images. Forming relatively similar model images is called relative orientation. That is, the relative orientation determines the position and inclination of the projection center of each of the left and right cameras so that the two corresponding light beams of the captured image meet.

FIG. 4 is a diagram for explaining relative orientation. Next, the details of the orientation calculation of each model image will be described. By this calculation, the positions of the left and right cameras (three-dimensional coordinates and three-axis tilt) are obtained.
The parameters relating to the positions of these cameras are obtained by the following coplanar conditional expression.

The origin of the model coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. For the scale, the base line length is taken as the unit length. The parameters to be obtained at this time are the rotation angle κ ₁ of the left camera, the rotation angle φ _{1 of} the Y axis, the rotation angle κ _{2 of the} right camera, the rotation angle φ ₂ of the Y axis, and the rotation of the X axis. the five of the rotation angle of the corner ω _2. In this case, since the rotation angle ω ₁ of the X axis of the left camera is 0, it is not necessary to consider.

Under such a condition, the coplanar conditional expression of (Expression 1) becomes as shown in (Expression 2), and each parameter can be obtained by solving this expression.

Here, between the model coordinate system XYZ and the camera coordinate system xyz, the following coordinate transformation relational expressions (formula 3) and (formula 4) hold.

Using these equations, unknown parameters are obtained by the following procedure.
(A) The initial approximate value of the unknown parameter is normally 0.
(B) The coplanar conditional expression (Expression 2) is Taylor-expanded around the approximate value, and the value of the differential coefficient when linearized is obtained by (Expression 3) and (Expression 4), and an observation equation is established.
(C) A least square method is applied to obtain a correction amount for the approximate value.
(D) Correct the approximate value.
(E) Using the corrected approximate value, the operations (b) to (e) are repeated until convergence.
By obtaining unknown parameters (κ ₁ , φ ₁ , κ ₂ , φ ₂ , ω ₂ ), the position and tilt of the camera can be obtained.
If the position of the camera is obtained by the relative orientation method, it is possible to obtain the three-dimensional coordinates on the object space point, that is, the three-dimensional coordinates of the feature points by the stereo method.

[Stereo method]
FIG. 5 is a diagram for explaining the stereo method.
For simplicity, two cameras (C1 and C2) are used, the optical axes are parallel, the distance c from the camera lens principal point to the CCD plane is equal, and the CCD is placed perpendicular to the optical axis. It shall be. Let B be the distance between the two optical axes (baseline length).

The following relationship exists between the coordinates of the point P (x, y, z) on the object and the coordinates of the points P1 (x1, y1) and P2 (x2, y2) on the photographing screens of the cameras C1 and C2. .

x1 = cx / z --- (Formula 5)
y1 = y2 = cy / z --- (Formula 6)
x2-x1 = cB / z --- (Formula 7)

However, the origin of the entire coordinate system (x, y, z) is taken as the lens principal point of the camera C1.

Z is obtained from (Equation 7), and x and y are obtained from (Equation 5) and (Equation 6) using this.
If the position coordinates of the obtained feature points can be mapped to the imaging range image, the imaging range image is partitioned by selecting and connecting three or more feature points surrounding the imaging area from the mapped feature points, and the imaging range image is obtained. Can be formed as a wire frame image.

  If the three-dimensional coordinates of each feature point are obtained, a three-dimensional model image, an ortho image, and an all-around image can be formed. If feature points are arranged and constructed in the shooting range image space from the obtained three-dimensional coordinates, a three-dimensional model image is projected. If projected onto a plan view in parallel projection, an ortho image is projected. If it is arranged continuously, it becomes an all-around image. That is, the shooting range image can be expressed by a three-dimensional model image, an ortho image, and an all-around image. Further, it can be expressed by a texture image, a wire frame image, or the like. In addition, the three-dimensional model image can be expressed from an arbitrary direction using a perspective view or a projection view. Further, the accuracy of the shooting range image can be improved by using the three-dimensional coordinates.

[Image processing unit]
Returning to FIG. 1, the image processing unit 8 divides the region of the measurement object 2 into a shooting region composed of a region (including the feature point) surrounded by three or more feature points in the shot image. For each feature point, a feature point shooting number counting unit 82 for counting the number of shooting times included in a different shooting image, and a region for counting the number of shooting times included in a different shooting image for each shooting region. The feature point whose position coordinates are measured by the shooting number counting unit 83, the measurement object boundary extraction unit 84 for extracting the boundary between the measurement object 2 and the background, and the three-dimensional position measurement unit 7 is written in the shooting range image, A shooting range image forming unit 85 that connects three or more feature points that surround the shooting region among the feature points written in the coordinate space of each other to divide each shooting region and form a shooting range image as a wire frame image; An imaging missing area determination unit 86 determines whether the shadow area image capturing missing area there has a photographic missing information notification unit 87 for displaying the determination result of the photographing missing area determination unit 86 to the display unit 4.

  The imaging range image is an image that represents the imaging range of the measurement object 2 in a three-dimensional space. The shooting range image is initially only in the coordinate space, but each feature point is written by the shooting range image forming unit 85, and three or more feature points surrounding the shooting region are connected and partitioned for each shooting region. It is formed as a wire frame image and, if necessary, formed as a texture image by applying (pasting) the captured image to the partitioned imaging region. Further, as shooting progresses, feature points and shooting areas are added, and the range expressed as a wire frame image or texture image increases.

  The imaging area classification unit 81 classifies the area of the measuring object 2 into imaging areas in the captured image. The region of the measuring object 2 can be divided into a number of regions by a polygon having three or more feature points as vertices. In the present embodiment, since it is a goal to grasp the overlapping situation of the shooting areas, it is preferable to divide the shooting area into a size corresponding to the range of the shot image. If the number of shooting areas is increased, the image processing takes time, and if the shooting area is made too large, the number of shots cannot be counted out of the shot image. Therefore, the shooting area is appropriately sized according to the range of the shot image. Is preferred. For example, a range that is substantially in one plane and in focus is divided into 1 to 4 according to the area. The shooting area may be the maximum polygonal area surrounded by the feature points in the shot image (the range actually shot), but in this case, if it is found that some other shot images overlap, Since there is a difference in the number of times of photographing between the overlapping portion and the non-overlapping portion, it is preferable to divide into an overlapping portion and a non-overlapping portion. Although a small area surrounded by three feature points may be used, it is preferable to integrate a plurality of small areas having the same number of times of shooting because the number of shooting areas increases and the processing amount of the computer increases. In order to see the overlap of the captured images, about 1/2 to 1/10 of the entire captured image is preferable, and 1/3 to 1/5 is more preferable.

  The object boundary extraction unit 84 extracts the boundary between the measurement object 2 and the background in the captured image. For the boundary between the measurement object 2 and the background in the captured image, feature points are also extracted from the boundary and its surroundings. Whether these feature points belong to the measurement object 2 or the background can be determined from the coordinate positions of the feature points measured three-dimensionally. That is, the three-dimensional coordinate changes intermittently at the boundary between the measurement object 2 and the background. In this way, the boundary between the measurement object 2 and the background can be extracted. Note that the feature points on the boundary usually belong to the measurement object 2, so that these feature points and the feature points on the measurement object 2 side can be connected to form an imaging region. In this case, for example, regarding the distance difference between the roof of the car and the hood, it may be determined that the distance difference is within the threshold and not the boundary between the car and the background, and another imaging region is formed by the roof and the hood. It is also possible to select one close to the focus and photograph the other again. It is also possible to extract the boundary from the difference in color between the measurement object 2 and the background. For example, the object boundary extraction unit 84 is made to recognize the color of the measurement object 2, and a portion having a color different from these is determined as the background. If the background is set to a specific color such as white or black, it is preferable to extract the difference between the color of the measurement object 2 and the background. In this case, the background is preferably a single pattern.

  The feature point photographing number counting unit 82 counts the number of photographing included in different photographed images for each feature point. That is, the feature point is extracted from each captured image stored in the captured image storage unit 51, and the number of times the feature point can be extracted is defined as the number of times the feature point is captured. The counted number of shootings is stored in the feature point shooting number storage unit 52 in association with the feature point number of the feature point. The feature points are extracted even if the number of times of shooting is one, and a feature point number is assigned and stored in the feature point shooting number storage unit 52. The feature point photographing number storage unit 52 does not need to store the number of photographing times for all feature points, and may select and store at least a sufficient number of feature points to form a photographing region.

  The area photographing number counting unit 83 counts the number of photographing included in different photographed images for each photographing area. That is, feature points surrounding the shooting area are extracted from each captured image stored in the storage unit 5, and the number of times that all feature points surrounding the shooting area can be extracted is set as the number of shooting times of the shooting area. If even one of these feature points is missing, the entire imaging area has not been imaged and is not counted. The counted number of photographing is stored in the region photographing number storage unit 53 in association with the feature point number of the feature point surrounding the photographing region. Even if the number of times of shooting is one, a shooting region is extracted, and a shooting region number is assigned and stored in the region shooting number storage unit 53. In addition, even if it is considered that the area covered by all the shooting areas in the shot image is narrower than the actual shooting range and the part outside these shooting areas has not been shot, If this is counted, there is no problem in performing the necessary overlap photography because it is a safe side in the sense of ensuring duplication.

  First, the shooting range image forming unit 85 writes the feature points whose position coordinates are measured by the three-dimensional position measurement unit 7 in the coordinate space of the shooting range image.

  FIG. 6 shows an example of a photographed image in which marks (color code target and retro target) are attached to the earthenware that is the photographing object 2. A color code target (see FIG. 23) and a retro target will be described later in Example 4, but are used here as feature points.

  FIG. 7 shows the positions of the feature points written in the coordinate space of the shooting range image. The position of the feature point in FIG. 7 is displayed by writing the position of the mark in FIG. 6 in the coordinate space of the shooting range image. The position of the feature point is displayed as a square □ in the coordinate space of the perspective view of the imaging range image, and the corresponding feature point number (1 to 24) is displayed.

  Secondly, the shooting range image forming unit 85 connects three or more feature points surrounding the shooting area among the feature points written in the coordinate space of the shooting range image, and divides the shooting range image for each shooting area. Is formed as a wire frame image. That is, the feature point number is associated with the feature point written in the shooting range image. Therefore, a polygon is formed by connecting the position coordinates of the feature points of the photographing range image having the same feature point number as the three or more feature points surrounding the photographing region, and the polygon is formed in the photographing region image. It will fall under. In this way, for all the shooting areas (except for the invisible part such as the vehicle bottom) divided in the shot image, the same feature point codes in the shooting range image are attached to three or more feature points surrounding the shooting area. By capturing and segmenting the position coordinates of the feature points thus formed, the shooting range image can be formed as a wire frame image.

  FIG. 8 shows an example of a section of the shooting area in the shooting range image. This is an example in which an automobile area is divided into a plurality of imaging areas in a coordinate space of a perspective view. Most of these areas are quadrangular, and an area surrounded by at least three feature points is divided as one imaging area.

  FIG. 9 shows an example in which the shooting area in the shooting range image is distinguished by the number of times of shooting. The imaging range image is formed as a wire frame image. This is an example in which an automobile area is divided into a plurality of shooting areas and displayed in different colors in the coordinate space of the perspective view. The area displayed by color coding (by pattern in the figure) is an overlapping shooting area, the area displayed in white is a single shooting area, and the area displayed in black is an unshooting area. In this way, by color-coding the shooting area, it is possible to visually grasp the shooting shortage area, and the efficiency of shooting is promoted. According to the present embodiment, the three-dimensional coordinates of the feature points are not determined unless they are photographed in duplicate, so that only the overlapping photographing areas are displayed in the photographing range image. However, it is possible to display a single shooting area and an unphotographed area by predicting the shooting area and the positions of three or more feature points surrounding it in advance and inputting the predicted value in the shooting range image. It is. When the three-dimensional coordinates of the feature points are measured, the predicted value is replaced with the measured value. This shooting range image is an example of such a case.

  FIG. 10 shows an example in which shooting areas are sequentially added to the shooting range image. The measurement object 2 is an automobile, and the imaging range image is shown as a wire frame image in a perspective view. Each time the imaging unit 3 acquires a new captured image, the imaging range image forming unit 85 forms a new imaging range image, and the display unit 4 displays the newly formed imaging range image. For this reason, every time shooting is performed, a new shooting area is formed, and new shooting areas are added to the shooting range image. As the shooting progresses, shooting areas are sequentially added to the shooting range image. In FIG. 10A, a shooting area Q1 is formed, in FIG. 10B, a shooting area Q2 is added, in FIG. 10C, a shooting area Q3 is added, and in FIG. 10D, a shooting area Q4 is added. Has been.

  FIG. 11 shows a completed example of the shooting range image. After FIG. 10D, several shooting areas (Q5, etc.) are added to complete the shooting range image. The measurement object 2 is an automobile, and the imaging range image is shown as a wire frame image in a perspective view. Thus, it can be understood that the photographing is completed when all the photographing regions of the measurement object 2 are shown in the photographing range image. Note that although the opposite side cannot be seen only from the shooting range image viewed from one direction, it is not possible to determine the completion of shooting, but it is also possible to display an image from the opposite side, thereby determining the completion of shooting.

  FIG. 12 shows an example in which a photographic image is pasted and displayed on the photographic range image. The imaging range image is shown as a three-dimensional model image (texture image). A wagon car is represented in the coordinate space of the photographing range image represented by the perspective view, and the photographed image is applied (pasted) to each partitioned photographing region. For example, the image of the shooting area is extracted from the shot image and stored in the shooting area image storage unit 54, and the three-dimensional coordinates are measured in the coordinate space of the shooting range image by the shooting range image forming unit 85. A point is written, an image of each shooting area is extracted from the shooting area image storage unit 54, and each of the three or more feature points surrounding the shooting area matches the position of the feature point written in the shooting range image. A shooting range image as a texture image is formed by expanding and contracting the image of the shooting region. The portion not represented is a non-overlapping photographing region. Thereby, the shooting range image can be expressed like a real thing, and the shooting progress can be visually grasped.

  FIG. 13 shows an example in which the shooting range image is an ortho image. FIG. 13A shows a photographed image, and FIG. 13B shows an example in which an ortho image as a photographing range image is formed based on these photographed images. The ortho image is formed by arranging feature points in the imaging range image space from the three-dimensional coordinates obtained by the three-dimensional position measuring unit 7 and projecting them in parallel on a plan view. The overlapping shooting area is expressed like a real thing. Each time shooting is performed, a shooting area is added.

  FIG. 14 shows an example in which the shooting range image is a wire frame image. An ortho image as a photographing range image is divided into photographing regions, and each photographing region is displayed in a color-coded manner (by pattern in the drawing). Each time shooting is performed, a shooting area is added.

  FIG. 15 shows an example in which the shooting range image is an all-around image. In the all-around image, feature points are arranged in the imaging range image space from the three-dimensional coordinates obtained by the three-dimensional position measurement unit 7, and the center projection (center point in FIG. 14 is (1.50, It is formed by projecting each model image on a plane and arranging them continuously. FIG. 15 shows images from seven directions around and two directions in height. The entire image (in a state where shooting has been completed) is an image from 12 directions and 2 heights.

Returning to FIG. 1, the undershooting area determination unit 86 determines whether or not there is an undershooting area in the shooting range image. In the present embodiment, it is determined whether or not there is an undershooting region in the shooting range image based on the number of shootings counted by the region shooting number counting unit 83. Even if the number of times of photographing is one, a photographing region is formed, a photographing region number is given, and the number of photographing times is stored in the region photographing number storage unit 53. If there is an area where the number of times of shooting is one, there will be an insufficient shooting area. In addition, since images are taken while being overlapped, if there is an unphotographed area, there is always an area where the number of times of photography is one. Therefore, even if the unphotographed area is not represented in the photographing range image, it can be determined whether or not the photographing range image includes an insufficient photographing area based on the number of photographing counted by the region photographing number counting unit 83. The determination as to whether or not there is an insufficient photographing area is made every time photographing is performed. That is, every time shooting is performed, a shooting range image is newly formed, and it is determined whether or not there is a shooting shortage area. As described above, in this embodiment, the undershooting area determination unit 86 determines the presence or absence of an undershooting area, but it is also possible for the photographer to determine by looking at the shooting range image.
The undershooting area notifying unit 87 causes the display unit 4 to display the determination result of the undershooting area determining unit 86. By looking at the determination result displayed on the display unit 4, the photographer can know whether or not shooting is insufficient each time shooting is performed.

  The control unit 10 has a control program in a built-in memory, and controls the three-dimensional measurement image photographing device 1 and each part constituting the three-dimensional measurement image photographing device 1, controls the flow of signals and data, and serves as a three-dimensional measurement image photographing device. Execute the function.

[Processing flow]
FIG. 16 shows an example of a processing flow of image capturing. An example in which a new shooting range image is formed and displayed each time the shooting unit acquires a new shot image will be described.
First, the measurement object 2 is imaged by the imaging unit 3 (imaging process: S100). The captured image is stored in the captured image storage unit 51 (captured image storage step: S105). Next, feature points are extracted (feature point extraction step: S120). That is, feature points are extracted by the feature point extraction unit 61 from the captured image. Next, the corresponding point search unit 71 associates the feature points of two overlapping captured images from a plurality of captured images (corresponding point search step: S130). For the two captured images, for example, the last captured image and the newly captured image are taken as stereo images in the order of shooting. In addition, an already captured image having the largest overlapping area with a new captured image and a new captured image may be selected as a stereo image, and an already captured image having the largest number of feature points shared with the new captured image and a new captured image may be selected. The image may be selected as a stereo image. At this time, the already-captured image is set as a reference image, and a new captured image is set as a search image. Next, the three-dimensional coordinates of each feature point associated by the corresponding point search unit 71 are calculated (three-dimensional coordinate calculation step: S140). The orientation unit 72 performs orientation, obtains the position and tilt of the camera, and the three-dimensional position calculation unit 73 calculates the three-dimensional coordinates of each feature point. The measurement object boundary extraction unit 84 extracts the boundary between the measurement object 2 and the background from the three-dimensional coordinates of the feature points. Next, a shooting range image is formed by the shooting range image forming unit 85 (shooting range image forming step: S150).

  FIG. 17 shows an example of a processing flow of the photographing range image forming step (S150). First, the imaging area classification unit 81 classifies the area of the measurement object 2 in the captured image into imaging areas consisting of areas (including the feature points) surrounded by three or more marks, and assigns imaging area numbers. (Shooting area dividing step: S220). The captured images divided into the imaging areas are stored in the imaging area image storage unit 54 (imaging area image storage step: S225). On the other hand, the feature point photographing number counting unit 82 counts the number of photographing included in different photographed images for each feature point. Then, the area imaging frequency counting unit 83 counts the imaging frequency included in different captured images for each imaging area (area imaging frequency counting step: S230). For example, when all the feature points surrounding each shooting area are shot, the number of shootings is counted assuming that the shooting area is shot. If even one of these feature points is not photographed, the entire photographing area is not photographed, and therefore is not counted as the number of photographing. Then, the area shooting count storage unit 53 stores the counted shooting count of each shooting area in association with the feature point numbers of the feature points surrounding the shooting area (area shooting count storage step: S235).

  Next, in the imaging range image forming unit 85, each feature point obtained by the three-dimensional position measuring unit 7 is written in the coordinate space of the imaging range image representing the imaging range of the measurement object 2 in the three-dimensional space, and imaging is performed. Of the feature points written in the coordinate space of the range image, three or more feature points surrounding the shooting region are connected and partitioned for each shooting region, and the shooting range image is formed as a wire frame image (shooting range image forming step). : S240). That is, the feature point numbers are associated with three or more feature points surrounding the shooting regions classified by the shooting region sorting unit 81, and the feature points written in the shooting range image having the same feature point numbers as these feature points. By connecting the lines, a polygon is formed, which corresponds to the shooting area in the shooting range image. Next, the shooting area in the shooting range image is color-coded or the shot image is pasted on the shooting area (S250). When the shooting areas are color-coded, these shooting areas are typically overlapping shooting areas, and the overlapping shooting areas are color-coded for each shooting area. Thereby, it is possible to broadly divide the overlapping shooting area and the non-overlapping shooting area. It is also possible to represent a single shooting area and an unphotographed area by previously predicting the position of the shooting area and three or more feature points surrounding it and inputting the predicted value in the shooting range image. It is. In addition, when pasting a shot image in the shooting area, the image of each shooting area is extracted from the shooting area image storage unit 54, and the positions of three or more feature points surrounding the shooting area in the shooting area divided into shooting range images. The image of each shooting area is stretched and applied (pasted) so as to match. Thereby, the shooting range image is formed as a three-dimensional model image (texture image), the shooting range image can be expressed like a real thing, and the progress of shooting can be visually grasped. Further, from the photographing range image, an ortho image may be formed by parallel projection and an all-around image may be formed by center projection.

  Returning now to FIG. On the display unit 4, the shooting range image formed by the shooting range image forming unit 85 is displayed (display step: S160). By displaying the shooting area by color or by displaying a shooting range image with a shot image pasted in the shooting area, it is possible to visually grasp the shooting progress. Next, the undershooting area determination unit 86 determines whether or not there is an undershooting area in the shooting range image (undershooting area determination step: S170). In the present embodiment, it is determined whether or not there is an undershooting region in the shooting range image based on the number of shootings counted by the region shooting number counting unit. Then, the determination result of the insufficient photographing region determination unit 86 is displayed on the display unit 4 by the photographing insufficient region notification unit 87. The determination as to whether or not there is an insufficient photographing area is made every time photographing is performed. That is, every time shooting is performed, a shooting range image is newly formed, and it is determined whether or not there is a shooting shortage area. If it is determined that there is an insufficient photographing region, the process returns to the photographing step (S100), and the processing from the photographing step (S100) to the photographing insufficient region determination step (S170) is repeated. If it is determined that there is no undershooting area, the image shooting process flow ends.

  According to the present embodiment, when shooting while moving using a single camera, the shooting range is visually clearly shown using the shooting range image, so that the shooting progress can be grasped, and shooting is not excessive or insufficient. An image photographing apparatus for three-dimensional measurement that can be efficiently performed can be provided.

  In the first embodiment, an example is described in which a new shooting range image is formed and displayed each time shooting is performed. However, in the second embodiment, feature point extraction, three-dimensional coordinate calculation, and shooting range image collection are performed after shooting a predetermined number of times. An example will be described in which formation is performed and displayed, and if there is an undershooting area, further shooting is performed.

  FIG. 18 is a block diagram illustrating a configuration example of the three-dimensional measurement image capturing apparatus 1A according to the second embodiment. An imaging number determination unit 31 is added to the configuration (see FIG. 1) of the three-dimensional measurement image capturing apparatus according to the first embodiment. The number-of-shoots determination unit 31 is provided in the PC 15 and includes a number-of-shoots counter (not shown). The number of shots acquired by the number-of-shoots counter is counted to determine whether the planned number of shots has been reached. The scheduled number of times of shooting can be set to the number of times of shooting counter from the input unit 4A. Further, when the number of times of photographing reaches the number of times of scheduled photographing, a reset signal is output, the number of times of photographing is reset, and a reset signal is transmitted to the feature extraction unit 6. Further, the display unit 4 displays that the scheduled number of times of shooting has been reached, and the photographer waits without shooting until a new shooting range image is displayed.

  FIG. 19 shows an example of a processing flow of image shooting in the second embodiment. A shooting number determination step (S110) is inserted between the captured image storage step (S105) and the feature point extraction step (S120) in FIG. The number of shootings is determined based on whether or not a reset signal is output from the shooting number counter. When the feature extraction unit 6 receives the reset signal, the feature extraction unit 6 starts extracting feature points from a number of captured images corresponding to the scheduled number of times of capturing. Then, a feature point extraction step (S120) to a display step (S160) are performed. In the corresponding point search step (S130), a stereo image is selected. For example, the i-th and (i-1) -th (i.gtoreq.2) photographed images are made stereo images in the order of photographing. Also, any one of the 1st to (i-1) -th captured images having the largest overlapping area with the i-th captured image and the i-th captured image may be a stereo image, and the number of shared feature points with the i-th captured image is Any one of the 1st to i-1st photographed images and the i-th photographed image that are the most common may be stereo images.

  In the case of the first embodiment, since the progress status can be grasped for each sheet, the certainty is high. However, in the second embodiment, since the processing can be performed collectively, the processing time of the entire image photographing process can be shortened. Other configurations and processing flow are the same as in the first embodiment, and as in the first embodiment, it is possible to provide a three-dimensional measurement image photographing apparatus that can grasp the progress of photographing and can efficiently perform photographing without excess or deficiency. .

  In the first embodiment and the second embodiment, the example in which the three-dimensional coordinates of the feature points are obtained and the shot image is divided into shooting areas and the shooting range image is generated has been described. In the third embodiment, the shooting area image is divided. First, an example of forming a shooting range image by converting coordinates from the shot image will be described. Further, the determination of the presence / absence of an imaging insufficient region is performed by storing an imaging range image of a simulated object that simulates the measurement object, and comparing the imaging range image of the measurement object with the imaging range image of the simulation object Will be explained.

  FIG. 20 is a block diagram illustrating a configuration example of the three-dimensional measurement image capturing apparatus 1B according to the third embodiment. Compared to the configuration of the first embodiment (see FIG. 1), in the storage unit 5, a feature point position relationship storage unit 55 and a sample image storage unit 56 are added, a feature point shooting number storage unit 52, and a region shooting number storage unit. 53, the shooting area image storage unit 54 is deleted. Further, in the image processing unit 8, an image coordinate conversion unit 88 is added, and the imaging region sorting unit 81, the feature point imaging number counting unit 82, and the area imaging number counting unit 83 are deleted. The feature point position relationship storage unit 55 assigns feature point numbers to the feature points extracted by the feature extraction unit 6, the 3D coordinates obtained by the 3D position measurement unit 7, and the position on the screen in each captured image. Are stored in association with each other. The sample image storage unit 56 stores an imaging range image for a simulated object similar to the measurement object 2. The simulated object is an object similar to the measurement object or a model of the object. The more similar the shape, the better. However, there may be cases where you want to correct blurring or foreign matter contamination with the same type of object or the same object, to obtain a higher quality shot image, or to shoot under different conditions. The object includes the same type of object and the measurement object itself. In this embodiment, the imaging range image for the simulated object has an imaging insufficient area by comparing the imaging range image for the measurement object with the imaging range image for the simulated object in the imaging insufficient area determination unit 86. It is used to determine whether or not. Typically, the position of the feature point for the simulated object is three-dimensionally measured, and a shooting range image without an undershooting region is formed using these feature points. However, it is also possible to predict the shooting area and the positions of the feature points surrounding the shooting area in advance, store the shooting range image without the shooting shortage area created using this predicted value, and compare them using this.

  Further, the image coordinate conversion unit 88 performs coordinate conversion such as affine conversion and Helmart conversion. The shooting range image forming unit 5 stores the feature point position data (three-dimensional measurement values, position coordinates on the screen of the shot image) stored in the feature point position relationship storage unit 55 in the shot image storage unit 51. The stored photographed image is applied to the photographing range image after coordinate conversion so that the position of each feature point of the photographed image matches the position of each feature point in the photographing range image measured in three dimensions. Thereby, a photographing range image as a texture image is formed. The coordinate conversion is performed using the image coordinate conversion unit 88 using affine conversion or Helmat conversion. In FIG. 20, the number-of-shoots determination unit 31 is described. This is because a shooting range image can be formed after a predetermined number of shot images are acquired as in the second embodiment ( May be omitted).

  FIG. 21 shows an example of a processing flow for forming a shooting range image in the third embodiment. Compared to the processing flow in the first embodiment (see FIG. 17), the photographing range image forming process is different. This process is indicated by S150A. First, feature point numbers are assigned to the feature points extracted by the feature extraction unit 6 in the feature point position relation storage unit 55, and the three-dimensional coordinates obtained by the three-dimensional position measurement unit 7 and the screen in each captured image are displayed. The upper position is stored in association with each other (feature point position relationship storage step: S215). For example, the measurement object boundary extraction unit 84 extracts the boundary between the measurement object and the background from the three-dimensional coordinates of each feature point. Next, the shooting range image forming unit 85 forms a shooting range image by coordinate conversion (shooting range image forming step: S250A). That is, the captured image stored in the captured image storage unit 51 is stored on the basis of the position data of each feature point stored in the feature point positional relationship storage unit 55 (three-dimensional measurement values, position coordinates on the screen of the captured image). The image capturing range image is formed as a texture image by applying coordinate conversion to the image capturing range image so that the position of each feature point of the image capturing image matches the position of each feature point in the image capturing range image. Further, in the undershooting area determination step (S170), the undershooting area determination unit 86 compares the shooting range image related to the measurement object 2 with the shooting range image related to the simulated object, so that there is an undershooting area. Determine whether or not. That is, a shooting range image of a simulation target similar to the measurement target 2 is stored in the sample image storage unit 56 in advance, and the shooting shortage region determination unit 86 stores the shooting range image related to the measurement target as the simulation target. It is determined whether or not there is an undershooting area as compared with the shooting range image according to.

  Compared to the first embodiment, the coordinate transformation is performed in place of the process of dividing the shooting area and the process of pasting the shot image. However, the coordinate conversion is performed using the same feature points as three or more feature points surrounding the shooting area. In this case, almost the same shooting range image can be obtained. If a feature point used for conversion is selected, a texture image can be obtained directly by coordinate conversion because it is not divided into shooting regions, but it is not possible to display shooting range images that are divided for each shooting region and color-coded. Other configurations and processing flow are the same as those of the first embodiment, and as in the case of the first embodiment, the image capturing apparatus for three-dimensional measurement that can grasp the progress of shooting and can efficiently perform shooting without excess or deficiency. Can provide.

  In the above embodiment, the example of obtaining the three-dimensional coordinates by extracting the feature points of the measurement object has been described. However, in the fourth embodiment, a color code target having an identification code that can be identified by the measurement object is used. An example of pasting and using as a feature point will be described.

  FIG. 22 is a block diagram illustrating a configuration example of the three-dimensional measurement image capturing apparatus 1C according to the fourth embodiment. Compared to the configuration of the first embodiment (see FIG. 1), a mark extraction unit 62 and a code identification unit 63 are added to the feature extraction unit 6. The mark extraction unit 62 extracts a mark such as a color code target or a retro target as a feature point from the photographed image. The code identification unit 63 identifies the identification code of the mark extracted by the mark extraction unit 71, and associates the identification code with the feature point number assigned to the mark. Further, the three-dimensional position measurement unit 7 uses the mark extracted by the mark extraction unit 62 as a feature point, and obtains a three-dimensional coordinate of the mark position in the same manner as other feature points. Note that a shooting number determination unit 31 is added so that a shooting range image can be formed after a predetermined number of shot images are acquired as in the second embodiment, and coordinate conversion can be performed as in the third embodiment. In addition, a feature point position relationship storage unit 55 and a sample image storage unit 56 are added to the storage unit 5, and an image coordinate conversion unit 88 is added to the image processing unit 8 (these may be omitted).

[Color code target]
FIG. 23 shows an example of the color code target CT. 23A shows three color code unit areas, FIG. 23B shows six color code targets, and FIG. 23C shows nine color code targets. The color code targets CT (CT1 to CT3) in FIGS. 23A to 23C include a position detection pattern (retro target portion) P1, a reference color pattern (reference color portion) P2, and a color code pattern (color code portion). P3 and an empty pattern (white portion) P4.

  The retro target portion P1 is used for detecting the target itself, detecting its center of gravity, detecting the direction of the target, and detecting the target area.

  The reference color portion P2 is used for reference at the time of relative comparison and for color calibration for correcting color misregistration in order to cope with color misregistration due to shooting conditions such as illumination and a camera. Furthermore, the reference color portion P2 can be used for color correction of the color code target CT created by a simple method. For example, when using a color code target CT printed by a color printer (inkjet, laser, sublimation type printer, etc.) where color management is not performed, individual differences appear in the color of the printer used, but the reference color portion P2 By comparing and correcting the color of the color code portion P3 relative to each other, the influence of individual differences can be suppressed.

  The color code part P3 expresses a code by a combination of color schemes for each unit area. The number of codes that can be expressed varies depending on the number of code colors used for the code. For example, when the number of code colors is n, n × n × n codes can be represented. In order to increase the reliability, even when the condition that the colors used in other unit areas are not used redundantly, n × (n−1) × (n−2) codes can be expressed. If the number of code colors is increased, the number of codes can be increased. Furthermore, if the condition that the number of unit areas of the color code portion P3 is equal to the number of code colors is imposed, all the code colors are used for the color code portion P3. Therefore, not only the comparison with the reference color portion P2, By relatively comparing the colors between the unit areas of the color code part P3, the color of each unit area can be confirmed to determine the identification code, and the reliability can be improved. Furthermore, if a condition for making the area of each unit region all the same is added, the color code target CT can also be used for detection from the image. This is because the area occupied by each color is the same even between the color code targets CT having different identification codes, and thus almost the same dispersion value is obtained from the detection light from the entire color code portion. In addition, since the boundary between the unit areas is repeated at equal intervals and a clear color difference is detected, the target CT can be detected from the image from such a repeated pattern of detection light.

  The white portion P4 is used for detecting the orientation of the color code target CT and for color misregistration calibration. Among the four corners of the target CT, there is a place where no retro target is arranged, and this can be used for detecting the direction of the target CT. Thus, the white part P4 should just be a pattern different from a retro target. Therefore, a character string such as a number for visually confirming the code may be printed on the white portion, or may be used as a code area such as a barcode. Further, in order to increase the detection accuracy, it can be used as a template pattern for template matching.

  The mark extraction unit 62 extracts marks in a plurality of captured images acquired by the imaging unit 3 as feature points. In this embodiment, a color code target as a mark having an identification code that can be identified by itself is attached to a car or earthenware that is a measurement object, and the color code target can be used as a feature point.

  The code identification unit 63 identifies the identification code of the color code target CT. Using the color code target correspondence table that records the correspondence between the pattern arrangement and the code number, the identification code is determined from the color arrangement in the color code portion P3 of the color code target CT, and a number is assigned to the color code target CT.

  The position of the color code target CT is obtained by detecting the retro target P1. When a retro target is affixed in addition to the color code target CT, the mark position can be obtained by detecting the retro target even when a retro target is affixed instead of the color code target CT.

  FIG. 24 is an explanatory diagram of center-of-gravity position detection using a retro target. In this embodiment, the retro target is formed by two concentric circles. In FIG. 24A1, the brightness of the inner circle portion 204 that is the inner side of the small circle among the concentric circles is bright, and the brightness of the outer circle portion 206 that is an annular portion formed between the small circle and the great circle is dark. The retro target 200, FIG. 24 (A2) is a brightness distribution diagram in the diameter direction of the retro target 200 of (A1), and FIG. 24 (B1) is a retro target in which the brightness of the inner circle portion 204 is dark and the brightness of the outer circle portion 206 is bright. 200 and FIG. 24 (B2) show the brightness distribution diagram in the diameter direction of the retro target 200 of (B1). When the brightness of the inner circle portion 204 is bright as shown in FIG. 24 (A1), the retro target has a bright portion with a large amount of reflected light at the center of gravity in the photographed image of the measurement object 1, and thus the light amount distribution of the image. As shown in FIG. 24A2, the inner circle portion 204 and the center position of the retro target can be obtained from the threshold value To of the light amount distribution. In this embodiment, in the case of (A1), it suffices if one circle corresponding to a small circle is formed at the center of the square position detection pattern P1, and the outside of the great circle may be bright or dark. However, it is better to make it dark. When it is dark, substantially only one circle (small circle) is formed. In the case of (B1), the outside of the great circle may be bright or dark, but when the outside of the great circle is bright, only one dark circle (small circle) is formed on the substantially bright base. It is.

When the target existence range is determined, the barycentric position is calculated by, for example, the moment method. For example, the plane coordinates of the retro target 200 shown in FIG. 24 (A1) are (x, y). Then, (Expression 8) and (Expression 9) are calculated for points in the x and y directions in which the brightness of the retro target 200 is greater than or equal to the threshold value To.
xg = {Σx * f (x, y)} / Σf (x, y) ---- (Equation 8)
yg = {Σy * f (x, y)} / Σf (x, y) −−−− (formula 9)
(Xg, yg): coordinates of the center of gravity position, f (x, y): light quantity on (x, y) coordinates In the case of the retro target 200 shown in FIG. (Expression 8) and (Expression 9) are calculated for the points in the x and y directions. Thereby, the position of the center of gravity of the retro target 200 is obtained.

  FIG. 25 shows an example of the shooting range image in the fourth embodiment. The earthenware on which the color code target shown in FIG. 6 is pasted is used as the measurement object 2, and the color code target is extracted from the photographed image, its three-dimensional position coordinates are obtained, the photographing region is divided into the photographing range image, The captured image is pasted on the capturing area to form a capturing range image as a texture image. The overlapping shooting area is represented as a texture image, and the non-overlapping shooting area remains a space. As a feature point, only a retro target in a color code target may be used. However, if another retro target or a feature point is also used, a highly accurate shooting range image can be obtained. FIG. 25 shows the positions of color code targets (A1 to A8) and retro targets (T1 to T8) as feature points on the surface of the earthenware. One color code target has three retro targets. In this way, if the shooting range image is expressed as a texture image in a real-like manner, the progress of shooting can be visually grasped. Note that a shooting range image may be formed by coordinate conversion instead of dividing into shooting regions.

  FIG. 26 shows an example of a processing flow of image shooting in the fourth embodiment. Compared to the processing flow of Embodiment 1 (see FIG. 16), the color code target extraction step (S120A) is provided instead of the feature point extraction step (S120), and the color code target extraction step (S120A) and the corresponding point search step. There is a code identification step (S125) between (S130). In the color code target extraction step (S120A), the mark extraction unit 62 extracts color code targets in a plurality of photographed images acquired by the photographing unit 3 as feature points. In the code identification step (S125), the code identification unit 63 identifies the identification code of the color code target CT. Using the color code target correspondence table that records the correspondence between the pattern arrangement and the code number, the identification code is determined from the color arrangement in the color code portion P3 of the color code target CT, and a number is assigned to the color code target CT. By using a mark having an identification code that can identify itself as a feature point, feature point extraction and corresponding point search are facilitated using the identification code, and three-dimensional position measurement becomes efficient. Other configurations and processing flow are the same as in the first embodiment, and as in the first embodiment, it is possible to provide a three-dimensional measurement image photographing apparatus that can grasp the progress of photographing and can efficiently perform photographing without excess or deficiency. .

  In the fourth embodiment, an example is described in which a color code target having an identification code that can be identified by another person is attached to a measurement object and used as a feature point. However, in the fifth embodiment, the mark itself is not discriminating. An example in which the mark is used as an identification mark will be described. That is, even a retro target that does not have its own distinguishability can be given distinctiveness by the arrangement relationship with a mark color code target having a code that can identify itself.

  FIG. 27 is a diagram illustrating an example in which both a color code target and a retro target are used as distinguishability marks. Color code targets (A10 to A13) and retro targets (T10, T11, etc.) are affixed to the earthenware as the measurement object 2. For example, the retro target T10 is in a triangle formed by the color code targets (A10 to A12), and if the positional relationship is specified with respect to the color code targets (A10 to A12), discriminability occurs. Further, the retro target T11 is in a triangle formed by the color code targets (A11 to A13), and if the positional relationship is specified with respect to the color code targets (A11 to A13), discriminability occurs. For example, it can be specified by a direction and distance from a mark having a specific code, a projection point (intersection of perpendicular lines) and a distance on a triangle formed by three marks having a specific code, and the like. When this is used, a retro target (T10, T11, etc.) that is a non-code mark can be used as a distinguishability mark. Alternatively, this arrangement relationship may be projected onto a three-dimensional space, and non-code marks may be identified from the consistency of the arrangement relationship. A non-code mark identification unit (not shown) identifies a non-code mark from the arrangement relationship between the non-code mark extracted by the mark extraction unit 62 and a mark having another self-identifiable code, and the identification relationship is improved by the arrangement relationship. A feature point number is assigned to the generated non-code mark. As a result, a large number of retro targets can be used as discriminating marks, so that highly accurate three-dimensional measurement is possible even with a small number of color code targets.

  In the above embodiment, the shooting position is not specified. However, in the sixth embodiment, the coordinates of the planned shooting position are input in advance, the shooting is sequentially performed at the planned shooting position, and the photographer is moved to the next shooting planned position. An example of guiding will be described.

  FIG. 28 is a block diagram illustrating a configuration example of the three-dimensional measurement image capturing apparatus 1D according to the sixth embodiment. Compared to the configuration of the fourth embodiment (see FIG. 22), a planned shooting position input unit 41 is added to the input unit 4A, a planned shooting position storage unit 57 is added to the storage unit 5, and the guide information processing unit 9 Is added, and a shooting number determination unit 31 is added. The guide information processing unit 9 acquires information on a predetermined scheduled shooting position and the current position of the photographer, and creates information for guiding the photographer to the planned shooting position. A next scheduled shooting position selection unit 92 and a movement information notification unit 93 are provided. The scheduled shooting position input unit 41 inputs a plurality of predetermined scheduled shooting positions, and the scheduled shooting position storage unit 57 stores these scheduled shooting positions input from the planned shooting position input unit 41. The number-of-shoots determination unit 31 is used to determine the presence / absence of an insufficient shooting region. In other words, the shooting shortage area determination unit 86 determines that there is no shooting shortage area when all the planned shooting positions are stored as the already shot positions in the planned shooting position storage unit 57 (when the number of shootings reaches the planned shooting position). judge. The guide information processing unit 9 is configured in a personal computer (PC) 15.

  Also, the 3D position measurement unit 7 uses the 3D position calculation unit 73 from the screen positions in the plurality of captured images of the feature points extracted by the feature extraction unit 6 together with the 3D coordinates of the feature points as the shooting position (shooting unit). 3 present position) is obtained. A mark having a distinguishable identification code such as a color code mark is used as a feature point, and for example, it is arranged on the ground around the automobile 2 or the like as a measurement object. When the position of the extracted mark (color code target, retro target, etc.) is known, the DLT method can be used to obtain the three-dimensional coordinates of the feature point of the measurement object 2 and the shooting position.

[DLT method]
The DLT method approximates the relationship between the photographic coordinates and the three-dimensional coordinates (feature point coordinates) of the subject using a cubic projective transformation equation.
The basic expression of the DLT method is (Expression 10).

If the denominator is deleted from (Equation 10), the following line form can be derived.

Furthermore, when (Expression 11) is modified, the following expression is obtained.

When (Expression 12) is solved directly using the least square method, 11 unknown variables L _{1 to} L ₁₁ that determine the relationship between the photographic coordinates and the target point coordinates can be acquired. Therefore, the relationship between the photographic coordinates (x, y) of the subject and the three-dimensional coordinates (feature point coordinates) (X, Y, Z) of the subject can be obtained by solving (Equation 12) for six points whose position coordinates are known. Thus, the position coordinates (0, 0, 0) of the photographing position corresponding to the convergence point of the straight line connecting the photograph coordinates (x, y) and the feature point coordinates (X, Y, Z) can be obtained. In this case, the photographing position is a position (−X, −Y, −Z) with respect to the feature point coordinates.

[Texture mapping]
In addition, the 3D position measurement unit 7 uses, for example, texture mapping to capture the shooting position (shooting unit) from the screen positions in the plurality of shot images of the feature points extracted by the feature point extraction unit 61 in the 3D position calculation unit 73. 3 present position) is obtained.
The three-dimensional coordinates obtained by texture mapping can be used as the position of the feature point to be written in the shooting range image. In this way, the accuracy of the shooting range image can be improved. Next, a method for texture mapping an image photographed on a photograph onto a model image formed on spatial coordinates (X, Y, Z) will be described. The spatial coordinates of each pixel (pixel) on the image are calculated. In this processing, the image coordinates (x, y) on the photograph are converted into spatial coordinates (X, Y, Z). Spatial coordinates (X, Y, Z) are values calculated by three-dimensional measurement. Spatial coordinates (X, Y, Z) corresponding to image coordinates (x, y) of a photograph are given by the following equations. In this way, the density acquisition position of each pixel on the image is obtained, and the image is mapped on the three-dimensional space.

Here, (X ₀ , Y ₀ ): Upper left position of the photographic image in the spatial coordinate system, (ΔX, ΔY): Size of one pixel in the spatial coordinate system, (x, y): Image of the photographic image Coordinates (X, Y, Z): Spatial image coordinates a, b, c, d: Coefficients of a plane equation formed by a plurality of reference points for interpolating a certain image coordinate (x, y).

  This coefficient is, for example, a coefficient of a plane equation of triangle interpolation processing (Triangulated Irregular Network, TIN). TIN generates a mesh having triangular units as a method for interpolating three-dimensional coordinates, and is also called a triangular network. (For more information about TIN, see Masao Iri, Takeshi Koshizuka: Computational Geometry and Geographic Information Processing, pp 127, "Franz Aurenhammer, Atsuyoshi Sugihara: Voronoi Diagram, Introduction to One Basic Geometric Data Structure, ACM Computing (See “Surveys, Vol. 23, pp 345-405”).

  If the feature points of the photographic image can be mapped to the photographic range image, the photographic range image divided for each photographic region can be formed. As a result, it is possible to visually discriminate between the overlapping shooting area and the non-overlapping shooting area with respect to the shooting area, and to improve the shooting efficiency. In addition, by applying (pasting) a captured image to each imaging region, a three-dimensional model image (texture image) can be formed as shown in FIG. In addition, an ortho image and an all-around image can be formed from the three-dimensional model image. As a result, the existing shooting range can be visually represented in the shooting range image, and the progress status can be easily and accurately grasped.

  The movement information calculation unit 91 calculates the planned shooting position from the three-dimensional coordinates of the shooting position (current position of the shooting unit 3) obtained by the three-dimensional position calculation unit 73 and the planned shooting position stored in the planned shooting position storage unit 57. The direction and distance of the shooting position with respect to is obtained. The next scheduled shooting position selection unit 92 selects the next scheduled shooting position from a plurality of planned shooting positions based on the obtained position coordinates of the shooting position. The next scheduled shooting position selection unit 92 includes a position coordinate of a plurality of scheduled shooting positions and a scheduled shooting position table that stores whether shooting has been performed or not shot at the scheduled shooting position. With reference to the scheduled shooting position table, For example, the distance between the obtained shooting position and each scheduled shooting position (a distance that bypasses the measurement object when it is in between) is compared, and the calculated shooting position among the unscheduled shooting positions is the shortest The scheduled shooting position at a distance is selected as the next scheduled shooting position. The movement information notification unit 93 creates information to guide the photographer from the position information of the shooting position and the next shooting scheduled position, and displays the information on the display unit 4. The shooting number determination unit 31 is provided in the PC 15 and has a shooting number counter (not shown), and counts the number of shot images acquired by the shooting number counter. The number of scheduled shooting positions is set as the scheduled number of shots from the input unit 4A in the shooting number counter. The undershooting area determination unit 86 refers to the number-of-shoots determination unit 31 to determine whether or not the planned number of shots has been reached. When all the planned shooting positions are stored as the existing shooting positions in the planned shooting position storage unit 57, the count of the shooting number counter reaches the planned shooting number, so it can be determined whether or not the planned shooting number has been reached. That is, the number-of-shooting determination unit 31 has the same configuration as that in the second embodiment, but is different in that the number of planned shooting positions is used as the number of scheduled shootings and that the determination is performed after displaying guide information instead of after shooting. . The number-of-shoots determination unit 31 counts a flag indicating the already-captured position in the scheduled shooting position storage unit 57 and determines whether all the scheduled shooting positions are stored as the already-captured positions in the scheduled shooting position storage unit 57. You may do it.

  FIG. 29 shows an example of a processing flow of image shooting in the sixth embodiment. First, the scheduled shooting position input unit 41 inputs a plurality of predetermined shooting scheduled positions (shooting scheduled position input step: S400), and the scheduled shooting position storage unit 57 inputs these from the scheduled shooting position input unit 41. Is stored (scheduled shooting position storage step: S401). Next, the measurement object 2 is imaged by the imaging unit 3 (imaging process: S410). The captured image is stored in the captured image storage unit 51 (captured image storage step: S415). Next, feature points are extracted (feature point extraction step: S420). Next, the corresponding point search unit 71 associates the feature points of two overlapping captured images from a plurality of captured images (corresponding point search step: S430). The shooting step (S410) to the corresponding point search step (S430) are the same as the shooting step (S100) to the corresponding point search step (S130) of the first embodiment (see FIG. 16). Next, the corresponding point search unit 71 calculates the three-dimensional coordinates of each feature point and calculates the three-dimensional coordinates of the shooting position (three-dimensional coordinate calculation step: S440). The orientation unit 72 performs orientation, obtains the position and tilt of the camera, and the 3D position calculation unit 73 calculates 3D coordinates of each feature point and imaging position. When the three-dimensional coordinates of the photographing position are calculated, guide information for the photographer is created by the guide information processing unit 9. First, in the movement information calculation unit 91, from the three-dimensional coordinates of the shooting position (current position of the shooting unit 3) obtained by the three-dimensional position calculation unit 73 and the planned shooting position stored in the planned shooting position storage unit 57, The direction and distance of the shooting position with respect to the planned shooting position are obtained (movement direction calculation step: S450). Next, the next scheduled shooting position selection unit 92 selects the next scheduled shooting position from a plurality of planned shooting positions based on the obtained position coordinates of the shooting position. Next, the movement information notification unit 93 creates guide information for guiding the photographer from the shooting position to the next scheduled shooting position based on the position information of the shooting position and the next scheduled shooting position, and displays the guide information on the display unit 4 (moving). Guide display step: S460). Next, it is determined whether or not the number of times of shooting has reached the scheduled number of times of shooting in the number of times of shooting determination unit 31 (shooting number determination step: S470). The number of shootings is determined based on whether or not a reset signal is output from the shooting number counter. The imaging process (S400) to the movement guide display process (S460) are repeated until the reset signal is output. When the reset signal is received, the shooting range image forming unit 85 forms a shooting range image from the number of shot images corresponding to the planned number of shots (shooting range image forming step: S480).

  The shooting range image forming step (S480) is the same as the shooting range image forming step (S150) of the first embodiment (see FIG. 17). Next, the imaging range image formed by the imaging range image forming unit 85 is displayed on the display unit 4 (display process: S490). The display step (S490) is the same as the display step (S160) of Example 1 (see FIG. 17). Next, the undershooting area determination unit 86 determines whether or not there is an undershooting area in the shooting range image (undershooting area determination step: S500). In the present embodiment, the scheduled shooting position already shot in the scheduled shooting position storage unit 57 is stored as a previously shot position, and shooting is performed when all scheduled shooting positions are stored in the scheduled shooting position storage unit 57. It is determined that there is no shortage area. Then, the determination result of the insufficient photographing region determination unit 86 is displayed on the display unit 4 by the photographing insufficient region notification unit 87. The determination as to whether or not there is an undershooting area is made when the number of shooting times reaches the number of planned shooting positions and the shooting range image is displayed. If it is determined that there is an undershooting area, the process returns to the scheduled shooting position input step (S400), and the processes from the shooting process (S410) to the undershooting area determination step (S500) are repeated. In this case, in the planned shooting position input step (S400), typically the planned shooting position excluding the existing shooting position is input. However, when a quality defect such as blurring occurs in the photographed image at the already photographed position, it can be input as the scheduled photographing position. If it is determined that there is no undershooting area, the image shooting process flow ends.

  FIG. 30 is a diagram showing the positional relationship between the measurement object and the planned shooting position. The positional relationship between the photographing object 2 and the planned photographing position is expressed two-dimensionally and will be referred to as a photographing position image 13. In the shooting position image 13, the planned shooting position is displayed around the installation area 14 of the shooting target 2. Numbers are assigned to the planned shooting positions (numbers 1 to 8 are shown in circles), and the object 2 is observed and photographed from each of the planned shooting positions (1 to 8 in the circles). The direction is represented by an arrow. The photographer may move to the scheduled shooting position in the order of the numbers assigned to the scheduled shooting position and sequentially shoot the shooting target object 2.

  FIG. 31 shows an example (part 1) in which the planned shooting position is written in the shooting range image. The planned shooting position is indicated by a plus mark + in a plan view as a shooting range image, and numbers 1001 to 1032 are attached. Numbers with missing (1022, 1024, 1025, 1029) are uninput planned shooting positions, but will be input in the future. The measurement object 2 is scheduled to be placed at the center (0.150, 0.125), and the planned shooting position is arranged so as to surround the entire circumference. Typically, three-dimensional coordinates (X, Y, Z) are input, but a plan view is suitable for viewing the planar arrangement.

  FIG. 32 shows an example (part 2) in which the planned shooting position is written in the shooting range image. In the perspective view as the imaging range image, the planned shooting position is indicated by a square □ and numbered. The measurement object 2 is scheduled to be installed at the center (0, -1.500, 0), and the scheduled shooting position is arranged so as to shoot from an obliquely upward direction so as to surround this from the entire periphery. . Generally, three-dimensional coordinates (X, Y, Z) are input. A perspective view is suitable for viewing the three-dimensional arrangement.

  FIG. 33 shows an example (part 1) in which the scheduled shooting position is indicated in the shooting range image. In the plan view as the photographing range image, the pre-photographed photographing position is indicated by a plus mark + and numbered. Unscheduled shooting scheduled positions are not marked with a plus mark + or a number. In the figure, the planned shooting positions with numbers 1003 and 1006 are not yet shot. Among these, an arrow is displayed at the planned shooting position number 1003, designated as the next shooting scheduled position, and the shooting direction is indicated by the direction of the arrow.

  FIG. 34 shows an example (part 2) in which the scheduled shooting position is indicated in the shooting range image. In the perspective view as the imaging range image, the already-scheduled shooting scheduled position is indicated by a square □ and numbered. A square □ and a number are not attached to unscheduled shooting scheduled positions (one place in the figure). An arrow (thick arrow in the figure) is displayed at the unscheduled shooting scheduled position, designated as the next shooting scheduled position, and the shooting direction is indicated by the direction of the arrow.

  FIG. 35 shows an example (part 1) in which a shooting area is formed in a shooting range image. In a plan view (wire frame image) as an imaging range image, an already taken imaging scheduled position is indicated by a plus mark + and numbered. Unscheduled shooting positions are not marked with a plus mark or number. In the figure, the planned shooting position of number 1006 is not shot. In addition, the automobile 2 as the measurement object is divided into shooting areas, the non-shooting area is shown in black (in the figure, a fine mesh pattern), the single shooting area is white, and the overlapping shooting area is shown with a pattern. According to the present embodiment, the three-dimensional coordinates of the feature points are not determined unless they are photographed in duplicate, so that only the overlapping photographing areas are displayed in the photographing range image. However, it is possible to display a single shooting area and an unphotographed area by predicting the shooting area and the positions of three or more feature points surrounding it in advance and inputting the predicted value in the shooting range image. It is. When the three-dimensional coordinates of the feature points are measured, the predicted value is replaced with the measured value. This shooting range image is an example of such a case.

  FIG. 36 shows an example (part 2) in which a shooting area is formed in the shooting range image. In the perspective view (wire frame image) as the photographing range image, the photographing planned position is indicated by a square □. The earthenware 2 as a measurement object is installed at the center, and among the shooting areas of the earthenware 2, the overlapping shooting areas are displayed in different colors (by patterns in the drawing), and the non-overlapping shooting areas are not displayed. In the figure, the scheduled shooting position for shooting from the direction of the arrow is an unshooted position (one location). If shooting is performed from this position, all the shooting areas of the earthenware 2 become overlapping shooting areas.

  FIG. 37 shows an example (part 3) in which a shooting area is formed in the shooting range image. FIG. 37 is a perspective view (wire frame image) after shooting from an unshooted scheduled shooting position in FIG. 36 is performed. The non-overlapping imaging area is eliminated from the imaging area of the earthenware 2.

  FIG. 38 shows an example (part 1) in which a shooting area is pasted and displayed on a shooting range image. In the perspective view (three-dimensional model image, texture image) as the imaging range image, the imaging planned position is indicated by a square □. A photographed image is applied (pasted) to an already photographed area of the automobile 2 as a measurement object. Among these, the single photographing area is displayed darkly. In addition, a photographed image is not pasted in the unphotographed area and is shown in black. In this embodiment, if the images are not overlapped, the three-dimensional coordinates of the feature points are not determined, and therefore, only the overlapped region is not displayed in the capturing range image. However, it is possible to display a single shooting area and an unphotographed area by predicting the shooting area and the positions of three or more feature points surrounding it in advance and inputting the predicted value in the shooting range image. It is. When the three-dimensional coordinates of the feature points are measured, the predicted value is replaced with the measured value. This shooting range image is an example of such a case.

  FIG. 39 shows an example (part 2) in which the shooting area is pasted and displayed on the shooting range image. In the perspective view (three-dimensional model image, texture image) as the imaging range image, the imaging planned position is indicated by a square □. In the figure, the planned shooting position for shooting from the direction of the arrow is an unshooted position (one place). The photographed image is applied (pasted) to the overlapping photographing region of the earthenware 2 as the measurement object. A photographed image is not pasted in the non-overlapping photographing region, and is a space. The inside of the pottery is displayed in gray. In the figure, one scheduled shooting position is an unshooted position, and if shooting is performed from this position, all the shooting areas of the earthenware 2 become overlapping shooting areas.

  According to the present embodiment, it is possible to provide a three-dimensional measurement image photographing method capable of grasping the progress of photographing and efficiently performing photographing without excess or deficiency. Furthermore, it is possible to plan the shooting by determining the planned shooting position and guiding the photographer from the current position to the next scheduled shooting position.

  In the sixth embodiment, the coordinates of the scheduled shooting position are input in advance, and shooting is sequentially performed at the scheduled shooting position, and the photographer is guided to the next scheduled shooting position. In the seventh embodiment, a live image is further described. An example in which a sample image is displayed and guided will be described.

  FIG. 40 is a block diagram illustrating a configuration example of the three-dimensional measurement image capturing apparatus according to the seventh embodiment. Compared to the configuration of the sixth embodiment (see FIG. 28), a live image storage unit 58 is added to the storage unit 5. The photographing unit 3 is configured by, for example, a video camera or a digital camera, and captures and acquires a photographed image that is a still image by a shutter operation or the like. In addition, live images taken by the video camera or digital camera while the photographer moves are always overwritten in the temporary memory of the storage unit 5 and transferred to the display unit 4. The display unit 4 displays a live image on a display such as a liquid crystal display or a viewfinder, for example. When the photographer wants to know the current position, a live image is obtained by photographing the measurement object 2 at the current position by a shutter operation, and is used for three-dimensional position measurement of the photographing position and guide information processing. Therefore, the live image storage unit 58 has a temporary memory and a memory for storing a live image at the current position. The live image is stored for a predetermined period (temporarily) in the memory that stores the live image at the current position. For example, the predetermined period may be a time sufficient to display guide information for the next scheduled shooting position. The live image stored in the memory for storing the live image at the current position may be single or plural. The photographed image acquired by the shutter operation is distinguished from the live image by inputting, for example, 1 for a photographed image and 2 for a live image from an input key (not shown) by the photographer. In addition, the storage unit 5 stores, in the sample image storage unit 56, a sample image of a simulated object that has been captured in advance with the same relative positional relationship as the planned shooting position. The same relative positional relationship means that the direction and the distance between the photographing unit 3 and the measurement object 2 are equal to the direction and the distance between the photographing apparatus that photographed the sample image and the simulated object. Each sample image is stored in association with the scheduled shooting position. The display unit 4 displays at least a live image at the current position of the photographing unit 3 and a sample image of the simulated object photographed at the next photographing scheduled position on the same screen. It is preferable to display both images side by side or side by side because they are easy to compare. However, if they are displayed on the same screen, they can be compared. In addition, the display unit 4 has a speaker when performing voice guidance.

  41 to 43 show display examples of live images and sample images. The live image 11 is displayed on the left side and the sample image 12 is displayed on the right side. The sample image 12 of the simulated object is, for example, when the object to be imaged 2 is an automobile, the relative positional relationship (direction and distance) that is substantially the same as the planned imaging position for other automobiles (the more similar the shape is). ) Refers to an image selected from a photographed image (still image) of a car that has already been photographed. The display unit 4 selects and displays the sample images stored in the sample image storage unit 58 of the storage unit 5 from the images associated with the next shooting scheduled position. In addition, a shooting position image 13 that two-dimensionally represents the positional relationship between the shooting target and the planned shooting position is displayed. In the shooting position image 13, the installation area 14 of the shooting object 2 is displayed, the shooting scheduled positions are numbered (indicated by the numbers 1 to 8 in the circles), and the respective shooting planned positions (◯ The direction in which the object 2 is observed and photographed from 1) to 1) is indicated by arrows. The photographer may move to the scheduled shooting position in the order of the numbers assigned to the scheduled shooting position and sequentially shoot the shooting target object 2. Shooting is performed by the shutter operation, and when the photographer inputs 1 from the input key, a captured image is acquired, the guide information unit 9 selects the next scheduled shooting position, and the sample image 12 is an image of the next scheduled shooting position. Change. In order to know the current position during movement, the photographer inputs 2 from the input key when acquiring a live image.

  The display screen in FIG. 41 shows a case where the photographing unit 3 is at the planned photographing position (2 within a circle). In the live image 11 and the sample image 12, a car photographed from the right front is displayed, and the directions of the cars are substantially the same. In the shooting position image 13, the next scheduled shooting position (2 in circles) is displayed in red, for example, and the other scheduled shooting positions (1, 3 to 8 in circles) are displayed in black, for example.

  The display screen of FIG. 42 shows a case where the photographing unit 3 is in the middle of the planned photographing position (1 in ○) and the planned photographing position (8 in ○). For example, it is a case where the photographer has overtaken when moving to the scheduled shooting position (8 in the circle) after shooting at the planned shooting position (1 to 7 in the circle). The live image 11 shows a car taken from the front and slightly left, the sample image 12 shows the car taken from the front left, and the shooting position image 13 shows the next shooting position (8 in circle). Is displayed in red, for example, and other scheduled shooting positions (1 to 7 in a circle) are displayed in black, for example. In addition, the live image 11 is displayed with a text “Please move to the right” and an arrow in the right direction is displayed. The guide information is notified so as to move in the direction of 8). In addition, an arrow directed to the planned shooting position (8 in ○) is displayed between the planned shooting position (1 in ○) and the planned shooting position (8 in ○) of the shooting position image 13, and the photographer takes a picture. The guide information is notified so as to move in the direction of the planned position (8 in the circle). Note that the guide information may display either one of the character display and the two arrow displays, and may announce “Please move to the right” by voice. If the photographer moves clockwise with respect to the car 8 in accordance with this guide information, the next scheduled shooting position (8 in ○) can be reached.

  FIG. 43 shows the case where the next shooting scheduled position (8 in circle) is reached. Since the direction of the car in the live image 11 and the sample image 12 substantially coincides with each other, it can be seen that the next shooting scheduled position (8 in ○) has been reached. In FIG. 44, a character display indicating that “OK” has been reached in the live image 11 or the shooting position image 13 is displayed. The guide information may display one of the two OK displays, or may announce “OK” by voice. When the captured image is acquired, the guide information unit 9 selects the next scheduled shooting position, and the sample image 12 changes to an image associated with the next scheduled shooting position. If shooting is performed at all scheduled shooting positions (1 to 8 in a circle), the image associated with the next scheduled shooting position disappears and the sample image 12 becomes blank. In this case, the sample image 12 may be displayed with a letter “shooting complete” or may be announced with a voice “shooting complete”.

  The movement information calculation unit 91 calculates the planned shooting position from the three-dimensional coordinates of the shooting position (current position of the shooting unit 3) obtained by the three-dimensional position calculation unit 73 and the planned shooting position stored in the planned shooting position storage unit 57. The direction and distance of the shooting position with respect to is obtained. The movement information calculation unit 91 compares the live image 11 and the sample image 12 at the current position of the photographing unit 3, that is, the photographer, calculates the moving direction of the photographing unit 3 with respect to the next scheduled photographing position, and moves information The notification unit 93 may display the movement direction calculated by the movement information calculation unit 91 on the display unit 4. For example, a large number of sample images 12 are stored in the database in association with the planned shooting positions (1 to 8 in the circles), and a pattern relatively similar to the live image 11 is extracted from the patterns of these sample images, and the live image is extracted. Compare with 11. For example, if the pattern is similar to both the planned shooting position (1 in ○) and the planned shooting position (2 in ○), the imaging unit 3 determines the planned shooting position (1 in ○) and the planned shooting position. If it is determined that it is in the middle of (2 in ○) and it is more similar to the sample image at the planned shooting position (1 in ○), it is closer to the planned shooting position (1 in ○). If it is more similar to the sample image at the scheduled shooting position (2 in the circle), it is determined that it is closer to the planned shooting position (2 in the circle). The similarity between the pattern of the live image 11 and the pattern of the sample image 12 is obtained by, for example, extracting feature points from the live image 11 and the sample image 12 and statistically processing the distance between the corresponding feature points. Also good. In addition, sample images associated with positions other than the planned shooting positions (1 to 8 in the circle) (for example, sample images shot with the same relative positional relationship as the middle of the planned shooting positions, samples shot from diagonally above) Image etc.) may also be used as a comparison target.

  The relative position (direction, distance) of the photographing unit 3 with respect to the planned photographing position is determined from the direction and size of the measurement object in the photographed image. For example, sample images taken from a number of directions and distances are stored in the image storage unit 6 in association with directions and distances, and the obtained live image 11 is compared with these sample images and has high similarity. The image is extracted, and the relative position of the photographing unit 3 with respect to the planned photographing position is determined. When the car is reflected in the acquired live image, the photographing unit 3 is near the car. When the car is small, it is determined that the photographing part 3 is far from the car. Further, when the live image 11 is acquired from the direction of the planned shooting position (1 in circle), it is the front of the car, and when it is acquired from the direction of the planned shooting position (2 in circle), the right side of the car is scheduled to be shot. If acquired from the direction of the position (3 within ○), the right side of the car, if acquired from the direction of the planned shooting position (4 within ○), the right rear of the car, the planned shooting position (5 within ○) When acquired from the direction of the rear of the car if acquired from the direction, from the direction of the left side of the car when acquired from the direction of the planned shooting position (6 in circle), to the left of the car if acquired from the direction of the planned position of shooting (7 within ○) On the other hand, when it is acquired from the direction of the planned shooting position (8 in ○), it is determined as the left front of the automobile. In addition, when it is acquired from these intermediate positions, for example, between the planned shooting position (1 in ◯) and the planned shooting position (2 in ◯), it is determined to be slightly in front of the automobile.

  The movement information notification unit 93 creates guide information so that the photographer moves from the current position of the photographing unit 3 to the next scheduled photographing position, and causes the display unit 4 to notify the user. The movement information notification unit 93 notifies the guide information based on the determination of the relative position of the imaging unit 3 with respect to the next imaging scheduled position by the movement information calculation unit 91. For example, when the photographing unit 3 is at the planned photographing position (1 in ○) or the planned photographing position (2 in ○), the guide information need not be created, and “OK” may be created. . Also, for example, when the planned shooting position (1 in ○) and the planned shooting position (8 in ○) are in the middle, and the next shooting planned position is the planned shooting position (8 in ○), “to the right “Please move” guide information is created and displayed on the display, which is the display unit 4, or announced on the speaker. Also, for example, when the planned shooting position (1 in circle) and the planned shooting position (2 in circle) are in the middle, and the next shooting scheduled position is the planned shooting position (2 in circle), “to the left “Please move” guide information is created and displayed on the display, which is the display unit 4, or announced on the speaker. Further, the guide information may be information that guides the direction and the distance, but the guide related to the distance can also be expressed by the direction, so that only the direction may be guided. Further, when the distance between the measurement object 2 and each scheduled shooting position (1 to 8 in the circle) is predetermined (for example, 5 m), such as when the measurement object 2 is photographed from the surroundings, photographing from a distant place is possible. When the influence of the distance can be ignored, a lateral guide such as “to the right” or “to the left” may be used.

  FIG. 44 shows an example of a processing flow of image shooting in the present embodiment. Compared with the processing flow of the sixth embodiment (see FIG. 29), a live image display step (S405) is inserted between the scheduled shooting position input step (S400) and the shooting step (S410). That is, the live image is displayed on the display unit 4. At that time, a sample image associated with the first scheduled shooting position is displayed together with the live image on the display unit 4. If the photographer determines that the first next scheduled shooting position has been reached while comparing the live image 11 and the sample image 12, the photographed image is acquired by a shutter operation (S410). The photographing step (S410) to the three-dimensional coordinate calculation step (S440) are the same as in the sixth embodiment. In the movement direction calculation step (S450), the movement information calculation unit 91 uses the three-dimensional coordinates of the shooting position (the photographer's current position acquired from the live image) obtained by the three-dimensional position calculation unit 73 and the planned shooting position. From the planned shooting position stored in the storage unit 57, the direction and distance of the shooting position with respect to the planned shooting position are obtained. In the movement guide display step (S460), information for guiding the photographer is created and displayed on the display unit 4. First, the next shooting scheduled position selection unit 92 selects the next shooting scheduled position from a plurality of shooting scheduled positions based on the obtained position coordinates of the shooting position, and then the movement information notification unit 93 Guide information for guiding the photographer from the shooting position to the next shooting scheduled position is created from the position information of the shooting position and the next shooting scheduled position, and the guide information is displayed on the screen including the live image and the sample image on the display unit 4. The

  After the movement guide display step (S460), the process proceeds to the photographing number determination step (S470). The steps after the shooting number determination step (S470) are the same as those in the sixth embodiment except that the return destination from the shooting number determination step (S470) is the live image display step (S405) instead of the shooting step (S410). It is.

  According to the present embodiment, it is possible to provide a three-dimensional measurement image photographing method capable of grasping the progress of photographing and efficiently performing photographing without excess or deficiency. Furthermore, it is possible to accurately guide the photographer from the current position to the next scheduled shooting position using the live image.

In the eighth embodiment, an example will be described in which the three-dimensional measurement image capturing apparatus can be separated into a plurality of units and can communicate with each other by wire or wirelessly.
For example, the first unit includes the photographing unit 3 and the display unit 4, and the second unit includes the PC 15, the input unit 4A, and the storage unit 5. The first unit is configured to be movable independently of the second unit. The first unit is portable, and the photographer can carry the first unit and take a picture while moving, and can act quickly with little burden. The second unit includes a three-dimensional position measurement unit 7 and an image processing unit 8 (including a shooting range image forming unit 85) in the PC 15, and can perform arithmetic processing and image processing. By installing the PC 15 in an office or the like, it is suitable for high-performance and high-speed computer processing. The first unit and the second unit are configured to be communicable by wire or wirelessly, and can perform image photographing processing by transmitting and receiving data and commands via communication. Other configurations and processing flow are the same as those in the above-described embodiment.

  The present invention can also be realized as an invention of a three-dimensional measurement image photographing method described in the flowcharts of the above-described embodiments, and as a program for causing a three-dimensional measurement image photographing apparatus to execute the method invention. is there. The program may be stored and used in a built-in storage unit of the three-dimensional measurement image capturing apparatus, may be stored and used in an external storage device, or may be downloaded from the Internet and used. Moreover, it is realizable also as a recording medium which recorded the said program.

  As described above, according to the present invention, when taking a picture while moving using a single camera, it is possible to grasp the progress of the picture taking, and to perform the picture taking for efficient and efficient photography. Equipment can be provided.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it is obvious that various modifications can be made to the embodiments without departing from the spirit of the present invention. .

  For example, in the above-described embodiments, the shooting range image is formed for each shooting or for each predetermined number of shots, the shooting range image is formed by dividing the shooting range image, or the feature points of the shot image are converted into coordinates. In the case of dividing into shooting areas, the example of displaying the shooting areas in different colors or displaying the shot images by pasting them has been described. However, these options can be freely selected. Also, it is possible to freely select options such as using a mark having an identification code that can identify each other as a feature point, determining a planned shooting position, performing a guide, and using a live image when determining a planned shooting position. . In addition, the example of using the imaging range image related to the simulation target object using the imaging frequency of the imaging area, using the scheduled imaging position number, or the imaging range image for the imaging insufficient area has been described. You may make it complement using the above. In addition, the size of the shooting area can be freely changed by changing the combination of marks surrounding it. For example, when it is desired to unify the number of times of photographing to three, the photographed image is selected so as to be about 1/3 of the whole photographed image, and the photographed image is photographed while moving so as to change by about 1/3. Further, if shooting is performed at a pair of positions shifted by one step at an arbitrary shooting position, an image close to a stereo image can always be obtained, and it is easy to use for orientation and three-dimensional measurement. In addition to the texture image and the wire frame image, the shooting range image may be represented by a bird's eye view or the like. Moreover, although the example which uses a perspective view as 3D coordinate space was demonstrated, you may use a projection view. In the above embodiment, an example in which the determination of the presence / absence of an undershooting area is automatically performed by the undershooting area determination unit 86 has been described. In addition, the predetermined number of shots, the number of feature points and marks used, the number of planned shooting positions, and the like can be changed as appropriate.

  The present invention can be used for three-dimensional measurement or the like that advances photographing while grasping the progress of photographing.

1, 1A to 1E 3D measurement image capturing device 2 Measurement object 3 Imaging unit 4 Display unit 4A Input unit 5 Storage unit 6 Feature extraction unit 7 3D position measurement unit 8 Image processing unit 9 Guide information processing unit 10 Control unit 11 Live Image 12 Sample Image 13 Shooting Position Image 14 Shooting Object Installation Area 15 Personal Computer (PC)
31 Shooting Count Determination Unit 41 Scheduled Shooting Position Input Unit 51 Shooting Image Storage Unit 52 Feature Point Shooting Count Storage Unit 53 Area Shooting Count Storage Unit 54 Shooting Area Image Storage Unit 55 Feature Point Position Relationship Storage Unit 56 Sample Image Storage Unit 57 Scheduled Shooting Position storage unit 58 Live image storage unit 61 Feature point extraction unit 62 Mark extraction unit 63 Code identification unit 71 Corresponding point search unit 72 Orientation unit 73 Three-dimensional position calculation unit 81 Shooting region segmentation unit 82 Feature point shooting number counting unit 83 Region shooting Number counting unit 84 Measuring object boundary extracting unit 85 Shooting range image forming unit 86 Shooting insufficient region determining unit 87 Shooting insufficient information notifying unit 88 Image coordinate converting unit 91 Moving information calculating unit 92 Next shooting scheduled position selecting unit 93 Moving information notifying unit 200 Retro target 204 Inner circle part 206 Outer circle part A1-A8, A10-A13 Color code target CT Color code De target P1 retro-target section P2 reference color section P3 color code section P4 white portion Q1~Q5 imaging area T1 to T8, T10, T11 retro-targets To threshold

Claims (15)

In a three-dimensional measurement image capturing apparatus that sequentially captures an object to be measured while overlapping with a single camera;
A photographing unit in which a photographer photographs the measurement object from a plurality of photographing positions to obtain a photographed image;
A feature point extraction unit for extracting feature points of the measurement object in the captured image acquired by the imaging unit;
An imaging region classifying unit that classifies an area of the measurement object of the captured image into an imaging region composed of an area (including the feature point) surrounded by three or more feature points extracted by the feature point extraction unit;
A three-dimensional position measurement unit for obtaining three-dimensional coordinates of the feature points from screen positions in a plurality of captured images of the feature points extracted by the feature point extraction unit;
Each feature point for which the three-dimensional coordinate is obtained by the three-dimensional position measurement unit is written in the coordinate space of the photographing range image that represents the photographing range of the measurement object in the three-dimensional space, and is written in the coordinate space of the photographing range image. A shooting range image forming unit that connects three or more feature points surrounding the shooting region among the feature points thus obtained, and divides the shooting point image for each shooting region, and forms the shooting range image as a wire frame image;
An undershooting area determination unit that determines whether or not there is an undershooting area in the shooting range image;
A display unit for displaying a shooting range image formed by the shooting range image forming unit;
An image capturing device for three-dimensional measurement. A shooting area image storage unit for storing the shot images by dividing them into the shooting areas;
The shooting range image forming unit includes, in a shooting region divided into the shooting range images, images of the shooting regions stored in the shooting region image storage unit, and positions of three or more feature points surrounding the shooting region. Fit to match and form the image area image as a texture image;
The three-dimensional measurement image capturing apparatus according to claim 1. Each time the imaging unit acquires a new captured image, the imaging range image forming unit forms a new imaging range image, and the display unit displays the newly formed imaging range image;
The image photographing device for three-dimensional measurement according to claim 1 or 2. After the imaging unit acquires a predetermined number of new captured images, the imaging range image forming unit forms a new imaging range image, and the display unit displays the newly formed imaging range image;
The image photographing device for three-dimensional measurement according to claim 1 or 2. An area photographing number counting unit for counting the number of photographing included in different photographed images for each of the photographing regions;
The undershooting area determination unit determines whether or not the shooting range image includes an undershooting area based on the number of shootings counted by the area shooting number counting unit;
The three-dimensional measurement image photographing device according to any one of claims 1 to 4. The three-dimensional position measurement unit obtains three-dimensional coordinates of the photographing position from screen positions in a plurality of photographed images of the extracted feature points;
A planned shooting position storage unit that stores a plurality of predetermined shooting planned positions;
The next shooting for selecting the next shooting scheduled position from the plurality of shooting planned positions by comparing the shooting position obtained by the three-dimensional position measuring unit with a plurality of shooting scheduled positions stored in the shooting planned position storage unit. A planned position selection unit;
A movement information calculation unit for obtaining a relative position of the photographing position with respect to the next photographing scheduled position;
A movement information notifying unit for creating guide information so that a photographer moves from the shooting position to the next shooting scheduled position based on the relative position obtained by the movement information calculating unit, and for notifying the display unit of the guide information; ;
The three-dimensional measurement image photographing device according to any one of claims 1 to 4. The photographing unit acquires a live image by photographing a measurement object at a current position by a moving photographer;
The three-dimensional position measurement unit obtains the three-dimensional coordinates of the current position when the live image is acquired, and the next photographing scheduled position selection unit obtains the position coordinates by the three-dimensional position measurement unit. Comparing the current position with a plurality of planned shooting positions stored in the planned shooting position storage unit, and selecting a planned next shooting position from the plurality of planned shooting positions;
The three-dimensional measurement image photographing device according to claim 6. The scheduled shooting position storage unit stores the shooting position for which the position coordinates have been obtained by the three-dimensional position measurement unit as an already shot position;
The next scheduled shooting position selection unit selects a scheduled next shooting position from the plurality of scheduled shooting positions by excluding the existing shooting position stored in the planned shooting position storage unit;
The three-dimensional measurement image photographing device according to claim 6 or 7. The undershooting area determination unit determines that there is no undershooting area when all the planned shooting positions are stored as the already-captured positions in the planned shooting position storage unit;
The three-dimensional measurement image capturing device according to claim 8. A sample image storage unit for storing an imaging range image related to the simulation target that simulates the measurement target;
The undershooting area determination unit determines whether or not there is an undershooting area by comparing the shooting range image related to the measurement object with the shooting range image related to the simulated object;
The three-dimensional measurement image photographing device according to any one of claims 1 to 4. An insufficient photographing information notifying unit for displaying the determination result of the insufficient photographing region determining unit on the display unit;
The image photographing device for three-dimensional measurement according to claim 5, claim 9 or claim 10. The feature point includes a mark having an identification code that can identify itself and others;
A code identification unit for identifying an identification code of the mark extracted by the feature point extraction unit;
The three-dimensional measurement image photographing device according to claim 1. The imaging range image is any one of a three-dimensional model image, an ortho image of the measurement object, or an all-around image in which the measurement object is projected continuously from the entire periphery toward the center point;
The three-dimensional measurement image photographing device according to claim 1. The first unit having the photographing unit and the display unit is configured to be movable independently of the second unit having the three-dimensional position measuring unit and the photographing range image forming unit, and the first unit and the second unit Is configured to be communicable by wire or wirelessly;
The three-dimensional measurement image photographing device according to claim 1. In a three-dimensional measurement image capturing method of sequentially capturing an object to be measured while overlapping with a single camera;
A photographing step in which a photographer photographs the measurement object from a plurality of photographing positions to obtain a photographed image;
A feature point extraction step of extracting feature points of the measurement object in the captured image acquired in the imaging step;
A shooting region dividing step of dividing the region of the measurement object of the captured image into a shooting region composed of a region (including the feature point) surrounded by three or more feature points extracted in the feature point extraction step;
A three-dimensional position measurement step for obtaining three-dimensional coordinates of the feature points from screen positions in a plurality of captured images of the feature points extracted in the feature point extraction step;
Each feature point obtained in the three-dimensional position measurement step is written in the coordinate space of the shooting range image representing the shooting range of the measurement object in a three-dimensional space, and the feature point written in the coordinate space of the shooting range image A shooting range image forming step in which three or more feature points surrounding the shooting region are connected and partitioned for each shooting region, and the shooting range image is formed as a wire frame image;
An undershooting area determination step of determining whether or not there is an undershooting area in the shooting range image;
A display step of displaying the shooting range image formed in the shooting range image forming step;
An image capturing method for three-dimensional measurement.