JP2008157780A - Three-dimensional data creating apparatus, image acquiring device, three-dimensional data creating method, image acquisition method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional data creating apparatus and related techniques, which reduce the influences of noise and can precisely measure the three-dimensional position of an object. <P>SOLUTION: The three-dimensional data creating apparatus 1A comprises: stereo image photographic means (cameras 2A, 2B) for acquiring a stereo image, by photographing the object to be measured from different view points; a determining section 31 for determining whether a relative position between the object to be measured and the stereo image photographic means is varied in a period in which a plurality of stereo image sets are acquired; an integration processing section 32 for generating an integrated image for creating three-dimensional data by integrating the plurality of stereoscopic image sets for respective images corresponding to the view points, which are acquired in the period, in which the relative position is not varied; and a three-dimensional data calculating section 33 for calculating the three-dimensional data of the object to be measured, by using the integrated image for creating the three-dimensional data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a three-dimensional data generation apparatus for a measurement object and a technology related thereto.

  There is a technique for obtaining a three-dimensional position of an object based on the principle of triangulation using a stereo image (see Patent Document 1).

JP-A-8-5369

  However, noise is often present in the captured image, and the presence of such noise affects the measurement accuracy of the three-dimensional position. In particular, in searching for corresponding points in a stereo image, there is a high possibility that noise affects measurement accuracy.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a three-dimensional data generation apparatus that can reduce the influence of noise and can measure a three-dimensional position of a measurement object with high accuracy and a technology related thereto.

  In order to solve the above-mentioned problem, the invention of claim 1 is a three-dimensional data generation device that generates three-dimensional data of a measurement object, and is a stereo that captures a stereo image by photographing the measurement object from different viewpoints. Acquired during a period in which the relative position change does not exist, an image capturing unit, a determination unit that determines presence / absence of a relative position change between the measurement object and the stereo image capturing unit in an acquisition period of a plurality of sets of stereo images Integrating image generating means for integrating a plurality of sets of stereo images for each viewpoint image to generate an integrated image for generating three-dimensional data, and using the integrated image for generating three-dimensional data, three-dimensional data of the measurement object And a three-dimensional data calculating means for calculating.

  According to a second aspect of the present invention, in the three-dimensional data generation apparatus according to the first aspect of the invention, the determination unit determines whether or not there is a change in the relative position by comparing the plurality of sets of stereo images for each viewpoint image. It is characterized by.

  According to a third aspect of the present invention, in the three-dimensional data generation apparatus according to the first or second aspect of the invention, the plurality of sets of stereo images include stereo images that are sequentially photographed at predetermined time intervals. To do.

  According to a fourth aspect of the present invention, in the three-dimensional data generation device according to any one of the first to third aspects of the present invention, the determination unit is configured to perform the determination based on data in a setting area in the plurality of sets of stereo images. The presence or absence of a relative position change is determined.

  According to a fifth aspect of the present invention, in the three-dimensional data generation apparatus according to the fourth aspect of the invention, the setting area can be changed according to an operation input by an operator.

  According to a sixth aspect of the invention, in the three-dimensional data generation apparatus according to the fourth aspect of the invention, the setting area is determined based on color information of the plurality of sets of stereo images.

  According to a seventh aspect of the invention, in the three-dimensional data generation apparatus according to the fourth aspect of the invention, the setting area is determined based on lightness information of the plurality of sets of stereo images.

  According to an eighth aspect of the present invention, in the three-dimensional data generation apparatus according to any one of the first to seventh aspects, the input image storage means for storing an input image that is a newly acquired stereo image, and the input The image processing apparatus further includes integrated image storage means for storing an integrated image obtained by integrating the image and a past stereo image acquired before the input image for each viewpoint image.

  According to a ninth aspect of the present invention, in the three-dimensional data generation apparatus according to any one of the first to seventh aspects of the present invention, an input image storage for storing a stereo image acquired at the i-th time (where i is a natural number). And an integrated image storage means having a multi-stage frame memory from the first stage to the K-th stage (where K is a natural number), wherein the j-th stage of the multi-stage frame memory (where j is a natural number less than or equal to K) Stores an integrated image obtained by integrating a plurality of sets of stereo images acquired from the (i−j) th to the i-th time for each viewpoint image.

  According to a tenth aspect of the present invention, in the three-dimensional data generation apparatus according to the ninth aspect of the invention, the integral image storage means is configured such that the integral image stored in the j-th frame memory is an integral image for generating the three-dimensional data. Flag information indicating whether or not it is appropriate is set according to a determination result by the determination means.

  According to an eleventh aspect of the present invention, in the three-dimensional data generation apparatus according to any one of the eighth to tenth aspects, the integral image generation means is a plurality of integral images stored in the integral image storage means. Then, among the images determined to be appropriate as the integral image for generating the three-dimensional data, the image having the largest number of integrations is determined as the integrated image for generating the three-dimensional data.

  According to a twelfth aspect of the present invention, in the three-dimensional data generation apparatus according to any one of the eighth to tenth aspects of the present invention, the integral image generation means includes a plurality of integral images stored in the integral image storage means. Then, among the images determined to be appropriate as the integrated image for generating the three-dimensional data, an image generated by being integrated a predetermined number of times or more is determined as the integrated image for generating the three-dimensional data. .

  According to a thirteenth aspect of the present invention, in the three-dimensional data generation apparatus according to any one of the first to twelfth aspects, the three-dimensional data calculation means includes a plurality of the three corresponding to different parts of the measurement object. A plurality of partial data is generated based on the integral image for generating the dimensional data, and the three-dimensional data of the measurement object is generated by combining the plurality of partial data.

  According to a fourteenth aspect of the present invention, in the three-dimensional data generation apparatus according to the thirteenth aspect of the present invention, the three-dimensional data calculation means is based on an integral image in which the number of integrations satisfies a predetermined condition among the plurality of partial data. Texture information is given only to the partial data generated in this way.

  According to a fifteenth aspect of the present invention, in the three-dimensional data generation device according to any one of the first to fourteenth aspects, the three-dimensional data calculation means is configured to generate the three-dimensional data in which the number of integrations satisfies a predetermined condition. Only for the integral image, partial data which is partial three-dimensional data of the measurement object is generated.

  According to a sixteenth aspect of the present invention, in the three-dimensional data generating apparatus according to any one of the first to fifteenth aspects, the three-dimensional data calculating means sets a parameter in the corresponding point search processing of the integral image as the integral It is determined according to the number of times of image integration.

  The invention of claim 17 is an image acquisition device for acquiring image data for generating three-dimensional data relating to a measurement object, and a stereo image photographing means for photographing the measurement object from different viewpoints to obtain a stereo image. Determining means for determining presence / absence of a relative position change between the measurement object and the stereo image photographing means in an acquisition period of a plurality of sets of stereo images; and a plurality of sets of stereo acquired in a period in which the relative position change does not exist Integral image generating means for integrating an image for each viewpoint image to generate an integrated image, and output means for outputting the integrated image as image data for generating the three-dimensional data.

  The invention of claim 18 is a three-dimensional data generation method for generating three-dimensional data of a measurement object, the stereo image photographing means photographing the measurement object from different viewpoints, and obtaining a stereo image; Determining the presence or absence of a relative position change between the measurement object and the stereo image capturing means in an acquisition period of a plurality of sets of stereo images; and a plurality of sets of stereo images acquired in a period in which the relative position change does not exist. Integrating each viewpoint image to generate an integrated image for generating three-dimensional data, and calculating three-dimensional data of the measurement object using the integrated image for generating three-dimensional data. Features.

  The invention according to claim 19 is an image acquisition method for acquiring image data for generating three-dimensional data related to a measurement object, and acquires a stereo image by imaging the measurement object from different viewpoints by a stereo image capturing means. Determining a relative position change between the measurement object and the stereo image photographing means in an acquisition period of a plurality of sets of stereo images, and a plurality of sets acquired in a period in which the relative position change does not exist And outputting an integrated image obtained by integrating the stereo image for each viewpoint image as the image data for generating the three-dimensional data.

  The invention of claim 20 is a program for causing a computer to execute the three-dimensional data generation method according to the invention of claim 18.

  The invention of claim 21 is a program for causing a computer to execute the image acquisition method according to claim 19 of the invention.

  According to the invention described in claims 1 to 21, it is possible to reduce the influence of noise and measure the three-dimensional position of the measurement object with high accuracy.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. Configuration>
FIG. 1 is a schematic diagram illustrating a configuration of a three-dimensional data generation apparatus 1A according to an embodiment of the present invention, and FIG. 2 is an external front view of the three-dimensional data generation apparatus 1A. FIG. 3 is a side view showing the relationship between the three-dimensional data generation device 1A and a measurement object (here, a human head) at a certain photographing time point, and FIG. 4 is a top view in the same situation as FIG. FIG. FIG. 5 is a top view showing how the three-dimensional data generation apparatus 1A moves as needed. Further, FIG. 6 is a conceptual diagram for explaining an operation of generating an “integrated image” described later.

  As illustrated in FIG. 2, the three-dimensional data generation apparatus 1 </ b> A includes a main body 10 and a carriage unit 40 that supports and moves the main body 10. As shown in FIG. 1, the main unit 10 is configured as a computer system having an imaging unit 2 having a plurality of (here, two) cameras (for example, color CCD cameras) 2A and 2B, a CPU, and the like. And a control unit 30. Various processing units are functionally realized by executing a predetermined program in the computer system.

  The two cameras 2A and 2B are arranged apart from each other by the base line length L (see FIG. 2), and these cameras 2A and 2B are respectively taken images GA, GB is acquired (see FIG. 6). The set SG (GA, GB) of the captured images is acquired as constituting a stereo image. In this embodiment, the imaging unit 2 sequentially acquires such stereo images at a minute time interval Δt (for example, 1/30 seconds). Thereby, a plurality of sets of stereo images SGi (GAi, GBi) are acquired in time series. Here, the two cameras 2A and 2B are arranged apart from each other in the vertical direction, but the present invention is not limited to this. For example, the two cameras 2A and 2B may be arranged apart from each other in the horizontal direction.

  The control unit 30 includes a determination unit 31, an integration processing unit 32, a three-dimensional data calculation unit 33, and a storage unit 34. The control unit 30 generates three-dimensional data of the measurement object using a stereo image based on the principle of triangulation. This three-dimensional data is configured to include, for example, position information of a plurality of feature points.

  However, in this embodiment, three-dimensional data is not generated from each set of stereo images acquired by the imaging unit 2, but “integrated images (IZ, FIG. 6) ”is used to generate three-dimensional data. The integrated image (specifically, a set of integrated images) IZ is an image IA, IB (set) generated by adding (integrating) a plurality of sets of stereo images SGi (GAi, GBi) for each viewpoint. In other words, the integral image IZ is generated by adding (integrating) a plurality of sets of stereo images SG for each of the images (also referred to as viewpoint images) GA and GB obtained from the viewpoints of the plurality of cameras 2A and 2B. It is also expressed as an image IZ (IA, IB) (a set of stereo images). The “integrated image” is not limited to an image obtained only by adding the pixel values for each corresponding pixel of a plurality of integration target images, and the pixel value is added for each corresponding pixel of a plurality of integration target images. Images obtained by averaging (including weighted average) are also included.

  Furthermore, in this embodiment, measurement is performed using an integral image satisfying the conditions described later (specifically, an integral image related to substantially the same subject image) among the integral images IZ thus obtained. Calculate three-dimensional data of the object. According to this, it is possible to reduce the influence of noise and acquire the three-dimensional data of the measurement object with high accuracy.

  The integration processing unit 32 integrates a plurality of sets of stereo images for each viewpoint image to generate an “integrated image”. The integration processing unit 32 uses the multistage frame memory in the storage unit 34 to generate a plurality of sets of integration images that are integration images of different periods.

  The determination unit 31 determines whether or not there is a change in the relative position between the measurement object (for example, the head of a person) and the imaging unit 2 in a specific period.

  Here, the presence or absence of relative position change between the two sets of stereo images is compared for each viewpoint (in other words, the images (viewpoint images obtained from each viewpoint). The case where the presence or absence of the relative position change of both is determined is compared. According to this, it is not necessary to provide a sensor for detecting a relative position change separately from the image system.

  Specifically, as shown in FIG. 7, an input image Gin (for example, GAi) newly input at a certain input time point is compared with a past integrated image IZ (for example, IA) until just before the input time point. Then, the presence or absence of a relative position change between the measurement object HD and the imaging unit 2 (for example, camera 2A) is determined. Specifically, an input image (for example, GAi) and a past integrated image (for example, IA) are compared for each pixel, and the number of pixels whose difference amount for each pixel is equal to or greater than a predetermined value is measured. The substantial identity of the subject images can be determined based on whether or not it is below a predetermined threshold. However, the present invention is not limited to this. For example, as shown in FIG. 13, the movement amount of the pixel block BK composed of a plurality of pixels as used in MPEG or the like (the pixel block BK between the two images IA and GAi). And the relative position between the measurement object HD and the imaging unit 2 is not changed on the condition that the number of pixel blocks for which the movement amount is equal to or greater than a predetermined amount is equal to or less than a predetermined threshold. You may make it determine. 7 and 13, only one viewpoint image GA is shown, but the same applies to the other viewpoint image GB.

  The three-dimensional data calculation unit 33 determines that an “integrated image” (“integration for generating three-dimensional data”) is determined by the determination unit 31 from the generated “integrated image”. 3D data of the measurement object is calculated using the “image”. Here, it is determined that the “integrated image” acquired in a period in which there is no relative position change is a photograph of substantially the same subject image.

  In this embodiment, partial three-dimensional data (also referred to as “partial data”) of the measurement target is generated by the three-dimensional data generation process based on a specific “integrated image”. This “partial data” is three-dimensional data relating to a portion that is visible from a specific direction in the entire measurement object. Then, a plurality of different partial data relating to the measurement target is generated by stereo images (particularly integrated images) obtained by imaging the measurement target from various directions, and further, the three-dimensional data of the measurement target is synthesized by combining the plurality of partial data. Data is generated.

  As illustrated in FIG. 5, the three-dimensional data generation apparatus 1 </ b> A (specifically, the imaging unit 2) includes a plurality of imaging positions P <b> 1, P <b> 2, P <b> 3, and P <b> 4 by moving the main body unit 10 using the carriage unit 40. The measurement object (specifically, the head of a person) HD is photographed from the position. Here, the photographing positions P1, P2, P3, and P4 are points on a circle where the measurement object HD is arranged at the center, and are positions that divide the circumference of the circle into four equal parts. .

  Specifically, the three-dimensional data generation device 1A is stationary for a predetermined period (about 1 second) and captures an image of the front side of the measurement object HD from the capturing position P1. Then, the three-dimensional data generation device 1A moves to the shooting position P2 and then rests for a predetermined period, and takes an image of the right side surface of the measurement object HD from the shooting position P2. After that, the three-dimensional data generation device 1A moves to the shooting position P3 and then rests for a predetermined period, and takes an image behind the measurement object HD from the shooting position P3. Further, after that, the three-dimensional data generation device 1A moves to the shooting position P4 and then rests for a predetermined period, and takes an image of the left side surface of the measurement object HD from the shooting position P4. In this way, the three-dimensional data generation device 1A images the measurement object viewed from different directions. Then, a plurality (four in this case) of partial data (three-dimensional data) is generated based on the photographing results from each direction. Furthermore, the entire data (three-dimensional data) of the measurement object is generated by combining the plurality of partial data. As described above, it is possible to improve the accuracy of each partial data by using the “integrated image” when acquiring each partial data, and it is possible to improve the accuracy of the whole data. .

  According to this, while the movement and stationary of the main body unit 10 using the carriage unit 40 are repeated, an integrated image based on the captured image during the stationary period is generated, and accurate three-dimensional data is generated based on the integrated image. Can be generated.

  Hereinafter, such an operation will be described in detail.

<2. Operation>
<2-1. Overview>
FIG. 8 is a flowchart showing the operation of the three-dimensional data generation apparatus 1A. Here, as shown in FIGS. 9 and 10, a case where both the base image GA and the reference image GB acquired in time series are stored in a multi-stage frame memory (also referred to as multi-stage memory) is illustrated. The multistage frame memory for the reference image GA and the multistage frame memory for the reference image GB are each configured in the storage unit 34.

  In a period in which it is determined that substantially the same subject image is continuously captured, the “integrated image” is updated at any time in steps S11 to S15. In this embodiment, a plurality of integrated images for different periods are acquired using a multistage frame memory.

  On the other hand, if it is determined that a substantially different subject image has been taken, the integration for generating the three-dimensional data that satisfies a predetermined condition among a plurality of integration images stored in the multistage frame memory up to that time is determined. The image is determined and output (steps S16 to S17). As this condition, for example, “the one determined to be suitable as an integral image for generating three-dimensional data, which has the highest number of integrations and has been integrated more than a predetermined number of times” is adopted. can do. As a result, an integrated image acquired during a period in which it is determined that substantially the same subject image is continuously captured can be acquired as an integrated image for generating three-dimensional data. Thereafter, when the end of the photographing operation is determined in step S18, the process proceeds to step S19, and the three-dimensional data of the measurement object HD is calculated using the integrated image output in step S17.

  According to this, for example, each integrated image in the still period at the photographing positions P1, P2, P3, P4 is stored in the three-dimensional data generation device 1A, and each partial data is generated based on each integrated image, Overall data is generated by integrating the partial data.

<2-2. Details of integral image generation operation>
Next, details of the operations in steps S11 to S17 will be described. Here, the description will focus on the operation of generating an integral image related to the reference image GA, but the operation of generating an integrated image related to the reference image GB is performed in the same manner.

  FIG. 11 is a diagram showing information stored in the multistage frame memory. In FIG. 11, the storage information of each frame memory Fj (j = 0, 1,..., K) at time T (i−1) and the storage information of each frame memory Fj at time T (i). It is shown. In addition, “◯” and “x” attached to the upper right of each frame memory Fj indicate the contents of the identity flag FLj. This flag FLj is flag information indicating whether or not the integral image stored in the j-th frame memory Fj is appropriate as an integral image for generating three-dimensional data.

  As shown in FIG. 11, in this multi-stage frame memory, the (0th) 0th frame memory F0 is used as a storage area for the latest input image (in other words, an input image storage unit). The K-stage multi-stage frame memory from the (first stage) first frame memory F1 to the (K-th stage) K-frame memory FK stores past integral images over a predetermined period of time. It is an area to be done. Specifically, the first frame memory F1 is an area in which an integrated image of the latest input image and the previous input image is stored. Further, the second frame memory F2 stores an integrated image of the latest input image, the previous input image, and the previous input image, in other words, from the latest input image. This is an area in which the integral image of the image up to the previous input image is stored. The third frame memory F3 is an area in which an integral image of images from the latest input image to the three previous input images is stored. As described above, the j-th frame memory Fj is an area in which an integral image of images from the latest input image to the j-th previous input image is stored.

  For example, at time T (i), when the latest input image G (i) is input and the predetermined update process is completed, information (data) in each column on the right side of FIG. 11 is stored in each frame memory Fj. become. That is, the latest input image G (i) is stored in the 0th frame memory F0, and integral images over different periods are stored in the first frame memory F1 to the Kth frame memory FK. Specifically, the first frame memory F1 stores an integral image of images from the latest input image G (i) to the previous input image G (i-1). The second frame memory F2 stores an integrated image of images from the latest input image G (i) to the previous input image G (i-2). Similarly, an integrated image of (j + 1) images from the latest input image G (i) to the j previous input image G (i-j) is stored in the jth frame memory Fj. Note that i is an integer (specifically, a natural number). In particular, when i is a natural number of 2 or more, an integral image is stored in any one of the frame memories Fj.

  Further, as described above, such a multistage frame memory is provided for each of the viewpoint images GA and GB. Therefore, the j-th stage (j is a natural number equal to or less than K) (that is, the frame memory Fj) of the multi-stage frame memory is a plurality of sets of stereo images acquired from the (i−j) -th to the i-th time for each viewpoint image. The integrated image integrated into is stored.

  Next, the update operation of the multistage frame memory will be described with reference to FIGS. Here, the update operation when a new input image G (i) is input in a state where each piece of information as shown in FIG. 11 is stored in each multistage frame memory Fj at time T (i−1). explain.

  As shown in FIG. 11, an image G (i-1) is input at time T (i-1), and G (i-) is stored in the frame memory Fj (j = 0, 1,..., K), respectively. 1), G (i-2) + G (i-1), G (i-3) + G (i-2) + G (i-1), G (i-4) + G (i-3) + G (i -2) Assume that + G (i-1), ... is stored.

  In this situation, when a new image G (i) is input (FIG. 8, step S11), first, the new object G (i) is compared with the integrated image, thereby measuring object HD and imaging unit 2. Whether or not there is a relative position change with each camera 2A and 2B (specifically, each camera 2A, 2B) is determined (step S12). Then, according to the result of the comparison process, it is determined whether or not there is a relative position change, in other words, whether or not the identity of the subject image is maintained (step S13). In steps S12 and S13, a comparison determination process is performed in which the latest input image is compared with the past integrated image and determination is performed using the comparison result. As the comparison determination process, for example, an integration image having a corresponding flag FL value of “1” (“◯”) and having the largest number of integrations is selected as a comparison target from among a plurality of integration images. It is possible to adopt a mode for determining the identity between the integrated image and the latest input image. However, the present invention is not limited to this. For example, a comparison determination process using a comparison result between the latest input image and the previous input image may be performed.

  As described above, in this comparison determination process, if it is determined that a substantially different subject image has been captured, the process proceeds to steps S16 to S18, and if the capture process is continued, the process proceeds to step S14. On the other hand, if it is determined that the identity of the subject image is maintained, the process directly proceeds to step S14 without executing steps S16 to S18.

  In step S14, a comparison process between the integrated image stored in the jth frame memory Fj and the new image G (i) is executed, and stored in each frame memory Fj according to the result of the comparison process. A determination process for determining whether or not the integral image is appropriate as an integral image for generating three-dimensional data is performed. Then, the identity flag FLj provided corresponding to each frame memory Fj according to the result of the determination processing (corresponding to the frame memory F (j + 1) one step ahead as described below) The value of the flag FL (j + 1)) to be set is set. As the value of the flag, for example, when it is determined that the identity is maintained (ie, there is no relative position change) (“◯”), “1” is stored, and the identity is not maintained (ie, When it is determined that there is a relative position change (“×”), “0 (zero)” is stored.

  At this time, the result of the comparison process between the integral image stored in the j-th frame memory Fj and the new image G (i) is one stage ahead, that is, the (j + 1) -th frame memory F (j + 1). ) Flag area (identity flag FL (j + 1)). By storing the data one step ahead, the process of updating the identity flag for each new integrated image after the update in the next step S15 is executed. Note that this comparison process does not necessarily have to be performed on all frame memories, and the frame memories (corresponding flag values) determined to be identical at the time of comparison among the plurality of frame memories Fj. Frame memory in which “1” is stored as a target). In this case, “0 (zero)” may be stored as a comparison processing result in a flag related to the frame memory that is not actually subjected to the comparison processing.

  Thereafter, in step S15, a process of updating a plurality of integral images stored in the multistage frame memory is performed. Specifically, each past integrated image stored in the frame memory Fj of the j-th stage (j = 0,..., (K−1)) is each one stage ahead, that is, the (j + 1) -th frame. A process of updating the integral image stored in each stage of the multistage frame memory by transferring to the memory F (j + 1) and further adding a new input image G (i) to each frame memory Fj. Is done. In other words, the integrated image stored in the (j-1) th frame memory F (j-1), specifically, the new input image is stored in the jth frame memory Fj. An image obtained by integrating G (i) and the integrated image of the image from the immediately preceding input image G (i-1) to the (j-1) previous input image G (i-j) is stored as a new image. Is done. Further, as described above, in step S14, the identity flag for each of the new integrated images updated in step S15 is updated in advance.

  For example, in the frame memory FK, an integrated image of the integrated image and the image G (i) stored in the frame memory F (K−1) until immediately before is newly stored. Further, as a new value of the flag FLK corresponding to the frame memory FK, there is a value corresponding to the comparison processing result between the integrated image and the image G (i) stored in the frame memory F (K-1) until immediately before. Stored.

  Similarly, an integrated image of the integrated image and the image G (i) stored in the frame memory F2 until immediately before is newly stored in the frame memory F3. Further, as a new value of the flag FL3 corresponding to the frame memory F3, a value corresponding to the comparison processing result between the integral image and the image G (i) stored in the frame memory F2 until immediately before is stored.

  Further, in the frame memory F2, an integrated image of the integrated image and the image G (i) stored in the frame memory F1 until immediately before is newly stored. In addition, as a new value of the flag FL2 corresponding to the frame memory F2, a value corresponding to the comparison processing result between the integrated image and the image G (i) stored in the frame memory F1 until immediately before is stored.

  Furthermore, the frame memory F1 newly stores an integral image of the image G (i-1) and the image G (i) that have been stored in the frame memory F0 until immediately before. Further, as a new value of the flag FL1 corresponding to the frame memory F1, there is a value corresponding to the comparison processing result between the image G (i-1) and the image G (i) stored in the frame memory F0 until immediately before. Stored.

  Also, the new image G (i) is stored as it is in the 0th stage frame memory F0.

  Here, the influence of the input image at each time is made equal by adopting a weighted average process as the addition process. Specifically, in the j-th frame memory Fj, the integral image related to (j + 1) images is stored. Therefore, the integral image stored in the frame memory Fj and the new input image G (i ) And weighted with a weight of (j + 1): 1, and stored in the frame memory Fj as a new integrated image.

  According to the operation as described above, for example, as shown in FIG. 12, when an image G31 is acquired at time T31 immediately after the end of movement, an integrated image of two images G30 and G31 relating to substantially the same subject image. Is stored in the frame memory F1. At this time, it is assumed that the integrated images stored in the frame memories F2 to FK are determined to be integrated images relating to different subject images.

  Thereafter, when the image G32 is further acquired at time T32, an integrated image of the two images G31 and G32 relating to substantially the same subject image is stored in the frame memory F1, and 3 relating to the substantially identical subject image. An integrated image of the two images G30, G31, G32 is stored in the frame memory F2.

  Similarly, when the image G33 is acquired at time T33, an integrated image of the two images G32 and G33 relating to substantially the same subject image is stored in the frame memory F1, and 3 relating to the substantially identical subject image. An integrated image of the two images G31, G32, G33 is stored in the frame memory F2. Further, an integrated image of four images G30, G31, G32, and G33 relating to substantially the same subject image is stored in the frame memory F3.

  As described above, as the stationary time of the main body 10 increases, an integrated image having a larger number of integration target images, in other words, an integrated image having a larger number of integrations is generated.

<2-3. Integral image selection operation and 3D data generation operation>
In step S16 described above, among the plurality of integral images stored in the multi-stage frame memory, the integral image having the corresponding flag FL “1” and the largest number of integrations is selected as an integral for generating three-dimensional data. Determine as an image. Then, the determined integrated image and the number of integrations are stored in a predetermined area in the storage unit 34 as an output result (step S17). Thereafter, in step S19, three-dimensional data of the measurement object is calculated using the integrated image determined in step S16. In step S19, preferably, the three-dimensional data including the texture information of the object surface is calculated together with the three-dimensional shape data.

  Further, in the above-described shooting operation end determination process in step S18, for example, depending on whether or not there is an operation instruction from the operator and / or whether or not a predetermined number of integral images have been acquired. It is possible to determine whether or not to end a series of shooting operations.

  If it is determined in step S18 that the shooting process is to be continued, the process returns to step S14 and the same operation is repeated.

  Then, by repeating the operation as described above, a plurality of three-dimensional data generation integrated images corresponding to different parts of the measurement object HD are generated, and the plurality of three-dimensional data generation integrated images are generated. Based on this, a plurality of partial data is generated. For example, the partial data is generated in the order of storage for the integrated image stored in step S17 (step S19).

  When a plurality of partial data are generated, a process of combining the plurality of partial data is also performed in step SP19. Thereby, three-dimensional data (specifically, overall data) of the measurement object is generated.

  In the three-dimensional data generation process in step S19, a process for calculating three-dimensional data (such as a three-dimensional position) is performed based on the integrated image obtained as a stereo image. In the corresponding point search process between stereo images executed as a part of this calculation process, a correlation calculation method called phase only correlation (POC) which is a kind of Fourier transform method is preferably used. Used. According to this, the corresponding point search between the stereo images can be performed with high accuracy, and highly accurate three-dimensional position data can be obtained.

  In the phase only correlation method, it is required to determine the window size WS of the window used for the Fourier transform. As the window size WS, for example, an appropriate fixed value determined by an appropriate adjustment operation can be adopted. However, here, a mode in which the window size WS is determined according to the number of integrations of the integrated image obtained as described above will be exemplified.

  In general, if the window size WS is reduced, the signal-to-noise ratio (SNR) is lowered and the error is likely to increase. In particular, when there is a lot of noise, the SN ratio is greatly reduced, and the error is likely to increase. On the other hand, increasing the window size WS decreases the resolution (in each direction (for example, the X direction and the Y direction)).

  Therefore, it is preferable to change the window size WS according to the amount of noise. Here, it is assumed that the amount of noise is estimated based on the number of integrations (= number of integration target images-1) at the time of generating an integral image, and the window size WS is changed according to the number of integrations. For example, when the number of integration target images is C1, the area of the window size WS in the POC may be reduced to 1 / √ (C1). Specifically, when the number of integrations is 3, that is, when the number of integration target images is 4, it is assumed that the noise level is reduced to 1/2, and the area of the POC window size WS is set to 1/2. According to this, it is possible to suppress a decrease in resolution and an increase in error.

  As described above, the parameter (specifically, the window size WS in the phase-only correlation method) for searching for corresponding points between certain integral images (stereo images) is set according to the number of integrations when the integral image is generated. It is possible to determine.

<3. Variations>
In the above embodiment, among the plurality of integral images stored in the multistage frame memory, the integral image having the corresponding flag FL “1” and having the largest number of integrations is determined as the integral image for generating the three-dimensional data. Although the case where it does is illustrated, it is not limited to this. For example, an integrated image that is generated when the corresponding flag FL is “1” and is integrated a predetermined number of times or more may be determined as an integrated image for generating three-dimensional data. More specifically, in response to the generation of an integral image in which the corresponding flag FL is “1” and the number of integrations is equal to or greater than the predetermined number TH, the integral image is converted into an integral image for generating three-dimensional data. Can be determined and output.

  Moreover, although the case where the partial data was produced | generated for the integral image stored by step S17 in the order of storage (step S19) was illustrated in the said embodiment, it is not limited to this. For example, partial data may be generated in order from the integrated image having the largest number of integrations. According to this, three-dimensional data can be generated efficiently.

  Moreover, in the said embodiment, although the identity etc. of a some image are determined using the whole area | region of an image, it is not limited to this. For example, the presence or absence of a relative position change between the measurement object HD and the imaging unit 2 using data of a partial area (setting area) of the entire area, in other words, substantially the same subject image in a plurality of images. You may make it determine sex etc. For example, a partial area at the center of the image can be fixedly set as a setting area (also referred to as a designated area).

  Further, such a setting area may be set (changeable) in accordance with an operation input by the operator. For example, the size and position of the setting area may be set using the operation screen and operation input unit (not shown) of the three-dimensional data generation apparatus 1A.

  Alternatively, the setting area may be determined based on color information of a plurality of sets of stereo images. Specifically, for example, a region excluding a blue region (in other words, a region in which a color region other than blue is designated) may be automatically determined as the setting region. According to this, it is possible to detect whether or not an object having a color other than blue moves when shooting with a blue background.

  Alternatively, the setting area may be determined based on lightness information of a plurality of sets of stereo images. Specifically, for example, a region excluding a black region (in other words, a region in which a bright region other than black is designated) may be automatically determined as the setting region. This makes it possible to detect the presence or absence of movement of a relatively bright object other than black during shooting with a black background.

  In addition, the three-dimensional data generation device 1A is annularly laid on the floor so that the three-dimensional data generation device 1A can be easily moved to the photographing positions P1, P2, P3, and P4 in the above embodiment. You may make it drive | work on a rail and stop when it drive | worked the predetermined distance (1/4 round).

  In addition, the shooting positions P1, P2, P3, and P4 in the above embodiment may be positions determined in advance, but are not limited thereto, and may be arbitrarily determined by the photographer.

  For example, as shown in FIG. 14, the photographer HT operates the carriage unit 40 to move the three-dimensional data generation device 1A to a desired position, sets a desired shooting direction and a desired shooting distance, and sets a measurement target. You may make it photograph a thing. According to this, data of a specific part (desired part) of the measurement object can be preferentially photographed according to the will of the photographer. In particular, since the operator can acquire data of a desired part by moving the main body 10 while moving the main body 10 while taking a moving image, and stationary the main body 10 at an arbitrary position, it is very convenient.

  In this case, the position and orientation after movement of the three-dimensional data generation device 1A are calculated based on a position / orientation detection sensor 41 (movement amount detection sensor, attitude detection sensor, etc.) attached to the carriage unit 40. It is preferable. If this calculation result is used, the position and orientation after the movement of the three-dimensional data generation apparatus 1A are accurately calculated, and the captured image is more accurately specified as to which part of the measurement object is captured. It is possible.

  In this case, when a plurality of partial data is generated based on different integrated images acquired from different shooting positions, a process for matching the position and orientation between the partial data is performed as necessary. Is preferred. As such processing, for example, the common feature points in each partial data are matched, and the degree of coincidence of the normals of the common plane (patch) including the common feature points in each partial data becomes a predetermined level or more. As described above, the process of changing the position and orientation of each partial data can be performed.

  Moreover, in the said embodiment, the case where the measurement object HD and the imaging part 2 are moved relatively by moving the three-dimensional data generation apparatus 1A, and the measurement object HD is imaged from various angles is exemplified. However, it is not limited to this. For example, conversely, by moving the measurement object HD, the measurement object HD and the imaging unit 2 may be relatively moved, and the measurement object HD may be photographed from various angles. More specifically, the measurement object HD is photographed from various angles by repeating the operation of placing the measurement object on the turntable and rotating the turntable by a predetermined angle and then stopping the measurement object for a predetermined period. It may be.

  Moreover, in the said embodiment, although the case where an image acquisition process (process of step S11-S18) and the production | generation process (step S19) of three-dimensional data (step S19) were performed in series (sequential) was illustrated, it is limited to this. Instead, they may be executed in parallel.

  Moreover, in the said embodiment, although the case where the production | generation of 3D data was performed in 1A of 3D data generation devices was illustrated, it is not limited to this. For example, after executing the processing (for example, the processing in steps S11 to S18) until the generation of the integral image for generating the three-dimensional data as described above with an apparatus (also referred to as an image acquisition apparatus) having the same configuration as described above, Once the image data for generating the three-dimensional data is output. Then, the generated image data for generating three-dimensional data may be used to execute a three-dimensional data generation process (step S19) with a separate device.

  Moreover, in the said embodiment, although the case where partial data was produced | generated about all the integral images acquired by the image acquisition process (process of step S11-S18) was illustrated, it is not limited to this.

  For example, partial data may be generated only for an integral image that satisfies a predetermined condition. Specifically, partial data may be generated only for an integrated image (integrated image for generating three-dimensional data) corresponding to the number of integrations of a relatively high threshold value TH1 or more (for example, TH1 = 10).

  Further, the partial data may be created stepwise according to the number of integrations. Specifically, first, partial data is generated only for an integrated image whose integration count is equal to or greater than the first threshold TH1, and then the integration count is smaller than the first threshold TH1 and equal to or greater than the second threshold. Partial data may be generated only for the integral image. According to this, three-dimensional data can be generated efficiently.

  Moreover, although the case where texture information is also added to all the partial data calculated in step S19 has been illustrated in the above embodiment, the present invention is not limited to this. For example, texture information may be added only to partial data that satisfies a predetermined condition. Specifically, texture information may be added only to partial data generated using an integrated image corresponding to the number of integrations of a relatively high threshold value TH1 or more.

  In the above embodiment, a case where a stereo image (binocular stereo image) based on two cameras is illustrated is exemplified, but the present invention is not limited to this, and a stereo image based on three or more cameras ( A stereo image obtained by three or more eyes may be acquired. In this case as well, the above-described integral image may be generated for each camera viewpoint image (viewpoint image).

It is the schematic which shows the structure of a three-dimensional data generation apparatus. It is an external appearance front view of a three-dimensional data generation device. It is a side view which shows the relationship between the three-dimensional data generation apparatus and measurement object in a certain imaging | photography time. It is a top view in the situation of FIG. It is a top view which shows a mode that a three-dimensional data generation apparatus moves at any time. FIG. 5 is a conceptual diagram illustrating an operation of generating “integrated image”. It is a figure explaining the comparison process of a new input image and the past integral image. It is a flowchart which shows operation | movement of a three-dimensional data generation apparatus. It is a figure which shows the multistage frame memory regarding a reference | standard image. It is a figure which shows the multistage frame memory regarding a reference image. It is a figure which shows the storage information of a multistage frame memory. It is a figure which shows a mode that a some integrated image is produced | generated sequentially. It is a figure which shows the movement amount of a block. It is a figure which shows a mode that a photographer moves a three-dimensional data generation apparatus to arbitrary imaging positions.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1A 3D data generator 2 Image pick-up part 2A, 2B Camera 10 Main part 30 Control part 40 Dolly part F, Fj Frame memory FL, FLj Flag GA Reference image GB Reference image HD Measurement object IZ Integration image L Base length P1, P2 , P3, P4 Shooting position

Claims (21)

A three-dimensional data generation device that generates three-dimensional data of a measurement object,
Stereo image photographing means for photographing the measurement object from different viewpoints to obtain a stereo image;
Determination means for determining the presence or absence of a relative position change between the measurement object and the stereo image photographing means in an acquisition period of a plurality of sets of stereo images;
Integrated image generation means for integrating a plurality of sets of stereo images acquired during a period in which there is no relative position change for each viewpoint image to generate an integrated image for generating three-dimensional data;
3D data calculating means for calculating 3D data of the measurement object using the integrated image for generating the 3D data;
A three-dimensional data generation device comprising: The three-dimensional data generation device according to claim 1,
The determination unit compares the plurality of sets of stereo images for each viewpoint image to determine the presence or absence of the relative position change. In the three-dimensional data generation device according to claim 1 or 2,
The three-dimensional data generation apparatus, wherein the plurality of sets of stereo images include stereo images that are sequentially photographed at predetermined time intervals. The three-dimensional data generation device according to any one of claims 1 to 3,
The three-dimensional data generation apparatus, wherein the determination unit determines whether or not there is a change in the relative position based on data in a setting area in the plurality of sets of stereo images. The three-dimensional data generation device according to claim 4,
The three-dimensional data generation apparatus, wherein the setting area can be changed according to an operation input by an operator. The three-dimensional data generation device according to claim 4,
The three-dimensional data generation apparatus, wherein the setting area is determined based on color information of the plurality of sets of stereo images. The three-dimensional data generation device according to claim 4,
The three-dimensional data generation device, wherein the setting area is determined based on lightness information of the plurality of sets of stereo images. In the three-dimensional data generation device according to any one of claims 1 to 7,
Input image storage means for storing an input image which is a newly acquired stereo image;
Integrated image storage means for storing an integrated image obtained by integrating the input image and a past stereo image acquired before the input image for each viewpoint image;
The three-dimensional data generation device further comprising: In the three-dimensional data generation device according to any one of claims 1 to 7,
Input image storage means for storing a stereo image acquired at the i-th time (where i is a natural number);
Integrated image storage means having a multi-stage frame memory from the first stage to the K-th stage (where K is a natural number);
With
The j-th stage (where j is a natural number equal to or less than K) of the multi-stage frame memory is an integrated image obtained by integrating a plurality of sets of stereo images acquired from the (i−j) -th to the i-th time for each viewpoint image. A three-dimensional data generation apparatus characterized by storing the data. The three-dimensional data generation device according to claim 9,
The integral image storage means displays flag information indicating whether or not the integral image stored in the j-th frame memory is appropriate as the integral image for generating the three-dimensional data according to the determination result by the determination means. A three-dimensional data generation device characterized in that the setting is performed. The three-dimensional data generation device according to any one of claims 8 to 10,
The integral image generation means is the one having the largest number of integrations among the plurality of integral images stored in the integral image storage means and determined to be appropriate as the integral image for generating the three-dimensional data. A three-dimensional data generation apparatus, wherein the three-dimensional data generation apparatus determines the integral image for generating the three-dimensional data. The three-dimensional data generation device according to any one of claims 8 to 10,
The integral image generation means is integrated and generated a predetermined number of times or more among the plurality of integral images stored in the integral image storage means that are determined to be appropriate as the integral image for generating the three-dimensional data. A three-dimensional data generation apparatus characterized by determining an integrated image for generating the three-dimensional data. The three-dimensional data generation device according to any one of claims 1 to 12,
The three-dimensional data calculating means generates a plurality of partial data based on a plurality of integrated images for generating the three-dimensional data corresponding to different parts of the measurement object, and further synthesizes the plurality of partial data. A three-dimensional data generation device, characterized in that three-dimensional data of the measurement object is generated by the above. The three-dimensional data generation device according to claim 13,
The three-dimensional data calculating means adds texture information only to partial data generated based on an integral image in which the number of integrations satisfies a predetermined condition among the plurality of partial data. Data generator. The three-dimensional data generation device according to any one of claims 1 to 14,
The three-dimensional data calculation means generates partial data that is partial three-dimensional data of the measurement object only with respect to the integration image for generating the three-dimensional data, the number of integrations satisfying a predetermined condition. A three-dimensional data generation apparatus. The three-dimensional data generation device according to any one of claims 1 to 15,
The three-dimensional data generation device, wherein the three-dimensional data calculation means determines a parameter in the corresponding image search processing for the integrated image according to the number of integrations of the integrated image. An image acquisition device for acquiring image data for generating three-dimensional data related to a measurement object,
Stereo image photographing means for photographing the measurement object from different viewpoints to obtain a stereo image;
Determination means for determining the presence or absence of a relative position change between the measurement object and the stereo image photographing means in an acquisition period of a plurality of sets of stereo images;
Integrated image generation means for generating an integrated image by integrating a plurality of sets of stereo images acquired during a period in which there is no relative position change for each viewpoint image;
Output means for outputting the integrated image as image data for generating the three-dimensional data;
An image acquisition apparatus comprising: A 3D data generation method for generating 3D data of a measurement object,
Photographing the measurement object from different viewpoints by a stereo image photographing means to obtain a stereo image; and
Determining the presence or absence of a relative position change between the measurement object and the stereo image photographing means in an acquisition period of a plurality of sets of stereo images;
Integrating a plurality of sets of stereo images acquired during a period in which there is no relative position change for each viewpoint image to generate an integrated image for generating three-dimensional data;
Calculating three-dimensional data of the measurement object using the integrated image for generating the three-dimensional data;
A three-dimensional data generation method comprising: An image acquisition method for acquiring image data for generating three-dimensional data related to a measurement object,
Photographing the measurement object from different viewpoints by a stereo image photographing means to obtain a stereo image; and
Determining the presence or absence of a relative position change between the measurement object and the stereo image photographing means in an acquisition period of a plurality of sets of stereo images;
Outputting an integrated image obtained by integrating a plurality of sets of stereo images acquired during a period in which the relative position change does not exist for each viewpoint image as image data for generating the three-dimensional data;
An image acquisition method comprising:   A program for causing a computer to execute the three-dimensional data generation method according to claim 18.   A program for causing a computer to execute the image acquisition method according to claim 19.

Images