JP2009168536A - Three-dimensional shape measuring device and method, three-dimensional shape regenerating device and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a display speed and precision of a three-dimensional shape compatible when generating three-dimensional data. <P>SOLUTION: A three-dimensional data generating part 30 generates the three-dimensional data from image data obtained by imaging an object with imaging parts 21A, 21B. Then, first and second three-dimensional data different in precision are generated. For an example, the first three-dimensional data is the tree-dimensional data for display, and the second three-dimensional data is the three-dimensional data for recording. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a three-dimensional shape measuring apparatus and method for generating three-dimensional data representing a three-dimensional shape of a subject from image data acquired by photographing, and a three-dimensional shape reproducing apparatus and method for reproducing the generated three-dimensional data. , And a program for causing a computer to execute a three-dimensional shape measurement method and a three-dimensional shape reproduction method.

  Corresponding points that are pixels corresponding to a plurality of images (a reference image by a reference camera and a reference image by a reference camera) obtained by capturing an image of a subject using two or more cameras provided at different positions (Stereo matching) and applying the triangulation principle to the position difference (parallax) between the corresponding pixel on the reference image and the pixel on the reference image, the pixel from the reference camera or reference camera A method has been proposed in which the distance to a point on the subject corresponding to is measured to generate three-dimensional data representing the three-dimensional shape of the subject (see Patent Document 1).

In such a method, a three-dimensional shape of a subject can be projected on a virtual two-dimensional plane using three-dimensional data, and can be represented as a three-dimensional image such as a mesh model or a volume model. Since the virtual viewpoint of the three-dimensional shape can be specified by rotating and moving the three-dimensional shape or changing the zoom magnification, it becomes easy to recognize the three-dimensional shape of the subject.
JP 2005-17262 A

  By the way, when generating three-dimensional data representing a three-dimensional shape, it is required to have high measurement accuracy and resolution. On the other hand, when displaying a three-dimensional shape, it is required to display at high speed. However, in the method described in Patent Document 1, since the same three-dimensional data is used for recording and for display, the display speed is reduced for recording or the measurement accuracy for display is reduced. There was no choice but to generate 3D data.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to make it possible to achieve both display speed and accuracy of a three-dimensional shape when generating three-dimensional data.

A three-dimensional shape measuring apparatus according to the present invention includes an image data acquisition unit that acquires a plurality of image data respectively representing a plurality of images acquired by photographing a subject from different positions;
3D data generating means for generating 3D data representing the 3D shape of the subject from the plurality of image data, wherein the 3D data generating means generates first and second 3D data having different accuracy. It is characterized by comprising.

  “Different accuracy” means that the accuracy of the calculated three-dimensional shape is different. Specifically, the measurement accuracy of the three-dimensional shape, the resolution of the three-dimensional shape, and the calculation for generating the three-dimensional shape It means that at least one of the methods or the like is different. For this reason, the first three-dimensional data with lower accuracy can be generated at a higher speed than the second three-dimensional data with higher accuracy.

In the three-dimensional shape measuring apparatus according to the present invention, the apparatus further includes calculation method determining means for determining a calculation method for generating the first three-dimensional data.
The three-dimensional data generation means may be means for generating the first three-dimensional data by the determined calculation method.

  Further, in the three-dimensional shape measuring apparatus according to the present invention, the calculation method determining means includes the resolution of the display means for displaying the three-dimensional shape, the display speed of the three-dimensional shape of the display means, and the output format of the imaging means. The calculation method may be determined based on at least one of the following.

In the three-dimensional shape measuring apparatus according to the present invention, when displaying the three-dimensional shape of the subject represented by the first three-dimensional data on the display means, the parameter for accepting designation of the display parameter of the three-dimensional shape. Designation means;
Display control means for displaying the three-dimensional shape represented by the first three-dimensional data on the display means according to the designated display parameter may be further provided.

In the three-dimensional shape measuring apparatus according to the present invention, when displaying the three-dimensional shape of the subject represented by the first three-dimensional data on the display means, the parameter for accepting designation of the display parameter of the three-dimensional shape. Designation means;
A calculation method determining means for determining a calculation method of the first three-dimensional data according to the display parameter;
The three-dimensional data generation means may be means for generating the first three-dimensional data by the determined calculation method.

  Further, in the three-dimensional shape measuring apparatus according to the present invention, the three-dimensional data generation means sequentially generates a plurality of three-dimensional data having different accuracy in stages from a low accuracy, and generates the three-dimensional data in the middle of the generation. It is good also as a means to output as said 1st three-dimensional data.

  In the three-dimensional shape measuring apparatus according to the present invention, the three-dimensional data generation means includes a first means for generating the first three-dimensional data and a second means for generating the second three-dimensional data. It is good also as what comprises a means.

A three-dimensional shape reproduction apparatus according to the present invention comprises: the second three-dimensional data acquired by the three-dimensional shape measurement apparatus that designates display parameters according to the present invention; and data acquisition means for acquiring the display parameters;
And display means for displaying a three-dimensional shape represented by the second three-dimensional data based on the display parameter.

The three-dimensional shape measurement method according to the present invention acquires a plurality of image data respectively representing a plurality of images acquired by photographing a subject from different positions,
When generating three-dimensional data representing the three-dimensional shape of the subject from the plurality of image data, first and second three-dimensional data having different accuracy are generated.

In the three-dimensional shape measurement method according to the present invention, when the three-dimensional shape of the subject represented by the first three-dimensional data is displayed on the display means, designation of display parameters for the three-dimensional shape is accepted,
The three-dimensional shape represented by the first three-dimensional data may be displayed on the display unit by the designated display parameter.

The three-dimensional shape reproduction method according to the present invention acquires the second three-dimensional data and the display parameter acquired by the three-dimensional shape measurement method specifying the display parameter according to the present invention,
A three-dimensional shape represented by the second three-dimensional data is displayed based on the display parameter.

  In addition, you may provide as a program for making a computer perform the three-dimensional shape measuring method and three-dimensional shape reproduction | regeneration method by this invention.

  According to the present invention, when generating three-dimensional data representing a three-dimensional shape of a subject, first and second three-dimensional data having different accuracy are generated. For this reason, when displaying a three-dimensional shape, if the first three-dimensional data with low accuracy is used, the three-dimensional shape can be displayed at high speed. Further, if the second 3D data with high accuracy is used for recording, it can be displayed later when time is available, and a highly accurate 3D shape can be used.

  In addition, by determining the calculation method for generating the first three-dimensional data, it is possible to generate the first three-dimensional data with the accuracy required, and as a result, the first three-dimensional data with the appropriate accuracy can be generated. Three-dimensional data can be output.

  In this case, by determining the calculation method based on the resolution of the display means, it is possible to have accuracy suitable for displaying the first three-dimensional data on the display means.

  In this case, since the calculation method is determined based on the display speed of the three-dimensional shape of the display means, the first three-dimensional data can be generated with the highest possible accuracy in time for the display speed. It is possible to have accuracy suitable for displaying the three-dimensional data on the display means.

  In this case, by determining the calculation method based on the output format of the imaging means, it is possible to output the first three-dimensional data with an accuracy suitable for the output format.

  Further, when displaying the three-dimensional shape of the subject represented by the first three-dimensional data on the display means, the designation of the display parameter of the three-dimensional shape is accepted, and the first three-dimensional data is represented by the designated display parameter. By displaying the three-dimensional shape, the three-dimensional shape can be displayed in a desired display mode.

  Further, by determining the calculation method of the first three-dimensional data according to the display parameter, calculation for display is performed based on the display parameter only for the three-dimensional shape to be displayed, thereby reducing the calculation amount. be able to.

  Also, the first and second three-dimensional data are generated by sequentially generating a plurality of three-dimensional data having different accuracy in stages from a low accuracy, and outputting the three-dimensional data in the middle of generation as the first three-dimensional data. Since it is not necessary to provide a dedicated means for generating data, the cost of the apparatus can be reduced.

  Further, by separately providing means for generating the first three-dimensional data and means for generating the second three-dimensional data, it is possible to efficiently generate the respective three-dimensional data.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing an internal configuration of a stereo camera 1 to which a three-dimensional shape measuring apparatus and a three-dimensional shape reproducing apparatus according to a first embodiment of the present invention are applied. As shown in FIG. 1, the stereo camera 1 according to the first embodiment includes two imaging units 21A and 21B, an imaging control unit 22, an image processing unit 23, a compression / decompression processing unit 24, a frame memory 25, and a media control unit 26. An internal memory 27 and a display control unit 28.

  FIG. 2 is a diagram illustrating the configuration of the imaging units 21A and 21B. As shown in FIG. 2, the imaging units 21A and 21B include lenses 10A and 10B, diaphragms 11A and 11B, shutters 12A and 12B, CCDs 13A and 13B, analog front ends (AFE) 14A and 14B, and an A / D conversion unit 15A. 15B is provided.

  The lenses 10A and 10B are composed of a plurality of functional lenses such as a focus lens for focusing on a subject and a zoom lens for realizing a zoom function, and their positions are adjusted by a lens driving unit (not shown). In the present embodiment, the focal position is assumed to be fixed.

  In the diaphragms 11A and 11B, the diaphragm diameter is adjusted based on the diaphragm value data obtained by the AE process by a diaphragm driving unit (not shown). In the present embodiment, it is assumed that the aperture value data is fixed.

  The shutters 12A and 12B are mechanical shutters, and are driven by a shutter driving unit (not shown) according to the shutter speed obtained by the AE process. In this embodiment, the shutter speed is assumed to be fixed.

  The CCDs 13A and 13B have a photoelectric surface in which a large number of light receiving elements are two-dimensionally arranged, and subject light is imaged on the photoelectric surface and subjected to photoelectric conversion to obtain an analog imaging signal. In addition, color filters in which R, G, and B color filters are regularly arranged are arranged on the front surfaces of the CCDs 13A and 13B.

  The AFEs 14A and 14B perform processing for removing noise of the analog imaging signal and processing for adjusting the gain of the analog imaging signal (hereinafter referred to as analog processing) for the analog imaging signals output from the CCDs 13A and 13B.

  The A / D converters 15A and 15B convert the analog imaging signals subjected to analog processing by the AFEs 14A and 14B into digital signals. Note that image data obtained by the CCDs 13A and 13B of the imaging units 21A and 21B and converted into digital signals is RAW data having R, G, and B density values for each pixel. The image represented by the image data acquired by the imaging unit 21A is referred to as a standard image G1, and the image represented by the image data acquired by the imaging unit 21B is referred to as a reference image G2.

  The imaging control unit 22 shoots at a predetermined frame rate, for example, 30 fps for displaying a through image before the release button is pressed, and performs shooting for recording when the release button is pressed. The imaging units 21A and 21B are controlled. The imaging control unit 22 may be provided in the imaging units 21A and 21B.

  In this embodiment, the focus position, aperture value data, and shutter speed are fixed, but AF processing and AE processing are performed to set the focus position, aperture value data, and shutter speed each time shooting is performed. It may be.

  The image processing unit 23 performs correction processing for correcting variations in sensitivity distribution of the image data and distortion of the optical system on the digital image data acquired by the imaging units 21A and 21B, and parallelizes the two images. For parallel processing. Further, image processing such as white balance adjustment processing, gradation correction, sharpness correction, and color correction is performed on the image after the parallel processing. Note that reference codes G1 and G2 before processing are also used for the standard image and the reference image after processing in the image processing unit 23.

  The compression / decompression processing unit 24 compresses image data representing the base image G1 and the reference image G2 obtained after the release button is pressed and processed by the image processing unit 23 in a compression format such as JPEG. The three-dimensional image file is generated together with the three-dimensional data generated as described later. This three-dimensional image file includes a standard image G1, a reference image G2, and three-dimensional data. The 3D image file is provided with a header in which incidental information such as the shooting date and time is described based on the Exif format or the like.

  The frame memory 25 is a working memory used when performing various processes including the processes performed by the image processing unit 23 on the image data of the base image G1 and the reference image G2 acquired by the imaging units 21A and 21B. is there.

  The media control unit 26 accesses the recording medium 29 and controls writing and reading of the 3D image file.

  The internal memory 27 stores various constants set in the stereo camera 1, programs executed by the CPU 36, and the like.

  The display control unit 28 is for displaying image data and three-dimensional data stored in the frame memory 25 on the monitor 20 and displaying an image recorded on the recording medium 29 on the monitor 20. In the present embodiment, the monitor 20 is a touch panel, and various instructions can be given from the monitor 20 to the stereo camera 1 when the photographer touches the monitor 20.

  In addition, the stereo camera 1 includes a three-dimensional data generation unit 30.

  The three-dimensional data generation unit 30 generates three-dimensional data using a stereo matching technique. For this reason, as shown in FIG. 3, the three-dimensional data generation unit 30 points in the real space mapped to a certain pixel Pa on the reference image G1 from the point O1, such as points P1, P2, and P3. Since there are a plurality of lines on the line of sight, there is a pixel Pa ′ on the reference image R corresponding to the pixel Pa on a straight line (epipolar line) that is a map of the points P1, P2, P3, etc. in the real space. Based on the reference image G2, the corresponding point between the base image G1 and the reference image G2 is searched. In FIG. 3, a point O1 is a viewpoint of the imaging unit 21A that is a reference camera, and a point O2 is a viewpoint of the imaging unit 21B that is a reference camera. Here, the viewpoint is the focal point of the optical system of the imaging units 21A and 21B. The search for corresponding points may use the base image G1 and the reference image G2 that have been subjected to image processing, but use the base image G1 and the reference image G2 that have been subjected only to the parallel processing before the image processing. Is preferred. In the following description, it is assumed that the search for corresponding points uses the base image G1 and the reference image G2 before image processing.

  Specifically, when searching for corresponding points, the three-dimensional data generation unit 30 moves a predetermined correlation window W along the epipolar line, and the reference image G1 and the reference image G2 are moved at each movement position. The correlation for the pixels in the correlation window W is calculated, and the central pixel of the correlation window W at the position where the correlation on the reference image G2 is maximized is taken as the corresponding point corresponding to the pixel Pa on the standard image G1. As the correlation evaluation value for evaluating the correlation, the sum of absolute differences, the reciprocal of the sum of squared differences, or the like can be used. In this case, the smaller the correlation evaluation value, the larger the correlation.

  FIG. 4 is a diagram for explaining the positional relationship between the base image and the reference image after the parallel processing. As shown in FIG. 4, the image plane that is the plane from which the standard image G1 and the reference image G2 are obtained in the imaging units 21A and 21B has an origin at the intersection with the optical axis of the imaging units 21A and 21B. In addition, the coordinate systems of the imaging units 21A and 21B on the image plane are (u, v) and (u ′, v ′), respectively. Here, since the optical axes of the imaging units 21A and 21B are made parallel by the parallel processing, the u axis and the u ′ axis on the image plane face the same direction on the same straight line. Further, since the epipolar line on the reference image G2 becomes parallel to the u ′ axis by the parallel processing, the u axis on the standard image G1 also coincides with the direction of the epipolar line of the reference image G2. .

  Here, it is assumed that the focal lengths of the imaging units 21A and 21B are f and the base line length is b. The focal length f and the baseline length b are calculated in advance as calibration parameters and stored in the internal memory 27. At this time, the distance data (X, Y, Z) representing the position in the three-dimensional space is expressed by the following formulas (1) to (3) based on the coordinate system of the imaging unit 21A.

X = b · u / (u−u ′) (1)
Y = b · v / (u−u ′) (2)
Z = b · f / (u−u ′) (3)
Here, u−u ′ is the amount of deviation (parallax) in the horizontal direction of the projection point on the image plane of the imaging units 21A and 21B.

  By calculating a plurality of distance data (X, Y, Z) in the three-dimensional space in this way, the three-dimensional shape of the subject included in the three-dimensional space can be represented, and a set of a plurality of distance data is 3 It becomes the dimension data V0. Here, the distance data X and Y are position information indicating the position of the pixel, and the distance data Z is information indicating the distance. The distance data is calculated only in the common range of the base image G1 and the reference image G2.

  The three-dimensional data generation unit 30 uses the obtained corresponding points to calculate 3 distance data (X, Y, Z) representing the distance from the imaging units 21A, 21B to the subject by the above formulas (1) to (3). A plurality of values are calculated in the dimension space, and three-dimensional data V0 representing the three-dimensional shape of the subject H, which is composed of the calculated plurality of distance data (X, Y, Z), is generated. In the present embodiment, the three-dimensional shape of the subject is represented by a mesh model, and the three-dimensional data V0 is represented by a mesh model. Further, a volume model may be used instead of the mesh model.

  Here, in the present embodiment, the three-dimensional data generation unit 30 generates three-dimensional data by performing a coarse / fine search method, subpixel position estimation, and occlusion interpolation, as shown in FIG. The coarse / fine search method, subpixel position estimation, and occlusion interpolation will be described below. First, the coarse / fine search method will be described. FIG. 6 is a diagram for explaining the coarse / fine search method.

  First, the three-dimensional data generation unit 30 generates a resolution standard image and a resolution reference image having different resolutions stepwise from the standard image G1 and the reference image G2. Specifically, when the number of pixels of the standard image G1 and the reference image G2 is 640 × 480 pixels, the medium resolution standard of the 160 × 120 pixel low resolution standard image G1 ″, the low resolution reference image G2 ″, and 320 × 240 pixels. An image G1 ′ and a medium resolution reference image G2 ′ are generated. The original image of 640 × 480 pixels is referred to as a high resolution standard image G1 and a high resolution reference image G2. Here, the resolution is not limited to three stages as described above, and may be two stages or four or more stages.

  Then, the three-dimensional data generation unit 30 first searches for corresponding points between the low resolution standard image G1 ″ and the low resolution reference image G2 ″ as shown in FIG. The searched corresponding points are searched for only in a region including a portion corresponding to the low resolution reference image G2 ″ in the medium resolution reference image G2 ′. That is, as shown in FIG. 6, the low resolution reference image G2 is searched. When the corresponding point P11 corresponding to the point P10 of the low resolution standard image G1 "is searched for in", the medium resolution reference image G2 'has the medium resolution only in the region A1 in the predetermined range including the point P12 corresponding to the corresponding point P11. Corresponding points corresponding to the point P10 of the reference image G1 ′ are searched for.When the corresponding point P12 is searched for in the medium resolution reference image G2 ′ as shown in FIG. A corresponding point corresponding to the point P10 of the high-resolution reference image G1 is searched only in a predetermined area A2 including the point P13 corresponding to P12.

  The three-dimensional data generation unit 30 generates medium-resolution and high-resolution three-dimensional data V0 ′ and V0 each time a corresponding point is searched for in a medium-high resolution reference image. Note that low-resolution three-dimensional data is not generated in the first embodiment. Here, the three-dimensional data generation unit 30 outputs the medium-resolution three-dimensional data V0 ′ as display three-dimensional data, and outputs the high-resolution three-dimensional data V0 as recording three-dimensional data. The three-dimensional data generation unit 30 generates high-resolution three-dimensional data V0 when the release button is pressed. Before the release button is pressed, only the generation of medium-resolution three-dimensional data V0 ′ is performed. Do. Accordingly, the medium resolution three-dimensional data V0 ′ is displayed on the monitor 20 as a through image.

  Further, when searching for corresponding points, the correlation window W is basically searched for corresponding points by moving one pixel at a time on the reference image G2 of each resolution, but in order to improve the accuracy of searching for corresponding points. It is preferable to estimate the position of the corresponding point at the moving position of less than one pixel. Such estimation of the corresponding point search position is referred to as sub-pixel position estimation. Hereinafter, subpixel position estimation will be described.

  FIG. 7 is a diagram for explaining a method of subpixel position estimation. Here, in order to simplify the description, the pixel positions are shown in a one-dimensional manner. In FIG. 7, the horizontal axis indicates the pixel position, and the vertical axis indicates the correlation evaluation value. In FIG. 7, it is assumed that the smaller the correlation evaluation value, the larger the correlation.

  As shown in FIG. 7, if the correlation evaluation values at the four pixel positions x1, x2, x3, and x4 are y1, y2, y3, and y4, respectively, the pixel position x4 has the largest correlation. When pixel position estimation is not performed, the corresponding point is the pixel position x4. However, considering the quadratic curve L1 passing through each point, the position where the correlation is greatest is the position xs between the pixel position x3 and the pixel position x4. The sub-pixel position estimation method is a method for estimating the position where the correlation is greatest between the pixel positions in order to further improve the accuracy based on the relationship between the correlation evaluation value and the pixel position.

  In the first embodiment, it is assumed that subpixel position estimation is performed only when searching for corresponding points using the high-resolution standard image G1 and the high-resolution reference image G2.

  Next, occlusion interpolation will be described. As shown in FIG. 8, depending on the shape of the subject H, there may be a hidden point K0 that can be seen from the imaging unit 21A but cannot be seen from the imaging unit 21B. For such a hidden point K0, the distance data (X, Y, Z) cannot be calculated, so that the area B1 in the subject H cannot calculate three-dimensional data. Such an area is referred to as an occlusion area. For this reason, the three-dimensional data generation unit 30 uses the distance data (X, Y, Z) around the occlusion area to determine the position of the occlusion area. Distance data (X, Y, Z) is estimated. For example, as shown in FIG. 9, for the occlusion area B1 of the subject H, the distance data of the occlusion area B1 is calculated by interpolation using the distance data of the positions P20 and P21 at both ends of the occlusion area B1. Note that various known methods such as linear interpolation and higher-order interpolation can be used as the interpolation calculation.

  In the present embodiment, the occlusion region is interpolated only when the three-dimensional data V0 is generated using the high resolution standard image G1 and the high resolution reference image G2.

  The CPU 36 controls each unit of the stereo camera 1 in accordance with a signal from the input / output unit 37. Further, the operation mode of the stereo camera 1 is switched between the photographing mode and the reproduction mode in response to a mode switching instruction from the input / output unit 37.

  The input / output unit 37 includes various interfaces, a switch that can be operated by the photographer, a release button, an operation button, and the like. Note that a trackball described later is also provided.

  The data bus 38 is connected to each part constituting the stereo camera 1 and the CPU 36, and exchanges various data and various information in the stereo camera 1.

  Next, processing performed in the first embodiment will be described. FIG. 10 is a flowchart showing processing performed in the first embodiment. When the power of the stereo camera 1 is turned on and the operation mode is switched to the shooting mode, the CPU 36 starts processing, the imaging units 21A and 21B capture the subject, and the image data of the reference image G1 and the reference image G2 is obtained. Obtain (step ST1). Next, the three-dimensional data generation unit 30 generates low- and medium-resolution standard images G1 ″ and G1 ′ and reference images G2 ″ and G2 ′, and searches for corresponding points by the coarse / fine search method as described above. Medium resolution display three-dimensional data V0 'is generated (step ST2).

  In response to an instruction from the CPU 36, the display control unit 28 displays a three-dimensional image, which is a three-dimensional image represented by the display three-dimensional data V0 ', on the monitor 20 (step ST3).

  FIG. 11 is a rear view of the stereo camera 1 for explaining display of a three-dimensional image. As shown in FIG. 11, a monitor 20, a trackball 40, a zoom-in button 41, and a zoom-out button 42 are provided on the rear surface of the stereo camera 1. The trackball 40, the zoom-in button 41, and the zoom-out button 42 are part of the input / output unit 37. As shown in FIG. 11, a three-dimensional image G10 of the mesh model represented by the display three-dimensional data V0 ′ is displayed on the monitor 20. The three-dimensional image G10 is obtained by projecting the three-dimensional data V0 ′ onto a virtually set two-dimensional plane. On the left side of the three-dimensional image G10, the rotation amount of the viewpoint around the XYZ axes, the movement amount from the origin, and the zoom magnification are displayed, and on the right side of the three-dimensional image G10, the coordinates of the three-dimensional space are shown. .

  The photographer can use the trackball 40 to rotate the displayed three-dimensional image G10 around the XYZ axes so as to be viewed from a desired viewpoint. Further, the position of the viewpoint can be changed by touching the monitor 20 and moving the touched portion on the monitor 20. Further, the zoom magnification of the displayed three-dimensional image G10 can be changed using the zoom-in button 41 and the zoom-out button 42.

  It should be noted that the rotation position, movement position, and zoom magnification of the viewpoint thus changed are included as display parameters in the 3D image file together with the 3D data V0 for recording generated as described later. Specifically, the display parameters are included in the 3D image file by describing the display parameters in the header of the 3D image file.

  Next, the CPU 36 determines whether or not an instruction to change the viewpoint position and zoom magnification is made (step ST4), and if step ST4 is negative, the process proceeds to step ST6. If step ST4 is affirmed, the viewpoint position and zoom magnification are changed and the three-dimensional image G10 is displayed again (step ST5).

  Next, the CPU 36 determines whether or not the release button has been pressed (step ST6), and if step ST6 is negative, the process returns to step ST1. If step ST6 is positive, the three-dimensional data generation unit 30 performs subpixel position estimation and occlusion interpolation using the high-resolution standard image G1 and reference image G2, and generates three-dimensional data V0 for recording. (Step ST7). Then, in response to an instruction from the CPU 36, the compression / decompression processing unit 24 generates a three-dimensional image file from the three-dimensional data V0, display parameters, standard image G1, and reference image G2 image data (step ST8). Further, in response to an instruction from the CPU 36, the media control unit 26 records the three-dimensional image file on the recording medium 29 (step ST9), and ends the process.

  Next, processing at the time of reproducing the recorded three-dimensional image file will be described. FIG. 12 is a flowchart showing processing performed when a three-dimensional image file is played back in the first embodiment. When the operation mode of the stereo camera 1 is switched to the reproduction mode, the CPU 36 starts processing and accepts selection of a three-dimensional image file to be reproduced (step ST11). The selection of the 3D image file is performed by displaying a list of 3D image files recorded on the recording medium 29 on the monitor 20 and accepting an instruction to select a desired 3D image file from the list by the photographer. You can do that.

  Next, the CPU 36 accepts selection of whether or not to reproduce the 3D image using the display parameters included in the 3D image file, that is, selection of whether or not to use the display parameters at the time of shooting (step ST12). This may be performed, for example, by displaying the selection screen 50 on the monitor 20 as shown in FIG. 13 and accepting the selection of the YES button 51 or the NO button 52 by the photographer.

  Next, it is determined whether or not the use of display parameters at the time of photographing is selected (step ST13). When step ST13 is affirmed, display parameters described in the header of the three-dimensional image file are acquired (step ST14). The three-dimensional image represented by the three-dimensional data V0 stored in the three-dimensional image file is displayed on the monitor 20 based on the viewpoint position and zoom magnification based on the acquired parameters (step ST15), and the process is terminated. If step ST13 is negative, a predetermined default display parameter is acquired (step ST16), and the process proceeds to step ST15.

  The display mode of the three-dimensional image is the same as that shown in FIG. At this time, as in the case of shooting, changes in the rotation position, movement position, and zoom magnification of the viewpoint may be accepted. Alternatively, the instruction to reproduce the standard image G1 and the reference image G2 may be received, and the standard image G1 and the reference image G2 may be displayed on the monitor 20 as shown in FIG.

  As described above, according to the first embodiment, the medium-resolution three-dimensional data V0 ′ for display and the high-resolution three-dimensional data V0 for recording are generated. With V0 ′, the three-dimensional image G10 can be displayed on the monitor 20 at high speed. In addition, the recorded three-dimensional data V0 has high resolution and high measurement accuracy because subpixel position estimation and occlusion interpolation are performed. Therefore, it is possible to use a highly accurate three-dimensional shape by reproducing a three-dimensional image represented by the three-dimensional data V0 when there is a time after shooting.

  In addition, when the 3D image is reproduced, the 3D image can be displayed with the same viewpoint position and zoom magnification as those used during the shooting by using the display parameters used during the shooting. At this time, since selection of whether or not to use the display parameter used at the time of photographing is accepted, it is possible to reflect the photographer's intention regarding the display mode at the time of reproducing the three-dimensional data V0.

  Next, a second embodiment of the present invention will be described. FIG. 15 is a schematic block diagram showing an internal configuration of the stereo camera 1 to which the three-dimensional shape measuring apparatus and the three-dimensional shape reproducing apparatus according to the first embodiment of the present invention are applied. In the second embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted here. The stereo camera 1A according to the second embodiment is provided with an arithmetic method determining unit 31 for determining an arithmetic method used by the three-dimensional data generating unit 30 to generate three-dimensional data for display. Different from form. Note that the three-dimensional data generation unit 30 in the second embodiment also generates sub-resolution and medium resolution three-dimensional data V0 ″ and V0 ′ according to the determination result of the calculation method determination unit 31. Assume that pixel position estimation and occlusion interpolation may be performed.

  Here, the calculation method determination unit 31 determines the calculation method according to the resolution of the monitor 20, the speed required for reproducing the three-dimensional image of the monitor 20, the output formats of the imaging units 21A and 21B, and the display parameters. .

  Next, processing performed in the second embodiment will be described. Note that the processing performed in the second embodiment is performed in the first embodiment until the generation of the display three-dimensional data after shooting, and the display three-dimensional data based on the determined calculation method. The processing performed is the same as that of the first embodiment except for generating. For this reason, in the following description, only the process for determining the calculation method will be described. First, the process of determining the calculation method according to the resolution of the monitor 20 will be described as the first process. FIG. 16 is a flowchart showing a first process for determining an arithmetic method performed in the second embodiment.

  First, the calculation method determination unit 31 acquires the resolution of the monitor 20 (step ST21). Then, the shift amount of the corresponding point on the monitor 20 when the position of the corresponding point searched by the three-dimensional data generation unit 30 is shifted by one pixel is calculated (step ST22). Here, if the calculated shift amount is equal to or less than one pixel of the monitor 20, there is no difference in the displayed three-dimensional image, so there is no need to perform subpixel position estimation.

  Therefore, the calculation method determination unit 31 determines whether or not the shift amount of the corresponding point on the monitor 20 is equal to or smaller than one pixel of the monitor when the position of the corresponding point to be searched is shifted by one pixel (step ST23). ) If step ST23 is affirmed, a calculation method that does not perform sub-pixel position estimation is selected (step ST24), and the process ends. On the other hand, if step ST23 is negative, a calculation method for performing subpixel position estimation is selected (step ST25), and the process is terminated.

  This eliminates the need to perform a calculation for sub-pixel position estimation that is not reflected in the display of the three-dimensional image, thereby improving the speed of generating the three-dimensional data for display.

  Next, the processing for determining the calculation method according to the time required to reproduce the three-dimensional image on the monitor 20 will be described as the second processing. FIG. 17 is a flowchart showing the second processing of the calculation method determination performed in the second embodiment. Note that the second process is performed at the time of generating the display three-dimensional data for the second and subsequent times after generating the display three-dimensional data V0 ′ once as in the first embodiment.

  First, the calculation method determination part 31 acquires the data amount of the three-dimensional data for display produced | generated in the last process (step ST31). Then, a time Tf required to display one frame at the time of displaying a through image is calculated from the data amount of the acquired three-dimensional data (step ST32). The time required to display one frame is the time required to project 3D data onto a virtually set 2D plane and display a 2D display 3D image on the monitor 20.

  Here, in the present embodiment, since the corresponding points are searched from the standard image and the reference image having N levels of resolution in the coarse / fine search method, the steps necessary to search the corresponding points and calculate the three-dimensional data. The calculation time Ti for each is calculated (step ST33). In this embodiment, since N = 3, i = 1 to 3. Next, the corresponding point search stage i is set to an initial value (i = 1, step ST34), and it is determined whether or not the time required to calculate up to the i-th stage exceeds the time required to display one frame ( Ti> Tf: Step ST35).

  If step ST35 is affirmed, it is determined whether or not i> 1 (step ST36). If step ST36 is affirmed, i = i-1 (step ST37), and calculation is performed at the i-th stage 3 The three-dimensional data is determined as display three-dimensional data (step ST38), and the process is terminated. If step ST36 is negative, the process proceeds to step ST38.

  If step ST35 is negative, it is determined whether i = N (step ST39). If step ST39 is negative, i = i + 1 (step ST40) and the process returns to step ST35. If step ST39 is affirmed, the process proceeds to step ST38.

  This makes it possible to generate three-dimensional data for display with the highest possible accuracy within the displayable frame rate.

  Next, the process of determining the calculation method according to the output format of the imaging units 21A and 21B will be described as a third process. FIG. 18 is a flowchart showing a third process of the calculation method determination performed in the second embodiment.

  First, the calculation method determination unit 31 determines whether or not the output format of the imaging units 21A and 21B is a pixel mixed output (step ST51). If step ST51 is affirmed, all the data of the image data of the base image G1 and the reference image G2 are determined as a calculation method (first calculation method) to be used for calculation of three-dimensional data (step 1). ST52), the process ends.

  If step ST51 is negative, it is determined whether the output format of the imaging units 21A and 21B is an interlaced output (step ST53). If step ST53 is affirmed, an arithmetic method that uses only the odd-numbered field or even-numbered field data of the image data of the base image G1 and the reference image G2 to calculate three-dimensional data (the second arithmetic method). (Step ST54), and the process ends.

  If step ST53 is negative, neither the pixel mixed output nor the interlaced output is used, so in order to shorten the calculation time, a calculation method for thinning out data and calculating three-dimensional data is determined (step ST55). The process is terminated.

  Thereby, according to the output format of imaging part 21A, 21B, the three-dimensional data for a display with as high a measurement precision and resolution as possible can be produced | generated.

  Next, the process of determining the calculation method according to the display parameter will be described as a fourth process. FIG. 19 is a flowchart showing a fourth process for determining the calculation method performed in the second embodiment. Note that the fourth processing is performed at the time of generating the display three-dimensional data for the second and subsequent times after generating the display three-dimensional data V0 ′ once as in the first embodiment.

  First, the calculation method determination unit 31 acquires the current display parameter (step ST61). Then, in the three-dimensional image represented by the currently displayed three-dimensional data, a surface that is not displayed (that is, a surface that becomes occlusion) is detected among the surfaces constituting the mesh model (step ST62). And the non-display corresponding point which comprises only the surface which is not displayed is detected from each corresponding point which comprises a mesh (step ST63). Then, the calculation method determination unit 31 excludes the non-display corresponding point from the target of stereo matching at the time of generating the next three-dimensional data (step ST64), and the process ends.

  Thereby, since stereo matching is performed only for the three-dimensional shape displayed on the monitor 20, the amount of calculation can be reduced, and as a result, three-dimensional data for display can be generated at higher speed.

  In the second embodiment, a calculation method is selected according to at least one of the resolution of the monitor 20, the speed required to reproduce the three-dimensional image on the monitor 20, the output format of the imaging units 21A and 21B, and the display parameters. May be determined. In this case, the photographer may be input using the input / output unit 37 to determine the calculation method depending on any of these, but may be set in the stereo camera in advance.

  In the first and second embodiments, the three-dimensional data generating unit 30 generates three-dimensional data for display and recording. The stereo camera according to the third embodiment shown in FIG. As in 1B, a display three-dimensional data generation unit 32 that generates display three-dimensional data and a recording three-dimensional data generation unit 33 that generates recording three-dimensional data may be provided.

  In the first and second embodiments, after the generation of the display and recording three-dimensional data, the three-dimensional data is recursively regenerated or the three-dimensional shape is changed in order to improve accuracy. You may make it perform the smoothing process for adjusting. In the second embodiment, the calculation method determination unit 31 may determine whether to generate recursive three-dimensional data and perform a smoothing process.

  In the first and second embodiments, the image data of the standard image and the reference image are included in the three-dimensional image file. However, only the image data of the standard image may be included. Further, only the 3D data may be included in the 3D image file without including the image data of the standard image and the reference image.

  In the first and second embodiments, two imaging units 21A and 21B are provided, and three-dimensional data is generated from images acquired by the two imaging units 21A and 21B. The three-dimensional data may be generated from the acquired three or more image data.

  In the first and second embodiments, the three-dimensional shape measuring apparatus and the three-dimensional shape reproducing apparatus according to the present invention are applied to a stereo camera including the imaging units 21A and 21B. 21B, the three-dimensional shape measuring device, and the three-dimensional shape reproducing device may be provided separately. Further, the three-dimensional shape measuring device and the three-dimensional shape reproducing device may be provided separately.

  As mentioned above, although embodiment of this invention was described, the program which makes a computer function as a means corresponding to said three-dimensional data generation part 30, and performs a process as shown in FIG. It is one of the embodiments of the present invention. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention.

1 is a schematic block diagram showing an internal configuration of a stereo camera to which a three-dimensional shape measuring apparatus and a three-dimensional shape reproducing apparatus according to a first embodiment of the present invention are applied. Diagram showing the configuration of the imaging unit Diagram for explaining stereo matching The figure for demonstrating the positional relationship of the reference | standard image after a parallelization process, and a reference image Processing block diagram in the three-dimensional data generation unit Diagram for explaining the coarse / fine search method Diagram for explaining the method of subpixel position estimation Illustration for explaining hidden points Illustration for explaining occlusion interpolation The flowchart which shows the process performed in 1st Embodiment. Rear view of stereo camera for explaining display of 3D image The flowchart which shows the process performed at the time of reproduction | regeneration in 1st Embodiment The figure which shows the selection screen of whether to use the display parameter at the time of photography The figure which shows the state which displayed the reference | standard image and the reference image The schematic block diagram which shows the internal structure of the stereo camera to which the 3D shape measuring apparatus and 3D shape reproduction | regeneration apparatus by the 2nd Embodiment of this invention are applied. The flowchart which shows the 1st process of the calculation method determination performed in 2nd Embodiment. The flowchart which shows the 2nd process of the calculation method determination performed in 2nd Embodiment. The flowchart which shows the 3rd process of the calculation method determination performed in 2nd Embodiment. The flowchart which shows the 4th process of the calculation method determination performed in 2nd Embodiment. The schematic block diagram which shows the internal structure of the stereo camera to which the 3D shape measuring apparatus and 3D shape reproduction | regeneration apparatus by the 3rd Embodiment of this invention are applied.

Explanation of symbols

1, 1A, 1B Stereo camera 21A, 21B Imaging unit 30 Three-dimensional data generation unit 31 Calculation method determination unit 32 Display three-dimensional data generation unit 33 Recording three-dimensional data generation unit

Claims (14)

Image data acquisition means for acquiring a plurality of image data respectively representing a plurality of images acquired by photographing a subject from different positions;
3D data generating means for generating 3D data representing the 3D shape of the subject from the plurality of image data, wherein the 3D data generating means generates first and second 3D data having different accuracy. And a three-dimensional shape measuring apparatus. A calculation method determining means for determining a calculation method for generating the first three-dimensional data;
2. The three-dimensional shape measuring apparatus according to claim 1, wherein the three-dimensional data generating means is means for generating the first three-dimensional data by the determined calculation method.   The calculation method determining means determines the calculation method based on at least one of the resolution of the display means for displaying the three-dimensional shape, the display speed of the three-dimensional shape of the display means, and the output format of the imaging means. The three-dimensional shape measuring apparatus according to claim 2, wherein Parameter designation means for accepting designation of a display parameter of the three-dimensional shape when displaying the three-dimensional shape of the subject represented by the first three-dimensional data on a display means;
4. The display control unit according to claim 1, further comprising a display control unit that displays the three-dimensional shape represented by the first three-dimensional data on the display unit according to the designated display parameter. 3. The three-dimensional shape measuring apparatus according to item 1. Parameter designation means for accepting designation of a display parameter of the three-dimensional shape when displaying the three-dimensional shape of the subject represented by the first three-dimensional data on a display means;
A calculation method determining means for determining a calculation method of the first three-dimensional data according to the display parameter;
2. The three-dimensional shape measuring apparatus according to claim 1, wherein the three-dimensional data generating means is means for generating the first three-dimensional data by the determined calculation method.   The three-dimensional data generation means is means for sequentially generating a plurality of three-dimensional data having different accuracy in stages from a low accuracy and outputting the intermediate three-dimensional data as the first three-dimensional data. The three-dimensional shape measuring apparatus according to any one of claims 1 to 5, wherein   The three-dimensional data generation means includes first means for generating the first three-dimensional data and second means for generating the second three-dimensional data. The three-dimensional shape measuring apparatus according to any one of 1 to 5. Data acquisition means for acquiring the second three-dimensional data and the display parameter acquired by the three-dimensional shape measuring apparatus according to any one of claims 4 to 7,
A three-dimensional shape reproducing apparatus comprising: display means for displaying an image having a three-dimensional shape represented by the second three-dimensional data based on the display parameter. Obtaining a plurality of image data respectively representing a plurality of images obtained by photographing the subject from different positions;
A three-dimensional shape measuring method characterized by generating first and second three-dimensional data with different accuracy when generating three-dimensional data representing a three-dimensional shape of the subject from the plurality of image data. When displaying on the display means the three-dimensional shape of the subject represented by the first three-dimensional data, designation of display parameters for the three-dimensional shape is accepted,
10. The three-dimensional shape measurement method according to claim 9, wherein the three-dimensional shape represented by the first three-dimensional data is displayed on the display means by the designated display parameter. The second three-dimensional data and the display parameter acquired by the three-dimensional shape measurement method according to claim 10 are acquired,
A three-dimensional shape reproduction method, comprising: displaying a three-dimensional shape represented by the second three-dimensional data based on the display parameter. A procedure for acquiring a plurality of image data respectively representing a plurality of images acquired by photographing a subject from different positions;
A step of generating first and second three-dimensional data having different accuracy when generating three-dimensional data representing the three-dimensional shape of the subject from the plurality of image data. A program for causing a computer to execute a measurement method. A procedure for receiving designation of a display parameter of the three-dimensional shape when displaying the three-dimensional shape of the subject represented by the first three-dimensional data on a display unit;
13. The program according to claim 12, further comprising a step of displaying the three-dimensional shape represented by the first three-dimensional data on the display means by the designated display parameter. A procedure for acquiring the second three-dimensional data and the display parameter acquired by the three-dimensional shape measurement method according to claim 10;
A program for causing a computer to execute a three-dimensional shape reproduction method, comprising: displaying a three-dimensional shape represented by the second three-dimensional data based on the display parameter.

Images