JP2012202942A - Three-dimensional modeling device, three-dimensional modeling method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately create a three-dimensional model even when using a zoom function.SOLUTION: An image acquisition unit 11 acquires a pair of images. A feature point correspondence extraction section 12B extracts pairs of corresponding feature points from each image of the pair of images acquired by the image acquisition unit 11. An error correspondence removal section 12C removes a pair, for which a difference in vertical coordinates is equal to or larger than a predetermined value, as a pair of erroneous corresponding points from the pairs of corresponding feature points which are extracted by the feature point correspondence extraction section 12B. A first correction section 12D corrects the pair of images, which has been acquired by the image acquisition unit 11, on the basis of the vertical coordinates of the pairs of corresponding feature points from which the pair of erroneous corresponding points has been removed, so as to obtain images suitable for creating a three-dimensional model. A three-dimensional model generation unit 13 generates the three-dimensional model of a subject from the pair of images acquired by the image acquisition unit 11 and corrected by an image correction unit 12.

Description

  The present invention relates to a three-dimensional modeling apparatus, a three-dimensional modeling method, and a program for appropriately three-dimensionally modeling a subject.

  A subject such as a human being, an animal, or a work of art is imaged using a stereo camera including an imaging unit for the right eye and an imaging unit for the left eye, and a stereo image obtained by the imaging (an image for the right eye and an image for the left eye) ), A technique for constructing a three-dimensional model of a subject is known.

  Also disclosed is a technique that makes it possible to construct a three-dimensional model from a stereo image captured using the zoom function of a camera.

  For example, in the invention described in Patent Literature 1, correction data (camera information) measured in advance is stored in a table for each zoom position. Then, when a stereo image is captured, the captured zoom position is specified from the imaging control unit. Then, the correction data corresponding to the specified zoom position is acquired from the table, and the stereo image captured using the zoom function is corrected by performing distortion correction and parallelization of the stereo image based on the acquired correction data. Create a 3D model from an image.

JP 2008-241491 A

  The left and right imaging units of a stereo camera often have errors in internal parameters such as focal length due to the effects of aging and mechanical “play”. When such an error occurs, even if the correction and parallelization of the stereo image are performed using the correction data stored for each zoom position as in the invention described in Patent Document 1, the left and right There is a problem in that the accuracy of the created three-dimensional model is lowered because the height of the epipolar line differs in the image.

  Further, in the invention described in Patent Document 1, since there are variations in driving devices such as a motor that moves the lens position of the camera for zooming, it is necessary to measure correction data for each zoom position in advance in fine steps. is there. Therefore, it takes a long time to measure and there is a problem that the productivity of the product is lowered.

The present invention has been made in view of the above circumstances, and a three-dimensional modeling apparatus, a three-dimensional modeling method, and a program that can accurately create a three-dimensional model even when the zoom function is used. The purpose is to provide.
Another object of the present invention is to provide a three-dimensional modeling apparatus, a three-dimensional modeling method, and a program that can reduce the time required for measuring correction data for each zoom position.

In order to achieve the above object, the three-dimensional modeling apparatus of the present invention provides:
Image acquisition means for acquiring a plurality of images obtained by imaging the same subject from different angles;
Feature point correspondence acquisition means for acquiring a feature point from each image acquired by the image acquisition means, and acquiring a set corresponding to the feature point by matching the acquired feature points;
Among the feature point correspondence sets acquired by the feature point correspondence acquisition unit, an erroneous correspondence removal unit that removes, as an erroneous correspondence, a pair whose vertical coordinate difference is equal to or greater than a predetermined value;
The display magnification of the subject between the images acquired by the image acquisition unit based on the vertical coordinates of the set corresponding to the feature point acquired by the feature point correspondence acquisition unit and after the erroneous correspondence is removed by the erroneous correspondence removal unit First image correcting means for correcting the image so that the difference between the two and the difference between the vertical coordinates are within a predetermined range;
3D model generation means for generating a 3D model of the subject from each image corrected by the first image correction means;
It is characterized by providing.

  According to the present invention, a three-dimensional model can be created with high accuracy even when the zoom function is used. In addition, the time required for measuring correction data for each zoom position can be shortened.

It is a figure which shows the external appearance structure of the stereo camera which concerns on embodiment of this invention, (A) shows the front side, (B) shows the back side. It is a block diagram which shows the electric constitution of the stereo camera of this embodiment. In the stereo camera of this embodiment, it is a block diagram which shows the main function structures which concern on a stereo imaging | photography mode. It is a flowchart which shows the procedure of the three-dimensional modeling process in this embodiment. It is a flowchart which shows the procedure of the image correction process in this embodiment. It is a figure for demonstrating the correction | amendment of the image in this embodiment, (A) shows the image before miscorrespondence removal, (B) shows the image after miscorrespondence removal. It is a flowchart which shows the procedure of the three-dimensional model production | generation process in this embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, an example in which the present invention is applied to a digital stereo camera is shown. In addition, this invention is not limited by the following embodiment and drawing. Moreover, a change can be added to following embodiment and drawing in the range which does not change the summary of this invention. Moreover, the same code | symbol is attached | subjected to the same or an equivalent part in a figure.

  1A and 1B are external views of a stereo camera 1 according to this embodiment. As shown in FIG. 1A, a lens 111A, a lens 111B, and a strobe light emitting unit 400 are provided on the front surface of the stereo camera 1, and a shutter button 331 is provided on the upper surface. The lens 111 </ b> A and the lens 111 </ b> B are arranged at a predetermined interval so that the center positions thereof are on the same line in the horizontal direction when the stereo camera 1 is horizontal in the direction in which the shutter button 331 is upward. ing. The strobe light emitting unit 400 emits strobe light toward the subject as necessary. The shutter button 331 is a button for receiving a shutter operation instruction from the user.

  As shown in FIG. 1B, a display portion 310, operation keys 332, a power button 333, and a zoom button 334 are provided on the rear surface of the stereo camera 1. The display unit 310 is composed of, for example, a liquid crystal display device and functions as an electronic viewfinder for displaying various screens necessary for operating the stereo camera 1, live view images at the time of shooting, captured images, and the like. To do.

  The operation key 332 includes a cross key, a determination key, and the like, and accepts various operations from the user such as mode switching and display switching. The power button 333 is a button for accepting power on / off of the stereo camera 1 from the user.

  The zoom button 334 is a button for operating the zoom function of the stereo camera 1. By pressing the zoom button 334, the imaging magnifications of the first imaging unit 100A and the second imaging unit 100B are changed simultaneously.

  FIG. 2 is a block diagram showing an electrical configuration of the stereo camera 1. As shown in FIG. 2, the stereo camera 1 includes a first imaging unit 100A, a second imaging unit 100B, a data processing unit 200, an I / F unit 300, and a strobe light emitting unit 400.

  Each of the first imaging unit 100A and the second imaging unit 100B is a part that has a function of imaging a subject. The stereo camera 1 is a so-called compound-eye camera, and has a configuration including two imaging units as described above, but the first imaging unit 100A and the second imaging unit 100B have the same configuration. Hereinafter, “A” is added to the end of the reference numeral for the configuration of the first imaging unit 100A, and “B” is added to the end of the reference numeral for the configuration of the second imaging unit 100B.

  As shown in FIG. 2, the first imaging unit 100A (second imaging unit 100B) includes an optical device 110A (110B), an image sensor unit 120A (120B), a lens driving unit 130A (130B), and the like. Yes. The optical device 110A (110B) includes, for example, a movable lens, a diaphragm mechanism, a shutter mechanism, and the like, and performs an optical operation related to imaging. That is, the operation of the optical device 110A (110B) collects incident light and adjusts optical elements related to the angle of view, focus, exposure, and the like, such as focal length, aperture, and shutter speed.

  The shutter mechanism included in the optical device 110A (110B) is a so-called mechanical shutter. When the shutter operation is performed only by the operation of the image sensor, the optical device 110A (110B) may not include the shutter mechanism. Good. The optical device 110A (110B) operates under the control of the control unit 210 described later.

  The image sensor unit 120A (120B) generates an electrical signal corresponding to the incident light collected by the optical device 110A (110B). The image sensor unit 120A (120B) is composed of, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and performs an electrical signal according to the intensity of received light by performing photoelectric conversion. And the generated electrical signal is output to the data processing unit 200.

  The lens driving unit 130A (130B) includes a stepping motor, an actuator, and the like, and moves the lens of the optical device 110A (110B) in a direction perpendicular to the paper surface of FIG. 1 based on an instruction from the control unit 210. As the lens moves, the imaging magnification of the subject changes, and a zoom function is achieved. In the present embodiment, both lens driving units 130A and 130B are driven and controlled almost simultaneously, and the imaging magnifications of the first imaging unit and the second imaging unit are controlled to be the same even if the zoom function is used. However, in reality, when the zoom function is used due to the deterioration of the lens driving units 130A and 130B and the influence of mechanical “play”, an error occurs in the imaging magnification of the first imaging unit and the second imaging unit. There is a high possibility.

  As described above, the first imaging unit 100A and the second imaging unit 100B have the same configuration. More specifically, the specifications such as the focal length f and F value of the lens, the aperture range of the aperture mechanism, the size and number of pixels of the image sensor, the arrangement, and the pixel area are all the same. When the first imaging unit 100A and the second imaging unit 100B are operated simultaneously, two images (pair images) are captured for the same subject, but the optical axis positions are different in the horizontal direction.

  The data processing unit 200 processes the electrical signal generated by the imaging operation by the first imaging unit 100A and the second imaging unit 100B, generates digital data indicating the captured image, and performs image processing on the captured image. The data processing unit 200 includes a control unit 210, an image processing unit 220, an image memory 230, an image output unit 240, a storage unit 250, an external storage unit 260, a camera information storage unit 270, and the like.

  The control unit 210 includes, for example, a processor such as a CPU (Central Processing Unit), a main storage device such as a RAM (Random Access Memory), and the like, and executes a program stored in the storage unit 250 or the like. Each part of the stereo camera 1 is controlled. In the present embodiment, by executing a predetermined program, the control unit 210 realizes a function related to a three-dimensional modeling process described later.

  The image processing unit 220 includes, for example, an ADC (Analog-Digital Converter), a buffer memory, an image processing processor (so-called image processing engine), and the like, and is based on the electrical signals generated by the image sensor units 120A and 120B. Thus, digital data indicating the captured image is generated. That is, when the analog electrical signal output from the image sensor unit 120A (120B) is converted into a digital signal by the ADC and sequentially stored in the buffer memory, the image processing engine performs so-called development processing on the buffered digital data. By doing so, image quality adjustment and data compression are performed.

  The image memory 230 includes a storage device such as a RAM or a flash memory, and temporarily stores captured image data generated by the image processing unit 220, image data processed by the control unit 210, and the like.

  The image output unit 240 includes, for example, a circuit that generates RGB signals, converts image data stored in the image memory 230 into RGB signals and the like, and outputs them to a display screen (the display unit 310 and the like).

  The storage unit 250 includes a storage device such as a ROM (Read Only Memory) or a flash memory, and stores programs and data necessary for the operation of the stereo camera 1. In the present embodiment, it is assumed that the storage unit 250 stores data such as operation programs executed by the control unit 210 and the like, parameters and arithmetic expressions necessary for the execution.

  The external storage unit 260 includes a storage device that can be attached to and detached from the stereo camera 1 such as a memory card, and stores image data captured by the stereo camera 1, three-dimensional model data, and the like.

The camera information storage unit 270 stores information (camera information) necessary for creating a three-dimensional model from the pair images captured by the first imaging unit 100A and the second imaging unit 100B.
Specifically, the camera information is information such as the resolution, field angle, focus distance, focal length coefficient, image angle coefficient, and lens distortion coefficient of each of the first image capturing unit 100A and the second image capturing unit 100B. Of the camera information, information that changes in accordance with the zoom state of the first imaging unit 100A (second imaging unit 100B) (focal length coefficient, image angle coefficient, lens distortion coefficient, etc.) includes a plurality of zooms. A plurality of pieces of corresponding information are stored in the camera information storage unit 270 for each state. The camera information may be measured by using an existing camera calibration method using a calibration pattern or the like at the time of initial adjustment at the time of factory shipment, and stored in the camera information storage unit 270 in advance.

  The I / F unit 300 is a processing unit that performs a function related to an interface between the stereo camera 1 and a user or an external device, and includes a display unit 310, an external I / F unit 320, an operation unit 330, and the like. .

  As described above, the display unit 310 includes, for example, a liquid crystal display device, and displays and outputs various screens necessary for the user to operate the stereo camera 1, live view images at the time of shooting, captured images, and the like. To do. In the present embodiment, display output of a captured image or the like is performed based on an image signal (RGB signal) from the image output unit 240 or the like.

  The external I / F unit 320 includes, for example, a USB (Universal Serial Bus) connector, a video output terminal, and the like, and performs output of image data to an external computer device, display output of a captured image to an external monitor device, and the like. .

  The operation unit 330 includes various buttons and the like provided on the outer surface of the stereo camera 1, generates an input signal corresponding to an operation by the user, and sends it to the control unit 210. As described above, the buttons constituting the operation unit 330 include the shutter button 331, the operation key 332, the power button 333, the zoom button 334, and the like.

  The strobe light emitting unit 400 is constituted by, for example, a xenon lamp (xenon flash). The strobe light emitting unit 400 irradiates the subject with strobe light under the control of the control unit 210.

  Although the configuration of the stereo camera 1 necessary for realizing the present invention has been described above, the stereo camera 1 has a configuration for realizing the functions of a general stereo camera in addition to this. And

  Next, of the operations of the stereo camera 1, operations related to the stereo shooting mode will be described.

  FIG. 3 is a block diagram illustrating a main functional configuration for realizing the operation related to the stereo shooting mode in the stereo camera 1. As shown in FIG. 3, the stereo camera 1 functionally includes an image acquisition unit 11, an image correction unit 12, and a three-dimensional model generation unit 13. Each of these units is configured by the control unit 210, for example.

  The image acquisition unit 11 acquires a pair of images (hereinafter referred to as a pair image) in which the first imaging unit 100A and the second imaging unit 100B have captured the same subject from different angles. At this time, the image acquisition unit 11 acquires, together with the pair image, zoom control information indicating the zoom state of the first imaging unit 100A and the second imaging unit 100B when the pair image is captured.

  The image correction unit 12 corrects the pair image acquired by the image acquisition unit 11 so as to be an image suitable for creating a three-dimensional model. As shown in FIG. 3, the image correction unit 12 includes a second correction unit 12A, a feature point correspondence extraction unit 12B, an erroneous correspondence removal unit 12C, and a first correction unit 12D.

  Based on the zoom control information, the second correction unit 12A corrects the lens distortion of each image constituting the pair image, and the corresponding points on the image are arranged on the same horizontal line between the images. A parallelizing process to be corrected is performed.

  The feature point correspondence extracting unit 12B acquires a feature point from each image of the pair image corrected by the second correcting unit 12A, and a feature point correspondence set (corresponding between the paired image images) by associating the acquired feature point. A set of feature points to be extracted).

  The miscorresponding removal unit 12C removes as a miscorrespondence a pair having a vertical coordinate difference of a predetermined value or more in the pair corresponding to the feature point extracted by the feature point correspondence extracting unit 12B.

  The first correction unit 12D corrects the pair image corrected by the second correction unit 12A to an image suitable for creating a three-dimensional model based on the vertical coordinates of the set corresponding to the feature point after the erroneous correspondence is removed. .

  The three-dimensional model generation unit 13 generates a three-dimensional model of the subject from the pair images acquired by the image acquisition unit 11 and corrected by the image correction unit 12.

  Next, processing (three-dimensional modeling processing) during operation according to the stereo shooting mode will be described using the flowchart shown in FIG. The three-dimensional modeling process is started when the user operates the operation unit 330 such as the operation key 332 to select the stereo shooting mode.

  When the three-dimensional modeling process is started, first, the control unit 210 determines whether or not an end event has occurred (step S101). The end event occurs, for example, when the user performs a mode transition operation to a playback mode or when the power of the stereo camera 1 is turned off.

  When the end event has occurred (step S101; Yes), the three-dimensional modeling process ends. On the other hand, when the end event has not occurred (step S101; No), the control unit 210 performs an image based on image data obtained through one imaging unit (for example, the first imaging unit 100A) (so-called live). (View image) is displayed on the display unit 310 (step S102).

  Subsequently, the control unit 210 determines whether or not the zoom button 334 is pressed (step S103). If the zoom button 334 has not been pressed (step S103; No), the process proceeds to step S105.

  On the other hand, when the zoom button 334 is pressed (step S103; Yes), the control unit 210 changes the zoom state (imaging magnification) of the first imaging unit 100A and the second imaging unit 100B that capture an image of the subject (step S103). S104). Specifically, the control unit 210 controls the lens driving units 130A and 130B to move the lens positions of the optical devices 110A and 110B in accordance with the pressing time and the pressing position of the zoom button 334, thereby zooming in. To change. Note that the zoom state of the live view image displayed on the display unit also changes as appropriate according to the change in the zoom state.

  Subsequently, the control unit 210 determines whether or not the shutter button 331 has been pressed (step S105). When the shutter button 331 has not been pressed (step S105; No), the control unit 210 executes the process of step S101 again. On the other hand, when the shutter button 331 is pressed (step S105; Yes), the control unit 210 controls the first imaging unit 100A, the second imaging unit 100B, and the image processing unit 220 to image the subject. (Step S106).

  Subsequently, the control unit 210 (image acquisition unit 11) stores two stereo images (pair images) captured by the first imaging unit 100A and the second imaging unit 100B in the image memory 230 (step S107). At this time, the control unit 210 acquires zoom control information indicating a zoom state at the time of imaging from both the lens driving units 130A and 130B, and stores it in the image memory 230 or the storage unit 250. For example, when the lens driving units 130A and 130B are configured by stepping motors, the control unit 210 may acquire and store information indicating the number of steps of the stepping motor as zoom control information. In the following description, an image obtained as a result of imaging by the first imaging unit 100A among the paired images is an image A0, and an image obtained as a result of imaging by the second imaging unit 100B is an image B0.

  Subsequently, the control unit 210 (image correction unit 12) executes an image correction process for correcting the pair image stored in the image memory 230 into an image suitable for creating a three-dimensional model (step S108).

  Here, the details of the image correction processing will be described with reference to the flowchart of FIG.

First, the control unit 210 (second correction unit 12A) performs lens distortion correction and parallelization processing based on the zoom control information stored together with the pair images (step S201).
Specifically, first, the control unit 210 acquires camera information necessary for lens distortion correction and parallelization from the camera information storage unit 270. At this time, for the camera information that changes according to the zoom control information, camera information corresponding to the zoom control information stored together with the pair image is acquired. If camera information corresponding to the zoom control information is not stored in the camera information storage unit 270, if the corresponding camera information is stored in the nearby zoom control information, the camera information is used instead. . Or, by referring to a plurality of zoom control information in which there is corresponding camera information in the vicinity and weighting it from the relationship with the actual zoom control information, the camera information is linearly interpolated to obtain accurate camera information. Also good. And the control part 210 should just perform a lens distortion correction and a parallelization process using the camera information acquired in this way. In the following description, it is assumed that the pair image (image A0, image B0) has been corrected to the pair image (image A1, image B1) as a result of performing this processing.

  Here, FIG. 6A shows an example of a pair image (image A1, image B1) that has been subjected to lens distortion correction and parallelization by the process of step S201. At this stage, even if the paired images are parallelized due to the aging of both lens driving units 130A and 130B and the influence of mechanical play, a broken line between corresponding points between the image A1 and the image B1. There is a deviation in the height of the epipolar line indicated by.

Returning to FIG. 5, subsequently, the control unit 210 (feature point correspondence extracting unit 12B) acquires a feature point from each of the pair images subjected to the lens distortion correction and the parallelization processing (step S202).
For example, the control unit 210 uses the feature points having feature quantities that are robust to changes in the scale of the image, such as SURF (Speeded-Up Robust Features) feature points and SIFT (Scale-invariant feature transform) feature points as the feature points. What is necessary is just to acquire from B1.

  Subsequently, the control unit 210 (feature point correspondence extracting unit 12B) executes feature amount matching, and extracts a feature point correspondence set from the correspondence between each feature point of the image A1 and each feature point of the image B1 (step S1). S203). Specifically, the control unit 210 associates features that do not have feature points that are closest to each other and close to each other in the feature amount space.

Subsequently, the control unit 210 (miscorresponding removal unit 12C) removes a pair that has been incorrectly associated from the extracted pair of feature points (step S204).
Specifically, when the vertical coordinate difference between the corresponding feature points is equal to or greater than a predetermined threshold, the control unit 210 may remove the set corresponding to the feature points. Note that the predetermined threshold value may be set by obtaining an appropriate value taking into account the play of the zoom mechanism in advance through experiments or the like.

Subsequently, the control unit 210 (first correction unit 12D) corrects (converts) a coefficient (correction) for the pair image (image A1, image B1) from the vertical coordinate of the set corresponding to the feature point from which the erroneous correspondence is removed. Coefficient) is obtained (step S205).
Specifically, the control unit 210 substantially matches the display magnification of the subject in both the pair images (contains within a predetermined magnification range), and substantially matches the vertical coordinates of each feature point and the corresponding point. A correction coefficient for correcting (converting) the image A1 and the image B2 so as to be within the predetermined value is obtained using a known geometric technique.

Subsequently, the control unit 210 (first correction unit 12D) corrects (converts) the images A1 and B1 using the obtained correction coefficient (step S206). It is assumed that the image A2 and the image B2 are generated by this correction process. Further, the corrected image A2 and image B2 are stored in the image memory 230, for example. By the processing in step S206, as shown in FIG. 6B, the heights of the epipolar lines at the corresponding points in the image A2 and the image B2 are substantially the same. Therefore, it is possible to create an accurate three-dimensional model from the images A2 and B2.
Of course, instead of the images A1 and B1, the images A2 and B2 may be generated from the images A0 and B0. In general, image conversion is less likely to be performed once than image conversion is performed twice, so this is preferable depending on the situation.

  Returning to FIG. 4, when the image correction process (step S <b> 108) is completed, the control unit 210 (three-dimensional model generation unit 13) starts from the corrected pair images (image A <b> 2 and image B <b> 2) stored in the image memory 230. A three-dimensional model generation process for generating a three-dimensional model is executed (step S109).

  Here, the three-dimensional model generation process will be described with reference to the flowchart shown in FIG.

  First, the control unit 210 extracts feature point candidates (step S301). For example, the control unit 210 performs corner detection on the image A2. In corner detection, a point at which a corner feature amount such as Harris is maximized within a predetermined radius and not less than a predetermined threshold is selected as a corner point. Therefore, points having characteristics with respect to other points such as the tip of the subject are extracted as feature points.

  Subsequently, when the process of step S301 is completed, the control unit 210 performs stereo matching and searches the image B2 for a point (corresponding point) corresponding to the feature point of the image A2 (step S302). Specifically, the control unit 210 uses a template matching as a corresponding point if the similarity is greater than or equal to a predetermined threshold and the maximum (difference is equal to or less than the predetermined threshold). For template matching, various known techniques such as residual sum of absolute values (SAD), residual sum of squares (SSD), normalized correlation (NCC or ZNCC), and direction code correlation can be used.

  Subsequently, the control unit 210 performs disparity information (horizontal coordinate difference) of the corresponding points found in step S302, and camera information (field angle, baseline length, etc.) of the first imaging unit 100A and the second imaging unit 100B. Based on the above, position information of feature points is calculated (step S303). The generated position information of the feature points is stored in the storage unit 250, for example.

  Subsequently, the control unit 210 executes Delaunay triangulation based on the position information of the feature points calculated in step S303, and executes polygonization (step S304). The generated polygon information is stored in the storage unit 250, for example. When the control unit 210 completes the process of step S304, the control unit 210 ends the three-dimensional model generation process.

  Returning to FIG. 4, when the above three-dimensional model generation process (step S109) is completed, the control unit 210 displays the generated three-dimensional model data (step S110). Specifically, the control unit 210 converts the generated three-dimensional model data into confirmation two-dimensional data and displays it on the display unit 310. Then, the control unit 210 stores the 3D model data in the external storage unit 260 or the like (step S111), and returns to the process of step S101.

  As described above, according to the stereo camera 1 according to the present embodiment, pairs that are erroneously associated are removed from pairs corresponding to feature points extracted from a pair image that has been subjected to lens distortion correction and parallelization processing. . Then, based on the positional relationship of the pair corresponding to the feature points after removal, the pair images are corrected so that the heights of the epipolar lines of the corresponding points are substantially the same in both images of the pair images. Generate a model. Therefore, it is possible to create a three-dimensional model with high accuracy from a pair image captured using the zoom function.

  In the stereo camera 1 according to the present embodiment, the lens distortion correction and the parallelization process do not necessarily have to be performed with high accuracy. Therefore, it is not necessary to measure and store correction data (camera information) to be measured for each zoom state at a fine level for lens distortion correction and parallelization processing, and the measurement time can be shortened. Production efficiency can be improved.

  The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  For example, in the above-described embodiment, when the erroneous correspondence pair is removed in the image correction process (step S204 in FIG. 5), the pair having the difference in the vertical coordinate of the corresponding feature point equal to or greater than a predetermined threshold is removed as the erroneous correspondence. did. However, in addition to this condition, the erroneous correspondence set may be removed according to the following conditions A to C. By doing so, it becomes possible to remove the erroneous correspondence pair with higher accuracy, so that a more accurate three-dimensional model can be created.

(Condition A)
A rough distance between the stereo camera 1 and the subject is obtained using an autofocus function or the like, and a range in which parallax between the image A1 and the image B1 is to be obtained is obtained from the obtained distance. Then, when the parallax obtained from the horizontal coordinates of the corresponding feature point is not within this range, the set corresponding to the feature point is removed as an incorrect correspondence.

(Condition B)
For each feature point correspondence between the image A1 and the image B1, information (for example, Shape Context) indicating an arrangement corresponding to other feature points around is obtained. Then, when the similarity between pieces of information indicating the obtained arrangement is equal to or less than a predetermined value, the set corresponding to the feature point is removed as an incorrect correspondence.

(Condition C)
A correction coefficient is estimated with respect to the vertical coordinates of a set corresponding to each feature point by using a robust estimation method such as a RANSAC (RANdom SAmple Consensus) method or a minimum median method, and noise (mismatched set) is removed.

  In the above embodiment, the imaging magnification by the zoom function of the first imaging unit 100A and the second imaging unit 100B is controlled simultaneously, but the first imaging unit 100A and the second imaging unit 100B are individually controlled. The present invention is also applicable to a stereo camera that can change the imaging magnification.

  The number of cameras (imaging units) is not limited to two, and the present invention is also applicable to a camera that includes an arbitrary number of imaging units larger than that and creates a three-dimensional model from two or more captured images. Applicable.

  In the image correction process, it is not always necessary to perform the lens distortion correction and the parallelization process (step S201). In this case, the control unit 210 extracts a pair corresponding to the feature point directly from the pair image captured by the stereo camera 1 and removes the pair corresponding to the erroneous correspondence, and then the pair image so that the heights of the epipolar lines substantially match. May be corrected.

  It is also possible to make an existing stereo camera or the like function as the three-dimensional modeling apparatus according to the present invention. That is, the program as executed by the control unit 210 described above is applied to an existing stereo camera or the like, and the CPU or the like of the stereo camera or the like executes the program, so that the stereo camera or the like is It can function as a modeling device.

  Such a program distribution method is arbitrary, for example, a computer-readable recording medium such as a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), an MO (Magneto Optical Disk), or a memory card. It may be stored and distributed in a network, or distributed via a communication network such as the Internet.

  In this case, when the functions according to the present invention described above are realized by sharing an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion is stored in a recording medium or the like. May be.

  Although several embodiments of the present invention have been described so far, the present invention includes the invention described in the scope of claims and equivalents thereof. Hereinafter, the invention described in the scope of claims of the present application will be appended.

(Appendix 1)
Image acquisition means for acquiring a plurality of images obtained by imaging the same subject from different angles;
Feature point correspondence acquisition means for acquiring a feature point from each image acquired by the image acquisition means, and acquiring a set corresponding to the feature point by matching the acquired feature points;
Among the feature point correspondence sets acquired by the feature point correspondence acquisition unit, an erroneous correspondence removal unit that removes, as an erroneous correspondence, a pair whose vertical coordinate difference is a predetermined value or more;
The display magnification of the subject between the images acquired by the image acquisition unit based on the vertical coordinates of the set corresponding to the feature point acquired by the feature point correspondence acquisition unit and after the erroneous correspondence is removed by the erroneous correspondence removal unit First image correcting means for correcting the image so that the difference between the two and the difference between the vertical coordinates are within a predetermined range;
3D model generation means for generating a 3D model of the subject from each image corrected by the first image correction means;
A three-dimensional modeling apparatus comprising:

(Appendix 2)
The image acquisition means includes
Acquiring a plurality of images obtained by imaging the same subject from different angles together with zoom control information indicating a zoom state of each imaging device that has captured the plurality of images;
The three-dimensional modeling apparatus
Based on zoom control information acquired together with a plurality of images by the image acquisition means, further comprising a second image correction means for correcting the parallelism between the plurality of images and lens distortion;
The feature point correspondence acquisition means includes:
The feature acquisition unit acquires feature points from each image acquired by the image acquisition unit and corrected by the second image correction unit, and acquires a set of feature point correspondences by matching the acquired feature points;
The first image correcting means includes
Based on the vertical coordinates of the set corresponding to the feature points obtained by the feature point correspondence acquisition unit and after the erroneous correspondences are removed by the erroneous correspondence removal unit, the second image correction unit corrects the object between the images corrected by the second image correction unit. Further correcting the image corrected by the second image correction means so that the difference in display magnification and the difference in vertical coordinates are within a predetermined range;
The three-dimensional modeling apparatus according to Supplementary Note 1, wherein

(Appendix 3)
The feature point correspondence acquisition means includes:
Obtaining feature points having a feature quantity robust to the scale from each of the images, and obtaining a set corresponding to the feature points by associating the obtained feature points;
The three-dimensional modeling apparatus according to appendix 1 or 2, characterized in that:

(Appendix 4)
The erroneous correspondence removal unit obtains a parallax based on the horizontal coordinates of the feature point correspondence set acquired by the feature point correspondence acquisition unit, and if the parallax is not within a predetermined range, the feature point correspondence set is erroneously determined. Remove as correspondence,
The three-dimensional modeling apparatus according to any one of supplementary notes 1 to 3, characterized in that:

(Appendix 5)
The erroneous correspondence removing unit obtains information indicating an arrangement corresponding to another feature point around each of the feature point correspondences acquired by the feature point correspondence acquiring unit, and a similarity between the information indicating the obtained arrangements is predetermined. If the value is less than or equal to the value, the feature point correspondence set is removed as an erroneous correspondence.
The three-dimensional modeling apparatus according to any one of supplementary notes 1 to 4, characterized in that:

(Appendix 6)
The first image correcting means includes
Using the RANSAC method or the robust mediation method based on the minimum median method on the vertical coordinates of the feature point correspondence set obtained by the feature point correspondence acquisition unit and after the erroneous correspondence removal is removed by the erroneous correspondence removal unit Obtaining a correction coefficient such that the difference in display magnification of the subject and the difference in vertical coordinates between images are within a predetermined range, and correcting the image using the correction coefficient;
The three-dimensional modeling apparatus according to any one of appendices 1 to 5, characterized in that:

(Appendix 7)
A three-dimensional modeling method,
An image acquisition step of acquiring a plurality of images obtained by imaging the same subject from different angles;
A feature point correspondence obtaining step for obtaining a feature point from each image obtained in the image obtaining step and obtaining a set corresponding to the feature point by associating the obtained feature point;
Among the feature point correspondence sets acquired in the feature point correspondence acquisition step, an erroneous correspondence removal step of removing a pair whose vertical coordinate difference is a predetermined value or more as an erroneous correspondence;
Display magnification of the subject between the images acquired in the image acquisition step based on the vertical coordinates of the set corresponding to the feature point after the erroneous correspondence is removed in the erroneous correspondence removal step acquired in the feature point correspondence acquisition step A first image correction step for correcting the image so that the difference between the first and second coordinates is within a predetermined range;
A three-dimensional model generation step of generating a three-dimensional model of the subject from each image corrected in the first image correction step;
A three-dimensional modeling method comprising:

(Appendix 8)
On the computer,
An image acquisition function that acquires multiple images of the same subject taken from different angles,
A feature point correspondence acquisition function for acquiring feature points from each image acquired by the image acquisition function and acquiring a set corresponding to the feature points by associating the acquired feature points;
An erroneous correspondence removal function for removing a pair whose vertical coordinate difference is equal to or greater than a predetermined value from the feature point correspondence set acquired by the feature point correspondence acquisition function,
Display magnification of the subject between the images acquired by the image acquisition function based on the vertical coordinates of the set corresponding to the feature point obtained by the feature point correspondence acquisition function and after the erroneous correspondence removal by the erroneous correspondence removal function A first image correction function for correcting the image so that a difference between the first and second coordinates is within a predetermined range;
A three-dimensional model generation function for generating a three-dimensional model of the subject from each image corrected by the first image correction function;
A program characterized by realizing.

DESCRIPTION OF SYMBOLS 1 ... Stereo camera, 11 ... Image acquisition part, 12 ... Image correction part, 12A ... 2nd correction part, 12B ... Feature point corresponding | compatible extraction part, 12C ... Miscorresponding removal part, 12D ... 1st correction part, 13 ... Three-dimensional Model generator

Claims (8)

Image acquisition means for acquiring a plurality of images obtained by imaging the same subject from different angles;
Feature point correspondence acquisition means for acquiring a feature point from each image acquired by the image acquisition means, and acquiring a set corresponding to the feature point by matching the acquired feature points;
Among the feature point correspondence sets acquired by the feature point correspondence acquisition unit, an erroneous correspondence removal unit that removes, as an erroneous correspondence, a pair whose vertical coordinate difference is equal to or greater than a predetermined value;
The display magnification of the subject between the images acquired by the image acquisition unit based on the vertical coordinates of the set corresponding to the feature point acquired by the feature point correspondence acquisition unit and after the erroneous correspondence is removed by the erroneous correspondence removal unit First image correcting means for correcting the image so that the difference between the two and the difference between the vertical coordinates are within a predetermined range;
3D model generation means for generating a 3D model of the subject from each image corrected by the first image correction means;
A three-dimensional modeling apparatus comprising: The image acquisition means includes
Acquiring a plurality of images obtained by imaging the same subject from different angles together with zoom control information indicating a zoom state of each imaging device that has captured the plurality of images;
The three-dimensional modeling apparatus
Based on zoom control information acquired together with a plurality of images by the image acquisition means, further comprising a second image correction means for correcting the parallelism between the plurality of images and lens distortion;
The feature point correspondence acquisition means includes:
The feature acquisition unit acquires feature points from each image acquired by the image acquisition unit and corrected by the second image correction unit, and acquires a set of feature point correspondences by matching the acquired feature points;
The first image correcting means includes
Based on the vertical coordinates of the set corresponding to the feature points obtained by the feature point correspondence acquisition unit and after the erroneous correspondences are removed by the erroneous correspondence removal unit, the second image correction unit corrects the object between the images corrected by the second image correction unit. Further correcting the image corrected by the second image correction means so that the difference in display magnification and the difference in vertical coordinates are within a predetermined range;
The three-dimensional modeling apparatus according to claim 1. The feature point correspondence acquisition means includes:
Obtaining feature points having a feature quantity robust to the scale from each of the images, and obtaining a set corresponding to the feature points by associating the obtained feature points;
The three-dimensional modeling apparatus according to claim 1 or 2, characterized in that The erroneous correspondence removal unit obtains a parallax based on the horizontal coordinates of the feature point correspondence set acquired by the feature point correspondence acquisition unit, and if the parallax is not within a predetermined range, the feature point correspondence set is erroneously determined. Remove as correspondence,
The three-dimensional modeling apparatus according to any one of claims 1 to 3, wherein The erroneous correspondence removing unit obtains information indicating an arrangement corresponding to another feature point around each of the feature point correspondences acquired by the feature point correspondence acquiring unit, and a similarity between the information indicating the obtained arrangements is predetermined. If the value is less than or equal to the value, the feature point correspondence set is removed as an erroneous correspondence.
The three-dimensional modeling apparatus according to any one of claims 1 to 4, wherein the three-dimensional modeling apparatus is characterized in that: The first image correcting means includes
Using the RANSAC method or the robust mediation method based on the minimum median method on the vertical coordinates of the feature point correspondence set obtained by the feature point correspondence acquisition unit and after the erroneous correspondence removal is removed by the erroneous correspondence removal unit Obtaining a correction coefficient such that the difference in display magnification of the subject and the difference in vertical coordinates between images are within a predetermined range, and correcting the image using the correction coefficient;
The three-dimensional modeling apparatus according to any one of claims 1 to 5, wherein A three-dimensional modeling method,
An image acquisition step of acquiring a plurality of images obtained by imaging the same subject from different angles;
A feature point correspondence obtaining step for obtaining a feature point from each image obtained in the image obtaining step and obtaining a set corresponding to the feature point by associating the obtained feature point;
Among the feature point correspondence sets acquired in the feature point correspondence acquisition step, an erroneous correspondence removal step of removing a pair whose vertical coordinate difference is a predetermined value or more as an erroneous correspondence;
Display magnification of the subject between the images acquired in the image acquisition step based on the vertical coordinates of the set corresponding to the feature point after the erroneous correspondence is removed in the erroneous correspondence removal step acquired in the feature point correspondence acquisition step A first image correction step for correcting the image so that the difference between the first and second coordinates is within a predetermined range;
A three-dimensional model generation step of generating a three-dimensional model of the subject from each image corrected in the first image correction step;
A three-dimensional modeling method comprising: On the computer,
An image acquisition function that acquires multiple images of the same subject taken from different angles,
A feature point correspondence acquisition function for acquiring feature points from each image acquired by the image acquisition function and acquiring a set corresponding to the feature points by associating the acquired feature points;
An erroneous correspondence removal function for removing a pair whose vertical coordinate difference is equal to or greater than a predetermined value from the feature point correspondence set acquired by the feature point correspondence acquisition function,
Display magnification of the subject between the images acquired by the image acquisition function based on the vertical coordinates of the set corresponding to the feature point obtained by the feature point correspondence acquisition function and after the erroneous correspondence removal by the erroneous correspondence removal function A first image correction function for correcting the image so that a difference between the first and second coordinates is within a predetermined range;
A three-dimensional model generation function for generating a three-dimensional model of the subject from each image corrected by the first image correction function;
A program characterized by realizing.

Images