JP2009025225A - Three-dimensional imaging apparatus and control method for the same, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make possible to obtain mapping images simply and with high precision in consideration of hidden dots. <P>SOLUTION: By synchronizing driving of an imaging section 2A and an imaging section 2B for obtaining range images from a synchronous controller 20, range images and two-dimensional images are obtained. At a correspondent relationship calculating section 30 correspondent relationship between the pixels on the range images and the pixels on the two-dimensional images is computed. On this occasion, at a hidden dot detecting section 31 a hidden dot on a subject faceable from the imaging section 2A but unfaceable from the imaging section 2B is detected on the range images and the two-dimensional images. At a mapping section 32, based on the correspondence relationship, by mapping the two-dimensional images on the range images the mapping images are formed so as to make the hidden dot to be visually. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention provides a computer with a stereoscopic imaging apparatus, a stereoscopic imaging apparatus control method, and a stereoscopic imaging apparatus control method for simultaneously acquiring a distance image representing a stereoscopic shape of a subject and a two-dimensional image representing the contrast and / or color of the subject. It relates to a program to be executed.

  In a photographing device that captures an image by photographing a subject, the distance from the photographing device to the subject is measured by measuring the time until ranging light such as near infrared rays emitted toward the subject is reflected by the subject and returns. A distance image representing a three-dimensional shape of a subject is created by measuring a distance. The method of measuring the distance to the subject (ranging) using the reflection of light in this way is called a TOF (Time Of Flight) method, and various methods for measuring the distance using the TOF method are proposed. (See Patent Documents 1 to 3).

  Also proposed is a method of irradiating a subject with laser light and obtaining a distance image based on the flight time from when the laser light is emitted until the reflected light is detected, and a visible image based on the intensity of the reflected light. (See Patent Document 4).

Furthermore, an imaging means for acquiring a distance image using the TOF method and an imaging means for acquiring a two-dimensional image that is a visible image of the subject are provided, and the two-dimensional image is mapped to the distance image. A method for creating a mapping image representing a three-dimensional model has also been proposed (see Patent Document 5). In the method described in Patent Document 5, the correspondence between each pixel on the distance image and each pixel on the two-dimensional image is obtained in consideration of the parallax between the two imaging units, and the distance image is obtained based on this correspondence. Since the two-dimensional image is mapped, pixels corresponding to each other on the subject can be overlapped with each other, and thereby a mapping image can be created without causing a shift.
Japanese translation of PCT publication No. 2003-510561 Special table 2003-532122 gazette JP 2004-294420 A JP 2003-281686 A Japanese Patent Laying-Open No. 2005-87468

  When a subject is photographed using two imaging means as described in Patent Literature 5, the distance imaging acquisition means can face the subject on the subject due to the difference in position of the two imaging means. There may be a hidden point that cannot be seen from the imaging means for acquiring a two-dimensional image. As described above, regarding a hidden point, when a two-dimensional image is mapped to a distance image, two-dimensional shading information (that is, luminance information) should not exist.

  However, since the method described in Patent Document 5 does not consider hidden points at all, when a two-dimensional image is mapped to a distance image, pixel values of pixels that do not actually correspond are mapped onto the distance image. It will be done.

  For this reason, it is conceivable to arrange a half mirror on the optical path of the imaging means for acquiring the distance image and branch the reflected light into two to acquire the distance image and the two-dimensional image at the same time. However, in order to realize this, since it is necessary to position and arrange the two image pickup means and the half mirror with high accuracy, it is very difficult to adjust the position.

  The present invention has been made in view of the above circumstances, and an object thereof is to allow a mapping image to be easily and accurately acquired in consideration of hidden points.

The stereoscopic imaging device according to the present invention is configured to first capture distance data representing a stereoscopic shape of the subject by imaging the reflected light of the distance measuring light by the subject by irradiating the subject with the distance measuring light. Imaging means,
Second imaging means for obtaining image data representing a two-dimensional image representing the brightness of the subject arranged at a position different from the first imaging means;
Synchronization control means for synchronizing the driving of the first and second imaging means to cause the first and second imaging means to acquire the data for the distance image and the image data representing the two-dimensional image; ,
Distance image generation means for generating a distance image representing the three-dimensional shape of the subject based on the data for the distance image;
Correspondence calculation means for calculating the correspondence between the pixels on the distance image and the pixels on the two-dimensional image;
On the subject, a hidden point detecting means for detecting a hidden point that can be seen from the first imaging means but cannot be seen from the second imaging means;
Mapping means for mapping the two-dimensional image to the distance image so as to visually recognize the hidden point based on the correspondence relationship and generating a mapping image is provided.

  In the stereoscopic imaging apparatus according to the present invention, the hidden point detecting means calculates an outward normal vector for each pixel on the distance image, and each pixel on the distance image from the second imaging means. When the inner product of the normal vector and the line-of-sight vector is equal to or greater than a predetermined threshold for the target pixel on the distance image, the target pixel is detected as the hidden point. It may be a means.

  In the stereoscopic imaging apparatus according to the present invention, the hidden point detection unit calculates a line-of-sight vector that anticipates each pixel on the distance image from the second imaging unit, and for the target pixel on the distance image, Projecting the line-of-sight vector that looks into the target pixel on a distance image, and on the projection line corresponding to the projected line-of-sight vector on the distance image, the distance relationship between the line-of-sight vector and the projection line corresponding pixel on the projection line In other words, when at least one projection line-corresponding pixel is closer to the second imaging unit than the line-of-sight vector, the pixel of interest may be detected as the hidden point.

  In the stereoscopic imaging apparatus according to the present invention, the operation mode of the first imaging unit is changed to an operation mode similar to that of the second imaging unit, and a predetermined chart is provided to the first and second imaging units. And a calibration means for performing a calibration process for obtaining geometric information of the first and second imaging means based on the first and second chart images obtained by the imaging. It may be provided.

In this case, the first imaging means is provided with a filter that transmits only light in the wavelength range of the distance measuring light so that it can be taken in and out of the optical axis of the first imaging means,
The calibration means may be means for retracting the filter from the optical axis when the operation mode of the first imaging means is changed to an operation mode similar to that of the second imaging means.

According to the control method of the stereoscopic imaging apparatus according to the present invention, the first imaging unit irradiates the subject with the distance measuring light and images the reflected light of the distance measuring light by the subject to represent the three-dimensional shape of the subject. When acquiring distance image data and acquiring image data representing a two-dimensional image representing the brightness of the subject by the second imaging means arranged at a position different from the first imaging means, the first imaging means And synchronizing the driving of the second imaging means to obtain the data for the distance image and the image data representing the two-dimensional image,
Based on the data for the distance image, generate a distance image representing the three-dimensional shape of the subject,
Calculating a correspondence between a pixel on the distance image and a pixel on the two-dimensional image;
On the subject, a hidden point that can be seen from the first image pickup means but cannot be seen from the second image pickup means is detected,
Based on the correspondence relationship, the mapping image is generated by mapping the two-dimensional image to the distance image so that the hidden point can be visually recognized.

  In addition, you may provide as a program for making a computer perform the control method of the three-dimensional imaging device by this invention.

  According to the present invention, the driving of the first imaging unit and the second imaging unit is synchronized to acquire the data for the distance image and the image data representing the two-dimensional image, and based on the data for the distance image, A distance image representing a three-dimensional shape is generated. Then, the correspondence between the pixels on the distance image and the pixels on the two-dimensional image is calculated, and the hidden image that can be seen from the first imaging means but cannot be seen from the second imaging means on the subject. A point is detected. Then, the mapping image is created so that the hidden point can be visually recognized. Therefore, an accurate mapping image can be created in consideration of hidden points without complicating the configuration of the apparatus.

  Further, an outward normal vector is calculated for each pixel on the distance image, a line-of-sight vector that anticipates each pixel on the distance image from the second imaging unit is calculated, and the normal vector is calculated for the target pixel on the distance image. When the inner product of the line vector and the line-of-sight vector is a positive value, the hidden point can be easily detected by detecting the target pixel as the hidden point.

  In addition, a line-of-sight vector that anticipates each pixel on the distance image is calculated from the second imaging unit, and a line-of-sight vector that anticipates the pixel of interest on the distance image is projected for the pixel of interest on the distance image. When the distance relationship between the line-of-sight vector and the projection line corresponding pixel on the projection line is obtained on the projection line corresponding to the line-of-sight vector, and the at least one projection line-corresponding pixel is closer to the second imaging unit than the line-of-sight vector, By detecting as a hidden point, the hidden point can be detected with high accuracy although the amount of calculation increases.

  In addition, the operation mode of the first imaging unit is changed to the operation mode similar to that of the second imaging unit, the first and second imaging units are caused to capture a predetermined chart, and the first and second acquired by the imaging are performed. The calibration of the first and second imaging means is performed in the stereoscopic imaging apparatus of the present invention by performing a calibration process for acquiring geometric information of the first and second imaging means based on the two chart images. Can be performed.

  Here, the first imaging hand is provided with a filter that transmits only light in the wavelength range of the ranging light so that it can be taken in and out of the optical axis of the first imaging means, and the operation mode of the first imaging means Is changed to an operation mode similar to that of the second image pickup means, the image required for calibration can be easily acquired in the first image pickup means by retracting the filter from the optical axis.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a front perspective view showing an external configuration of a stereoscopic imaging apparatus according to an embodiment of the present invention, and FIG. 2 is a rear perspective view thereof. As shown in FIGS. 1 and 2, the stereoscopic imaging apparatus 1 according to the present embodiment includes a first imaging unit 2 </ b> A and a second imaging unit 2 </ b> B on the front side thereof. The first imaging unit 2A performs photographing for acquiring a distance image, and an LED group 3 composed of a large number of LEDs for irradiating distance measuring light composed of infrared light toward a subject around the first imaging unit 2A. Is provided. The second imaging unit 2B performs imaging for acquiring a two-dimensional image that is a normal image.

  On the top surface of the stereoscopic imaging apparatus 1, a power switch 4, a distance measurement / calibration switching switch 5, and a release button 6 for performing photographing are provided. Further, on the back side, a monitor 7 made of a two-dimensional image acquired by photographing, a mapping image, a liquid crystal for performing various displays, and a display changeover switch 8 for switching the display of the monitor 7 are provided.

  The right side as viewed from the back side is provided with an output terminal 9 for connecting to an external monitor, and a USB terminal 10 for outputting image data acquired by photographing and for external control input. .

  FIG. 3 is a schematic block diagram showing the internal configuration of the stereoscopic imaging apparatus according to the present embodiment.

  The first and second imaging units 2A and 2B include lenses 10A and 10B, diaphragms 11A and 11B, shutters 12A and 12B, CCDs 13A and 13B, analog front ends (AFE) 14A and 14B, and A / D conversion units 15A and 15B. Each is provided.

  The lenses 10A and 10B are composed of a plurality of functional lenses such as a focus lens for focusing on a subject and a zoom lens for realizing a zoom function, and their positions are adjusted by a lens driving unit (not shown).

  In the diaphragms 11A and 11B, the diaphragm diameter is adjusted based on the diaphragm value data obtained by the AE process by a diaphragm driving unit (not shown).

  The shutters 12A and 12B are mechanical shutters and are driven by a shutter driving unit (not shown) according to the shutter speed obtained by the AE process.

  The CCDs 13A and 13B have a photoelectric surface in which a large number of light receiving elements are two-dimensionally arranged, and subject light is imaged on the photoelectric surface and subjected to photoelectric conversion to obtain an analog photographing signal. A color filter in which R, G, and B color filters are regularly arranged is disposed on the front surface of the CCD 13B. FIG. 4 is a diagram showing the configuration of the CCD. Since the CCDs 13A and 13B have the same configuration, only the configuration of the CCD 13A will be described here.

  As shown in FIG. 4, the CCD 13A has a large number of light receiving elements 40 arranged two-dimensionally. In addition, the CCD 13A receives the electrical signals obtained by the light receiving elements 40 and the vertical transfer paths 41 composed of a plurality of vertical transfer CCDs for transferring the electrical signals obtained by the light receiving elements 40 for one line in the vertical direction. A horizontal transfer path 42 composed of a plurality of horizontal transfer CCDs for transferring one line in the horizontal direction and a read amplifier 43 for reading signal charges are provided.

  Each light receiving element 40 includes a photoelectric conversion element 44, a main accumulation unit 45, a main accumulation unit gate 46, a sub accumulation unit 47, a sub accumulation unit gate 48 and a transfer path gate 49.

  The main storage unit 45 is a storage unit for acquiring the signal charge output from the photoelectric conversion element 44 from the main storage unit gate 46 and temporarily storing it. The signal charge accumulated in the main accumulation unit 45 is transferred to the vertical transfer path 41 via the transfer path gate 49 by a transfer signal given by a pulse generator (not shown).

  The sub storage unit 47 is a storage unit for acquiring unnecessary charges from the sub storage unit gate 48 and temporarily storing them. The signal charge temporarily stored in the sub storage unit 47 is discarded to the substrate of the CCD 13A when the CCD 13A is reset.

  The detailed driving of the CCDs 13A and 13B will be described later.

  The AFEs 14A and 14B perform a process for removing noise of the analog signal and a process for adjusting the gain of the analog signal (hereinafter referred to as an analog process) on the analog imaging signals output from the CCDs 13A and 13B.

  The A / D converters 15A and 15B convert the analog signals processed by the AFEs 14A and 14B into digital signals. The image data acquired by the CCD 13B of the imaging unit 2B and converted into a digital signal is data having density values of R, G, and B for each pixel.

  In addition, the imaging unit 2A includes a visible light removal filter 16A for removing visible light on the front surface of the lens 10A and transmitting light in a wavelength region of infrared light. The visible light removing filter 16A is arranged so as to be freely put in and out of the optical axis of the imaging unit 2A by the filter driving unit 17. When the operation mode of the stereoscopic imaging apparatus 1 is set to the distance measurement mode, the visible light removal filter 16A is disposed on the optical axis, and when the calibration mode is set, the visible light removal filter 16A. Is retracted from the optical axis.

  The imaging unit 2 </ b> A includes a modulation unit 18 for modulating the LED group 3. Further, a drive control unit 19 for controlling the drive of the CCD 13A, the AFE 14A, the filter drive unit 17 and the modulation unit 18 is provided. As will be described later, the drive control unit 19 switches the CCD 13A, the AFE 14A, the filter drive unit 17, and the modulation unit 18 to a mode in which the operation mode of the stereoscopic imaging apparatus 1 is switched to the distance measurement mode and the calibration mode. Control accordingly.

  The imaging unit 2B includes an infrared light removal filter 16B that transmits light in the visible wavelength range and removes light in the infrared wavelength range in front of the lens 10B. The infrared light removal filter 16B is fixed on the optical axis of the imaging unit 2B.

  Since the stereoscopic imaging device 1 according to the present embodiment acquires a distance image and a two-dimensional image at the same time, the synchronization control unit 20 synchronizes the imaging units 2A and 2B to shoot a subject.

  The stereoscopic imaging device 1 also includes a shooting control unit 22, an image processing unit 23, a compression / decompression processing unit 24, a frame memory 25, a media control unit 26, an internal memory 27, and a display control unit 28.

  The imaging control unit 22 includes an AF processing unit and an AE processing unit. The AF processing unit determines the focal length of the lenses 10A and 10B, and outputs the instruction to a lens driving unit (not shown). The AE processing unit determines the aperture value and the shutter speed, and outputs an instruction to the driving unit (not shown).

  The image processing unit 23 performs processing for adjusting white balance, image quality correction processing such as tone correction, sharpness correction, and color correction for digital image data representing a two-dimensional image acquired by the imaging unit 2B, CCD- YC processing is performed to convert the RAW data into YC data composed of Y data that is a luminance signal, Cb data that is a blue color difference signal, and Cr data that is a red color difference signal.

  The compression / decompression processing unit 24 compresses the image data of the two-dimensional image that has been corrected and converted by the image processing unit 23 and the image data of the mapping image generated as described later, such as JPEG. Perform compression processing in the format to generate an image file. A tag storing incidental information such as shooting date and time is added to the image file based on the Exif format or the like.

  The frame memory 25 is used when performing various processes including the above-described various image processes on the distance image data acquired by the imaging unit 2A and the image data representing the two-dimensional image acquired by the imaging unit 2B. This is a working memory.

  The media control unit 26 accesses the recording medium 37 and controls writing and reading of image files such as two-dimensional images and mapping images.

  The internal memory 27 stores various constants set in the stereoscopic imaging apparatus 1, a program executed by the CPU 34, and the like.

  The display control unit 28 is for displaying the image data stored in the frame memory 25 on the monitor 7 and for displaying the two-dimensional image and the mapping image recorded on the recording medium 37 on the monitor 7. .

  The stereoscopic imaging device 1 includes a distance calculation unit 29, a correspondence calculation unit 30, a hidden point detection unit 31, a mapping unit 32, a calibration unit 33, and a CPU 34.

  The distance calculation unit 29 calculates the distance from the stereoscopic imaging device 1 at each point on the subject using the distance image data acquired by the imaging unit 2A to generate a distance image. The generation of the distance image will be described later.

  The correspondence calculation unit 30 calculates the correspondence between each pixel on the distance image and each pixel on the two-dimensional image based on the parallax of the imaging units 2A and 2B. The calculation of the correspondence relationship will be described later.

  The hidden point detection unit 31 detects a hidden point on the subject that can be seen from the imaging unit 2A but cannot be seen from the imaging unit 2B. The detection of hidden points will also be described later.

  The mapping unit 32 maps the two-dimensional image to the distance image based on the correspondence calculated by the correspondence calculation unit 30, and generates a mapping image that is a three-dimensional shape model.

  The calibration unit 33 performs a calibration process for acquiring geometric information of the imaging unit 2A and the imaging unit 2B when the operation mode of the stereoscopic imaging device 1 is set to the calibration mode.

  The CPU 34 controls each unit of the stereoscopic imaging device 1 in accordance with a signal from the input unit 35 including the release button 6.

  The data bus 36 is connected to each unit constituting the stereoscopic imaging apparatus 1 and the CPU 34, and exchanges various data and various information in the stereoscopic imaging apparatus 1.

  Next, processing performed in the present embodiment will be described. FIG. 5 is a flowchart showing processing performed in the present embodiment. Note that the calibration processing of the imaging units 2A and 2B has been completed, and the geometric information of the imaging units 2A and 2B is stored in the internal memory 27.

  The user operates the distance measurement / calibration switch 5 to set the operation mode of the stereoscopic imaging apparatus 1 to the distance measurement mode, and operates the release button 6 to issue a shooting instruction, whereby the CPU 34 starts processing. Then, in response to an instruction from the CPU 34, the drive control unit 19 drives the filter driving unit 17, and arranges the visible light removal filter 16A on the optical axis of the imaging unit 2A (step ST1). If the visible light removal filter 16A has already been arranged on the optical axis of the imaging unit 2A, the process of step ST1 is omitted. Subsequently, the synchronization control unit 20 outputs a synchronization signal to the imaging units 2A and 2B according to an instruction from the CPU 34 (step ST2). As a result, the imaging units 2A and 2B start the shooting operation.

  FIG. 6 is a diagram illustrating the timing of operations of the imaging units 2A and 2B in the distance measurement mode. First, the operation of the imaging unit 2A will be described. When the synchronization signal is input to the imaging unit 2A, the CCD 13A is reset by the reset signal, and the modulation unit 18 performs the modulated light emission of the LED group 3 (step ST3). Specifically, as shown in FIG. 6, modulated light emission is performed three times by sinusoidal modulation of a predetermined frequency whose phase is shifted by 90 degrees. In FIG. 6, the modulated light emission of the LED group 3 is indicated as LED light emission. Here, the period of three times of modulated light emission is defined as period T1, period T2, and period T3, respectively. Also, the CCD 13A is reset by a reset signal at the start of the periods T1, T2, and T3.

  The light emitted from the LED group 3 is reflected by the subject, and the reflected light is received by the imaging unit 2A. The reflected light received by each light receiving element of the CCD 13A has a phase difference with the emitted light according to the distance from the stereoscopic imaging device 1 to the subject. The reflected light incident on the CCD 13A is opened in the main accumulating portion gate 46 of the CCD 13A in the first half of the period of the emitted light and opened in the main accumulating portion 45, and the sub accumulating portion gate 48 in the CCD 13A is opened in the latter half. Respectively. Here, at the start of the modulated light emission period, the signal charge accumulated in the sub accumulation unit 47 by the reset signal is discarded to the substrate (not shown) of the CCD 13A.

  In the period T1, the signal charge of the reflected light in the period of 0 to 180 degrees (0 to π) in the sine wave of the emitted light is accumulated in the main accumulation unit 45, and in the period T2, the signal charge of 90 degrees in the sine wave of the emitted light. The signal charges of the reflected light in the period of ˜270 degrees (π / 2 to 3π / 2) are accumulated in the main accumulation unit 45, and in the period T3, the signal charges are 180 degrees to 360 degrees (π to 2π) in the sine wave of the emitted light. The signal charge of the reflected light during the period is accumulated in the main accumulation unit 45.

  Then, after a predetermined number of times (four times in FIG. 6, actually 100 to 1000 times) in each of the periods T1, T2, and T3, the accumulated signal charge is CCD transferred to the AFE 14A by a transfer signal. Specifically, the signal charge accumulated in the main accumulation unit 45 is transferred to the vertical transfer path 41 via the transfer path gate 49. Next, the signal charges transferred to the vertical transfer path 41 are transferred to the horizontal transfer path 42 for each pixel. Then, the transferred signal charge is transferred to the read amplifier 43 through the horizontal transfer path 42. The signal charge is transferred from the read amplifier 43 to the AFE 14A.

  The signal charge transferred to the AFE 14A is subjected to analog processing, further converted into digital data by the A / D converter 15A, and transferred to the frame memory 25. Thereby, the distance image data is stored in the frame memory 25. As described above, the reception and transfer of the reflected light and the storage of the data in the frame memory 25 are the generation of the data for the distance image (step ST4).

  Next, the operation of the imaging unit 2B will be described. When the synchronization signal is input to the imaging unit 2B, the CCD 13B is reset by the reset signal, and exposure is performed for a predetermined time. As a result, the signal charge is accumulated in the main accumulation unit 45 of the CCD 13B. Then, the accumulated signal charge is CCD transferred to the AFE 14B by the transfer signal.

  The signal charges transferred to the AFE 14B are subjected to analog processing, further converted into digital image data by the A / D converter 15B, and transferred to the frame memory 25. Then, the image processing unit 23 performs necessary image processing on the image data. As a result, image data representing a two-dimensional image subjected to image processing is stored in the frame memory 25. As described above, reception, transfer of reflected light, storage of image data in the frame memory 25 and image processing are defined as generation of a two-dimensional image (step ST5).

  Note that the processing of step ST4 and step ST5 may be performed in parallel or sequentially. In the present embodiment, it is performed in parallel.

  Following step ST4, the distance calculation unit 29 calculates the distance from the stereoscopic imaging device 1 to the subject for each pixel (step ST6), and generates a distance image (step ST7). Hereinafter, calculation of the distance and generation of the distance image will be described. FIG. 7 is a diagram for explaining calculation of a distance and generation of a distance image.

  As shown in FIG. 7, since the emitted light from the LED group 3 is sinusoidally modulated, the intensity thereof periodically changes as shown by the curve K1, thereby reflecting the light received by the CCD 13A of the imaging unit 2A. The amount of light also changes periodically as shown by the curve K2. In addition, between the emitted light from the LED group 3 and the reflected light of the emitted light from the subject, a phase delay (phase difference) of φ [rad] occurs according to the distance from the stereoscopic imaging device 1 to the subject. Since the phase difference φ corresponds to the flight time of light, the distance from the stereoscopic imaging apparatus 1 to the subject can be obtained by obtaining the phase difference φ.

  Here, when the reflected light is delayed in phase difference φ from the emitted light, the distance L [m] from the stereoscopic imaging device 1 to the subject is expressed by the following equation (1).

L = c · φ / (4πF) (1)
However, c: speed of light, F: modulation frequency [Hz].

  The phase difference φ can be calculated by using the received light amount in the period T1, T2, T3 of the emitted light described above. Here, in the period T1, the signal charge of the reflected light in the period of 0 to 180 degrees (0 to π) in the sine wave of the emitted light is 90 to 270 degrees (π / The signal charge of the reflected light in the period of 2 to 3π / 2) is equal to the signal charge of the reflected light in the period of 180 degrees to 360 degrees (π to 2π) in the sine wave of the emitted light in the period T3. Accumulated. For this reason, the amount of received light in each period corresponds to the signal value of each pixel acquired in each of the periods T1, T2, and T3.

  Here, it is assumed that the received light amount of the curve K2 obtained in the phase of 0 to 90 degrees, 90 degrees to 180 degrees, 180 degrees to 270 degrees, 270 degrees to 360 degrees in the curve K1 is x1, x2, x3, x4. . In FIG. 7, for the sake of explanation, the periods T1, T2, T3 and the received light amounts x1, x2, x3, x4 are shown only for one cycle of the emitted light of the LED group 3. Further, the phase difference φ does not change during the calculation of the received light amounts x1, x2, x3, x4 (that is, the distance to the subject does not change), and the subject reflectance and the external light intensity do not change. And Accordingly, the received light amounts [T1], [T2], and [T3] in the periods T1, T2, and T3 are expressed by the following formulas (2) to (4) where the amplitude of the received light is A and the external light intensity is B. be able to.

[T1] = x1 + x2 = (2A · cos φ + π · (A + B)) / 2πF (2)
[T2] = x2 + x3 = (2A · sin φ + π · (A + B)) / 2πF (3)
[T3] = x3 + x4 = (− 2A · cos φ + π · (A + B)) / 2πF (4)
Thereby, the relationship of the following formula | equation (5)-(7) is materialized.

x1−x3 = [T1] − [T2] = − 2√2A · sin (φ−π / 4) / 2πF (5)
x2−x4 = [T2] − [T3] = 2√2A · cos (φ−π / 4) / 2πF (6)
-{(X1-x3) / (x2-x4)}
= {([T2]-[T1]) / ([T2]-[T3])}
= Tan (φ-π / 4) (7)
Therefore, the phase difference φ can be calculated by the following equation (8).

φ = tan−1 {([T2] − [T1]) / ([T2] − [T3])} + π / 4 (8)
The distance calculation unit 29 calculates the distance from the stereoscopic imaging device 1 to the subject for each pixel of the CCD 13A by using Expression (1) and Expression (8). Then, a distance image representing the three-dimensional shape of the subject is created using the calculated distance. Note that the pixel value of each pixel of the distance image has a three-dimensional value such as a two-dimensional position and distance on the image.

  Next, the correspondence calculation unit 30 calculates the correspondence between each pixel on the distance image and each pixel on the two-dimensional image (step ST8). FIG. 8 is a diagram for explaining the calculation of the correspondence. The imaging units 2A and 2B (a) satisfy the pinhole camera model (that is, the image has no distortion aberration), the image planes of the imaging units 2A and 2B (that is, the image planes of the CCDs 13A and 13B), and the imaging units. The straight lines s1 and s2 connecting the focal points O1 and O2 of 2A and 2B are orthogonal to the straight line s3 connecting the focal points O1 and O2 of the imaging units 2A and 2B, and the x direction in FIG. 8 of the image plane of the CCDs 13A and 13B is a straight line. It is assumed to be parallel to s3.

  Assuming that the point P0 on the subject is focused on the point G1 on the image plane of the CCD 13A and the point G2 on the image plane of the CCD 13B, the point G2 on the two-dimensional image corresponds to the point G1 on the distance image. It will be. The shift between the point G2 and the point G1, that is, the parallax occurs based on the difference in the positions of the imaging units 2A and 2B facing the point P0 on the subject. Here, assuming that the position corresponding to the point G1 of the CCD 13A on the CCD 13B is a point G1 ', the difference between the coordinates of the point G1' and the coordinates of the point G2 on the image plane of the CCD 13B is the parallax Δx. Δx is calculated by the following equation (9) using the base line length d, the focal length f of the imaging units 2A and 2B, and the distance L from the stereoscopic imaging device 1 to the subject.

Δx = d · f / L (9)
The base line length d and the focal length f are calculated by a calibration process described later. Actually, it is possible to accurately know information such as distortion, position and orientation of the CCDs 13A and 13B in the imaging units 2A and 2B by the calibration process, and therefore using these pieces of information, Correspondence with each pixel on the distance image can be obtained.

  Subsequently, the hidden point detection unit 31 detects a hidden point on the subject that can be seen from the imaging unit 2A but cannot be seen from the imaging unit 2B on the distance image and the two-dimensional image (step ST9). .

  FIG. 9 is a diagram for explaining a hidden point. As shown in FIG. 9, the point P0 on the subject H can face from the imaging unit 2A, but cannot face from the imaging unit 2B. For this reason, the point corresponding to the point P0 on the two-dimensional image acquired by the imaging unit 2B actually corresponds to the point P1 on the subject. Therefore, when a two-dimensional image is mapped on the distance image as will be described later, the pixel value of the point P1 on the two-dimensional image is mapped to the point P0 on the distance image, so that the pixel value of the point P0 is correct for the subject. It will not represent light and shade.

  The hidden point detection unit 31 first calculates a normal vector n at each pixel on the distance image in order to detect a hidden point on the two-dimensional image. FIG. 10 is a diagram for explaining the calculation of the normal vector. As shown in FIG. 10, when the coordinate of the target pixel [i, j] for obtaining the normal vector n is Pv [i, j], the normal vector n is expressed as follows using the coordinates of the pixels around the target pixel. It can be calculated by equation (10). Since Pv [i, j] is a three-dimensional coordinate, the normal vector n is a three-dimensional vector.

n = (Pv [i + 1, j] −Pv [i−1, j]) × (Pv [i, j + 1] −Pv [i, j−1]) (10)
Further, the hidden point detection unit 31 calculates the line-of-sight vector v of the imaging unit 2B with respect to the target pixel. The line-of-sight vector v can be calculated from the coordinates Pv [i, j] of the pixel of interest and the focal point coordinates of the imaging unit 2B obtained by the calibration process. The line-of-sight vector is also a three-dimensional vector.

  Then, the hidden point detection unit 31 calculates the inner product IP of the normal vector n and the line-of-sight vector v at the target pixel by the following equation (11).

IP = (n · v) / (| n | · | v |) (11)
Here, when the inner product IP is a positive value, the angle formed between the normal vector n and the line-of-sight vector v is less than 90 degrees, and thus the normal vector n is directed toward the line-of-sight vector v. . On the other hand, when the inner product is a negative value, the angle formed between the normal vector n and the line-of-sight vector v exceeds 90 degrees, and thus the normal vector is directed in the opposite direction to the line-of-sight vector v. Therefore, the hidden point detection unit 31 compares the inner product IP with a predetermined threshold value Th1 (Th1 ≧ 0, for example, Th1 = 0), and detects the target pixel as a hidden point when IP <Th1.

  By using such a method (referred to as the first method), hidden points can be detected at high speed. However, when the first method is used, as shown in FIG. 11, the point P0 at which the inner product IP is positive cannot be detected as a hidden point.

  Hereinafter, a second method for detecting hidden points will be described. First, the hidden point detection unit 31 projects the line of sight O2P0 from the focus O2 of the imaging unit 2B that acquires a two-dimensional image to the point P0 that is the target pixel on the distance image. FIG. 12 is a diagram for explaining the projection of the line of sight onto the distance image. Subsequently, the hidden point detection unit 31 extracts pixels (pixels indicated by diagonal lines in FIG. 12) on the distance image around the projected line of sight O2P0. Then, the coordinates of each pixel and the corresponding coordinates on the line of sight are compared in order from the pixel close to the imaging unit 2B (the pixel on the right side in FIG. 12) toward the target pixel, and either the line of sight or the pixel is on the imaging unit 2B side. Determine whether. Then, when there is a pixel closer to the imaging unit 2B than the line of sight, the target pixel is detected as a hidden point.

  Such a second method can accurately detect hidden points, but requires a long time for calculation. For this reason, it is preferable to switch between the first and second methods depending on which of accuracy and calculation time is important.

  Then, the mapping unit 32 maps the two-dimensional image to the distance image so that the hidden point can be visually recognized, and generates a mapping image that is a three-dimensional model (step ST10). Specifically, based on the correspondence between the pixel on the distance image and the pixel on the two-dimensional image calculated by the correspondence calculation unit 30, the pixel value of the pixel corresponding to the two-dimensional image for each pixel on the distance image A mapping image is generated by mapping. At this time, since the accurate pixel value cannot be mapped for the hidden point, the hidden point on the mapping image is mapped in gray as shown in FIG.

  Note that, as shown in FIG. 14, after mapping based on the correspondence, the brightness of the hidden points may be lowered as compared with other portions. Further, as shown in FIG. 15, a two-dimensional image may be arranged in the background of the mapping image, and the hidden points may be mapped as the corresponding background of the two-dimensional image.

  Then, the CPU 34 displays the generated mapping image on the monitor 7 (step ST11). Furthermore, it is determined whether or not a recording instruction has been given by the user (step ST12). If step ST12 is negative, the process returns to step ST2 and the processes after step ST2 are repeated. If step ST12 is affirmed, the mapping image is recorded on the recording medium 37 (step ST13), and the process is terminated. At this time, a two-dimensional image may be recorded on the recording medium 37 together with the mapping image.

  As described above, in the present embodiment, the distance image and the two-dimensional image are acquired by synchronizing the driving of the imaging units 2A and 2B, and the correspondence relationship between the pixel on the distance image and the pixel on the two-dimensional image is calculated. Further, a hidden point is detected, and a mapping image is created so that the hidden point can be visually recognized. Therefore, an accurate mapping image can be created in consideration of hidden points without complicating the configuration of the apparatus.

  Next, the calibration process will be described. FIG. 16 is a flowchart of the calibration process.

  The user operates the distance measurement / calibration switching switch 5 to set the stereoscopic imaging apparatus 1 to the calibration mode, and instructs the calibration process to be started by the CPU 34. In response to an instruction from the CPU 34, the drive control unit 19 drives the filter driving unit 17 to retract the visible light removing filter 16A from the optical axis of the imaging unit 2A (step ST21). If the visible light removal filter 16A has already been retracted from the optical axis of the imaging unit 2A, the process of step ST21 is omitted. Subsequently, the calibration chart shown in FIG. 17 is moved to the photographing range of the stereoscopic imaging apparatus 1 and arranged at a predetermined position (step ST22). Note that the movement and / or setting of the calibration chart may be performed automatically or manually.

  Next, the synchronization control unit 20 outputs a synchronization signal to the imaging units 2A and 2B according to an instruction from the CPU 34 (step ST23). As a result, the imaging units 2A and 2B start operating.

FIG. 18 is a diagram illustrating timing of operations of the imaging units 2A and 2B in the calibration mode. First, the operation of the imaging unit 2A will be described. When the synchronization signal is input to the imaging unit 2A, the CCD 13A is reset by the reset signal, and exposure is performed for a predetermined time. As a result, the signal charge is accumulated in the main accumulation unit 45 of the CCD 13A. Then, the accumulated signal charge is CCD transferred to the AFE 14A by the transfer signal. In the calibration mode, the image pickup unit 2A does not emit the light from the LED group 3 and does not accumulate in the sub accumulation unit. The image data is converted into digital image data by 15A and transferred to the frame memory 25. Then, the image processing unit 23 performs necessary image processing on the image data. As a result, the frame memory 25 stores image data representing a chart image that is a two-dimensional image of the calibration chart. As described above, the reception and transfer of the reflected light, the storage of the image data in the frame memory 25 and the image processing are the generation of the first chart image (step ST24). Note that since the color filter is not provided in the CCD 13A of the imaging unit 2A, the first chart image is a gray image.

  Next, the operation of the imaging unit 2B will be described. When the synchronization signal is input to the imaging unit 2B, the CCD 13B is reset by the reset signal, and exposure is performed for a predetermined time. As a result, the signal charge is accumulated in the main accumulation unit 45 of the CCD 13B. Then, the accumulated signal charge is CCD transferred to the AFE 14B by the transfer signal.

  The signal charges transferred to the AFE 14B are subjected to analog processing, further converted into digital image data by the A / D converter 15B, and transferred to the frame memory 25. Then, the image processing unit 23 performs necessary image processing on the image data. As a result, the frame memory 25 stores image data representing a chart image that is a two-dimensional image of the calibration chart. As described above, the reception of the reflected light, the transfer, the storage of the image data in the frame memory 25 and the image processing are the generation of the second chart image (step ST25).

  Note that the processing of step ST24 and step ST25 may be performed in parallel or sequentially. In this embodiment, it shall carry out sequentially.

  Next, the calibration unit 33 determines whether or not the generation of the first and second chart images has been performed a required number of times (step ST26), and if step ST26 is negative, the process returns to step ST22 to perform calibration. The chart is moved, and the processes after step ST23 are repeated.

  If step ST26 is affirmed, based on the first and second chart images, optical system distortion data of the imaging units 2A and 2B is generated (step ST27), and further stereo calibration data is generated (step ST28). . Specifically, distortions such as the lenses 10A and 10B and the CCDs 13A and 13B are generated as optical system distortion data, and information such as the baseline length, focal length, and orientation of the imaging units 2A and 2B is generated as stereo calibration data. .

  And the calibration part 33 preserve | saves the produced | generated optical system distortion data and stereo calibration data in the internal memory 27 (step ST29), and also arrange | positions the visible light removal filter 16A on an optical axis (step ST30), and performs a process. finish.

  As described above, the operation mode of the imaging unit 2A is changed to the same operation mode as that of the imaging unit 2B, the calibration charts are captured by the imaging units 2A and 2B, and the first and second two-dimensional images acquired by imaging. By performing the calibration process based on the above, the imaging units 2A and 2B can be calibrated in the stereoscopic imaging apparatus 1 according to the present embodiment.

  Here, since the imaging unit 2A includes the visible light removal filter 16A so that it can be taken in and out of the optical axis of the imaging unit 2A, the imaging unit 2A performs calibration by retracting the visible light removal filter 16A from the optical axis. The chart image required for the operation can be easily acquired.

  In the above-described embodiment, the display mode of the hidden point is not limited to that shown in FIGS. 13 to 15, and any method that allows visual recognition of the hidden point can be used.

  The stereoscopic imaging device 1 according to the embodiment of the present invention has been described above, but the computer is used as the distance calculation unit 29, the correspondence calculation unit 30, the hidden point detection unit 31, the mapping unit 32, and the calibration unit 33. A program that functions as a corresponding means and performs processing as shown in FIGS. 5 and 16 is also one embodiment of the present invention. A computer-readable recording medium in which such a program is recorded is also one embodiment of the present invention.

1 is a front perspective view showing an external configuration of a stereoscopic imaging apparatus according to an embodiment of the present invention. The rear side perspective view which shows the external appearance structure of the three-dimensional imaging device by embodiment of this invention Schematic block diagram showing the internal configuration of the stereoscopic imaging apparatus according to the present embodiment Diagram showing the structure of the CCD A flowchart showing processing performed in the present embodiment The figure which shows the timing of operation | movement of the imaging parts 2A and 2B in ranging mode Diagram for explaining distance calculation and distance image generation Diagram for explaining the calculation of correspondence Illustration for explaining hidden points Diagram for explaining normal vector calculation The figure which shows the case where a hidden point cannot be detected by the 1st method The figure for demonstrating projection of a gaze on a distance picture The figure which shows the aspect of a hidden point display on a mapping image (the 1) The figure which shows the aspect of a hidden point display on a mapping image (the 2) The figure which shows the aspect of a hidden point display on a mapping image (the 3) Flow chart of calibration process Diagram showing calibration chart The figure which shows the timing of operation | movement of the imaging parts 2A and 2B in a calibration mode

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo imaging device 2A, 2B Image pick-up part 3 LED group 7 Monitor 13A, 13B CCD
16A Visible light removing filter 16B Infrared light removing filter 17 Filter driving unit 18 Modulating unit 29 Distance calculating unit 30 Corresponding relationship calculating unit 31 Hidden point detecting unit 32 Mapping unit 33 Calibration unit 34 CPU

Claims (7)

First imaging means for capturing distance image data representing a three-dimensional shape of the subject by imaging the reflected light of the distance measuring light by the subject by irradiating the subject with distance measuring light;
Second imaging means for obtaining image data representing a two-dimensional image representing the brightness of the subject arranged at a position different from the first imaging means;
Synchronization control means for synchronizing the driving of the first and second imaging means to cause the first and second imaging means to acquire the data for the distance image and the image data representing the two-dimensional image; ,
Distance image generation means for generating a distance image representing the three-dimensional shape of the subject based on the data for the distance image;
Correspondence calculation means for calculating the correspondence between the pixels on the distance image and the pixels on the two-dimensional image;
On the subject, a hidden point detecting means for detecting a hidden point that can be seen from the first imaging means but cannot be seen from the second imaging means;
A stereoscopic imaging apparatus comprising: mapping means for mapping the two-dimensional image to the distance image so as to visually recognize the hidden point based on the correspondence relationship and generating a mapping image.   The hidden point detection means calculates an outward normal vector for each pixel on the distance image, calculates a line-of-sight vector that anticipates each pixel on the distance image from the second imaging means, and the distance image The upper pixel of interest is means for detecting the pixel of interest as the hidden point when the inner product of the normal vector and the line-of-sight vector is equal to or greater than a predetermined threshold value. The three-dimensional imaging device described.   The hidden point detecting means calculates a line-of-sight vector that anticipates each pixel on the distance image from the second imaging means, and for the pixel of interest on the distance image, the line-of-sight vector that anticipates the pixel of interest on the distance image And a distance relationship between the line-of-sight vector and the projection line corresponding pixel on the projection line is obtained on a projection line corresponding to the projected line-of-sight vector on the distance image, and at least one projection line corresponding pixel is The stereoscopic imaging apparatus according to claim 1, wherein the stereoscopic imaging device is a unit that detects the target pixel as the hidden point when located closer to the second imaging unit than the line-of-sight vector.   The operation mode of the first imaging unit is changed to an operation mode similar to that of the second imaging unit, the first and second imaging units are caused to capture a predetermined chart, and the first image acquired by the imaging is obtained. 2. The apparatus according to claim 1, further comprising calibration means for performing a calibration process for acquiring geometric information of the first and second imaging means based on the first and second chart images. The stereoscopic imaging device according to any one of 3. The first imaging means includes a filter that transmits only light in the wavelength range of the ranging light so that it can be taken in and out of the optical axis of the first imaging means;
The calibration means is means for retracting the filter from the optical axis when the operation mode of the first imaging means is changed to an operation mode similar to that of the second imaging means. Item 5. The stereoscopic imaging device according to Item 4. The first imaging means irradiates the subject with distance measuring light to capture the reflected light of the distance measuring light by the subject, obtain distance image data representing the three-dimensional shape of the subject, and When acquiring image data representing a two-dimensional image representing the brightness of the subject by the second imaging means arranged at a position different from the one imaging means, the driving of the first and second imaging means is synchronized. And obtaining the distance image data and the image data representing the two-dimensional image,
Based on the data for the distance image, generate a distance image representing the three-dimensional shape of the subject,
Calculating a correspondence between a pixel on the distance image and a pixel on the two-dimensional image;
On the subject, a hidden point that can be seen from the first image pickup means but cannot be seen from the second image pickup means is detected,
A control method for a stereoscopic imaging apparatus, wherein a mapping image is generated by mapping the two-dimensional image to the distance image so that the hidden point can be visually recognized based on the correspondence relationship. The first imaging means irradiates the subject with distance measuring light to capture the reflected light of the distance measuring light by the subject, obtain distance image data representing the three-dimensional shape of the subject, and When acquiring image data representing a two-dimensional image representing the brightness of the subject by the second imaging means arranged at a position different from the one imaging means, the driving of the first and second imaging means is synchronized. A procedure for obtaining the distance image data and the image data representing the two-dimensional image;
A procedure for generating a distance image representing the three-dimensional shape of the subject based on the data for the distance image;
A procedure for calculating a correspondence between a pixel on the distance image and a pixel on the two-dimensional image;
On the subject, a procedure for detecting a hidden point that can be faced from the first image pickup means but cannot be seen from the second image pickup means;
A method of controlling a stereoscopic imaging apparatus, comprising: mapping a two-dimensional image to the distance image to generate a mapping image based on the correspondence relationship so that the hidden point can be visually recognized. Program to let you.