JP2012209872A - 3d image generating method, 3d image generating program and 3d image generating apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a stereoscopic image in which a state of a moving body can stereoscopically be viewed in one image.SOLUTION: In a left viewpoint imaging section and a right viewpoint imaging section of a compound-eye camera, the moving body is imaged at every prescribed time during a period from time t0 to time t4. A left viewpoint image and a right viewpoint image are obtained during the period from time t0 to time t4 by imaging. The left viewpoint images at time t0, time t2 and t4 when the moving body is not overlapped if it is overlapped with the left viewpoint image at other time are selected as specified left viewpoint images from the left viewpoint images. Specified right viewpoint images are similarly selected on the right viewpoint images. Thus, a stereoscopic vision image of the moving body, in which the state of the moving body can stereoscopically be viewed in one image, is generated by synthesizing the specified left viewpoint image and the specified right viewpoint image.

Description

  The present invention relates to a 3D image generation method, a 3D image generation program, and a 3D image generation apparatus that generate a 3D image that enables stereoscopic viewing of a subject.

  A strobe photograph showing a moving subject (moving object) in one photograph is known. In a film camera that has been widely used as a conventional imaging means, a stroboscopic photograph is generated by performing stroboscopic light emission a plurality of times at regular intervals in a state where the aperture is opened and the film is exposed. In place of such conventional film cameras, digital cameras that are widely used today use a computer to process a plurality of images obtained by continuous shooting while a moving object moves, thereby obtaining a flash photography. Such an image is generated (see Patent Document 1).

JP 2010-166545 A

  Today, in addition to a single-lens digital camera that captures an image with a single imaging unit, a 3D image that enables stereoscopic viewing of the subject is acquired by imaging the subject with two imaging units for the left eye and the right eye. Demand for imaging devices is increasing. Some of such stereoscopic imaging devices have a function of acquiring a 3D still image or a 3D moving image. However, a stereoscopic image that allows a moving subject to be stereoscopically viewed in one image is obtained. There is no device having the 3D strobe function to be acquired at present (they are not described or suggested in Patent Document 1).

  Note that even in a stereoscopic imaging device, a stereoscopic image such as a flash photograph can be obtained by combining a plurality of left viewpoint images and right viewpoint images obtained by continuous shooting imaging. However, in such a stereoscopic image that is simply a combination of the left viewpoint image and the right viewpoint image, if there is a portion where the moving body moves slowly, the moving body in that portion may overlap. If the moving objects overlap in this way, the front-rear relationship of the moving objects becomes partially indistinguishable, and the observer may have difficulty in stereoscopically viewing the moving object.

  An object of the present invention is to provide a 3D image generation method, a 3D image generation program, and a 3D image generation apparatus capable of generating a stereoscopic image in which a moving object can be stereoscopically viewed in one image.

  In order to achieve the above object, the 3D image generation method of the present invention provides a plurality of first images obtained by continuously performing simultaneous imaging for imaging a moving object that is a subject moving from two different viewpoints at the same timing. An image acquisition step of acquiring one viewpoint image and a second viewpoint image, and a first viewpoint image captured at another timing from the plurality of first viewpoint images and second viewpoint images. When combining a specific first viewpoint image in which moving objects do not overlap when combined, and a second viewpoint image captured simultaneously with the specific first viewpoint image and captured at other timing An image selection step of selecting a specific second viewpoint image in which the moving objects do not overlap each other, and combining the selected specific first viewpoint image and the specific second viewpoint image, thereby making the state of the moving object one In the image And having a three-dimensional image generating step of generating a stereoscopically viewable stereoscopic image.

  The image selection step detects a moving direction of the moving object in the first viewpoint image at each imaging timing, and detects a moving direction of the moving object in the second viewpoint image at each imaging timing. Selecting a first viewpoint image in which the coordinates of a moving object relating to a moving direction satisfy a predetermined condition from the plurality of first viewpoint images and the second viewpoint image as the specific first viewpoint image And a first selection step of selecting, as the specific second viewpoint image, a second viewpoint image that is simultaneously captured with the specific first viewpoint image and in which the coordinates of the moving object relating to the movement direction satisfy a predetermined condition. Are preferably included.

  The image selection step includes: a second moving object detection step that detects a moving direction of a moving object in the first viewpoint image at each imaging timing; and a moving object related to the moving direction from the plurality of first viewpoint images. A first first viewpoint image whose coordinates satisfy a predetermined condition is temporarily selected, and a first first viewpoint image selected from the plurality of second viewpoint images is simultaneously captured with the first selected viewpoint image. A second selection step of selecting the second viewpoint image, a third moving object detection step of detecting a movement direction of the moving object in the second viewpoint image selected in the second selection step, and the second From the second viewpoint image selected in the selection step, the specific second viewpoint image in which the coordinates of the moving object relating to the moving direction satisfy a predetermined condition is selected, and the temporarily selected specific first viewpoint image From the above A third selection step of selecting a specific first viewpoint image captured simultaneously with the second viewpoint image, and the three-dimensional image generation step include the specific first selected at the third selection step. It is preferable to generate a stereoscopic image by combining the viewpoint image and the specific second viewpoint image selected in the third selection step.

  The predetermined condition is that the minimum position coordinate of the moving object at the first imaging timing is greater than or equal to a reference coordinate set from the maximum position coordinate of the moving object at the second imaging timing before the first imaging timing. preferable.

  It is preferable that the stereoscopic image generation step generates a stereoscopic image so that a moving object at a specific imaging timing is emphasized more than a moving object at another imaging timing.

  In the stereoscopic image generation step, in addition to the selected specific first viewpoint image and the specific second viewpoint image, the first viewpoint image not selected as the specific first viewpoint image It is preferable to generate a stereoscopic image by combining a part and a part of the second viewpoint image that has not been selected as the specific second viewpoint image.

  In the stereoscopic image generating step, it is preferable to generate a stereoscopic image in which a plurality of moving objects can be stereoscopically viewed in one image.

  A parallax amount determination step of determining whether or not a parallax amount indicating how far the same moving object is separated in the first viewpoint image and the second viewpoint image simultaneously captured exceeds a preset allowable value; The stereoscopic image generating step combines the images of the selected specific first viewpoint image and the specific second viewpoint image that have a parallax amount less than an allowable value, thereby generating a stereoscopic image Is preferably generated.

  The 3D image generation program of the present invention causes a computer to execute each step in the 3D image generation method of the present invention described above.

  The 3D image generation apparatus according to the present invention includes a plurality of first viewpoint images and a second viewpoint image obtained by continuously performing simultaneous imaging of a moving object that is a subject moving from two different viewpoints at the same timing. The moving object overlaps when an image acquisition means for acquiring a viewpoint image and the first viewpoint image captured at another timing from the plurality of first viewpoint images and the second viewpoint image are combined. A specific first viewpoint image and a specific first viewpoint image that is simultaneously captured with the specific first viewpoint image and combined with the second viewpoint image captured at other timings are not overlapped. The state of the moving object can be stereoscopically viewed in one image by synthesizing the selected first viewpoint image and the specific second viewpoint image with the image selecting means for selecting two viewpoint images. Generate stereoscopic images Characterized in that it comprises a three-dimensional image generation means.

  According to the present invention, it is possible to generate a stereoscopic image in which a moving object can be stereoscopically viewed in one image. The stereoscopic image is a specific first image in which moving objects do not overlap when a first viewpoint image captured at another timing is synthesized from among a plurality of first viewpoint images and second viewpoint images. Select a specific second viewpoint image in which moving objects do not overlap when the viewpoint image and the second viewpoint image captured at the same time as this specific first viewpoint image are combined. In addition, since the image is generated using the selected specific first viewpoint image and specific second viewpoint image, the image quality is not significantly deteriorated and an unnatural 3D image is not obtained.

  In addition, since the stereoscopic image is generated by multi-frame synthesis of the specific first viewpoint image and the specific second viewpoint image, a high-sensitivity and low-noise image quality can be obtained in an area where there is no moving object such as a background. The selection of the specific first viewpoint image and the specific second viewpoint image is performed using the coordinates of the moving direction of the moving object in the first viewpoint image and the second viewpoint image at each imaging timing. Therefore, it is possible to reliably avoid overlapping (multiplexing) of moving objects when the left viewpoint image and the right viewpoint image at other imaging timings are combined.

It is the figure of the compound eye camera seen from the front side. It is a figure of the compound eye camera seen from the back side. It is a figure which shows the electrical constitution of a compound eye camera. It is a figure which shows each part of a 2nd stereo image production | generation circuit. It is a figure for demonstrating the corresponding point of the moving body in each time. It is a figure for demonstrating selection of the specific left viewpoint image and specific right viewpoint image which are performed in 1st Embodiment. It is a figure which shows the moving body stereoscopic image of 1st Embodiment. It is a figure which shows the flow from acquisition of a left viewpoint image and a right viewpoint image to the production | generation of a moving body stereoscopic image. It is a figure which shows the procedure which produces | generates a moving body stereoscopic image from the specific left viewpoint image obtained according to the flow of FIG. 8, and a specific right viewpoint image. It is a figure which shows the flow from acquisition of a left viewpoint image and a right viewpoint image to provisional selection of a specific right viewpoint image. It is a figure which shows the flow until it produces | generates a moving body stereoscopic image following the flow of FIG. It is a figure which shows the procedure which produces | generates a moving body stereo image from the specific left viewpoint image obtained according to the flow of FIG.10 and FIG.11, and this specific right viewpoint image selected. It is a figure which shows the procedure which produces | generates the moving body stereoscopic image of 2nd Embodiment. It is a figure which shows the moving body stereoscopic vision image of 2nd Embodiment. It is a figure for demonstrating the process which cuts out the part from the left viewpoint image and right viewpoint image which are not a specific left viewpoint image and a specific right viewpoint image. It is a figure which shows the moving body stereoscopic image of 3rd Embodiment. It is a figure for demonstrating selection of the specific left viewpoint image and specific right viewpoint image which are performed in 4th Embodiment. It is a figure which shows the moving body stereoscopic image of 4th Embodiment. It is a figure for demonstrating selection of the specific left viewpoint image and specific right viewpoint image, and determination of a parallax amount which are performed in 5th Embodiment. It is a figure which shows the moving body stereoscopic image of 5th Embodiment. It is a figure which shows PC in which the moving body stereoscopic image generation program of this invention was installed.

  As shown in FIG. 1, the compound-eye camera 10 according to the first embodiment includes a left viewpoint 12, a right viewpoint imaging unit 13, and a strobe light emitting unit 14 on the front surface of the camera body 11, and a shutter on the upper surface of the camera body 11. A button 15 and a power switch 16 are provided. The left-viewpoint and right-viewpoint imaging units 12 and 13 are provided at a predetermined interval so that their optical axes are substantially parallel to each other. Further, the shutter button 15 can be pressed halfway and fully, and autofocusing is performed during the half-pressing operation, and during the full-pressing operation, the captured image at the time of the operation is stored in the memory card 20 (see FIG. 2). To record.

  As shown in FIG. 2, a monitor 18 and an operation unit 19 are provided on the back surface of the camera body 11. The bottom surface of the camera body 11 is provided with a card slot in which a memory card 20 such as an SD card is detachably loaded, and an openable / closable loading lid that covers the opening of the card slot (both are all). (Not shown).

  The monitor 18 includes an LCD for displaying an image and a lenticular sheet (both not shown) attached to the LCD. By observing a stereoscopic image displayed on the LCD through the lenticular sheet, the image is displayed. Can be stereoscopically viewed. The operation unit 19 is selected by a mode selector switch 22 for setting various modes, a menu button 23 for displaying a menu screen or a setting screen on the monitor 18, a cross key 24 for moving a cursor on the screen, and these switches. And an execution button 25 for determining a mode to be executed.

  In the present embodiment, the mode changeover switch 22 displays a still image mode for obtaining a stereoscopic image that enables a stereoscopic subject to be viewed on the monitor 18 and a state of a subject (moving object) that moves within a predetermined time as one image. The mode can be switched between a strobe mode for acquiring a moving body stereoscopic image, which is a stereoscopic image that can be stereoscopically viewed. In addition to the still image mode and the strobe mode, a moving image mode or the like may be provided.

  As shown in FIG. 3, the left viewpoint imaging unit 12 includes a lens unit 37 on which light from a subject is incident, a CCD 39 that captures a left viewpoint image formed by the lens unit 37 and outputs an imaging signal, The AFE (Analog Front End) 40 that converts the imaging signal output from the CCD 39 into a left viewpoint image signal (L image signal) by a predetermined process and outputs the converted L image signal to the image input controller 45, and the CCD 39 and An imaging control unit 41 for the left viewpoint is connected to the AFE 40 and controls the output of the imaging signal to the image input controller 45 based on an instruction from the CPU 33. Note that imaging of a subject may be performed by a CMOS (Complementary Metal Oxide Semiconductor) instead of the CCD.

  Similar to the left viewpoint imaging unit 12, the right viewpoint imaging unit 13 includes a lens unit 38, a CCD 48, an AFE 49, and a right viewpoint imaging control unit 50. The right viewpoint imaging unit 13 outputs a right viewpoint image signal (R image signal) obtained by imaging the right viewpoint image to the image input controller 45.

  The CPU 33 is provided inside the camera body 11 and electrically controls each part of the compound-eye camera 10. The CPU 33 is provided with a general imaging control unit 33 a that controls both the left-viewpoint imaging control unit 41 and the right-viewpoint imaging control unit 50. The overall imaging control unit 33a performs different control in the still image mode and the strobe mode. In the still image mode, the left viewpoint image to be recorded on the memory card 20 when the subject is simultaneously imaged by both the left viewpoint imaging unit 12 and the right viewpoint imaging unit 13 and the shutter button 15 is fully pressed. The L image signal and the R image signal that are the basis of the right viewpoint image are output to the image input controller 45.

  On the other hand, when the strobe mode is set, the left viewpoint is used from when the shutter button 15 is fully pressed at time t0 to when the shutter button 15 is fully pressed again at time tn after a predetermined time. The imaging unit 12 and the right-viewpoint imaging unit 13 are controlled so that the subject is imaged simultaneously. The imaging is continuously performed from time t0 to time tn at a constant time interval (t1-t0 = t2-t1 = ... = tn-t (n-1)). That is, continuous shooting is performed. Thus, (n + 1) L image signals obtained by imaging by the left parallax imaging unit 12 and (n + 1) R image signals obtained by imaging by the right parallax imaging unit 13 are image input controller 45. Is output.

  The image input controller 45 has a buffer of a predetermined capacity, temporarily stores the L and R image signals output from the left viewpoint imaging unit 12 and the right viewpoint imaging unit 13, and stores the stored L, The R image signal is transmitted to the signal processing circuit 56.

  The signal processing circuit 56 performs various processing such as gradation conversion, white balance correction, γ correction processing, and YC conversion processing on the L and R image signals from the image input controller 45, thereby converting the L and R image data. Generate. The generated L and R image data is stored in the RAM 34. The L and R image data stored in the RAM 34 is read by the first stereoscopic image generation circuit 60 and the second stereoscopic image generation circuit 61.

  The first stereoscopic image generation circuit 60 generates 3D still image data by combining the L and R image data read from the RAM 34 when the still image mode is set. The generated 3D still image data is transmitted to the display circuit 59, and the display circuit 59 displays the 3D still image on the monitor 18 based on the received 3D still image data. When the shutter button 15 is fully pressed, the still image data is transmitted to the display circuit 59 and the compression / expansion processing circuit 57.

  The compression / decompression processing circuit 57 performs compression processing on the still image data to generate compressed composite image data in a predetermined file format. The generated compressed composite image data is recorded on the memory card 20 by the media controller 58. The display circuit 59 displays a 3D still image at the time when the shutter button 15 is fully pressed on the monitor 18 based on the 3D still image data.

  As shown in FIG. 4, the second stereoscopic image generation circuit 61 is a circuit that generates a dynamic stereoscopic image in the strobe mode, and moves from L and R image data including information on the left viewpoint image and the right viewpoint image. A moving object detection unit 70 that detects a subject (moving object), and a left viewpoint image and a right viewpoint image at other times from among all left viewpoint images and right viewpoint images, based on the detection result of the moving object detection unit 70 An image selection unit 71 that selects a specific left viewpoint image and a right viewpoint image that do not overlap moving objects when combined, and a moving object stereoscopic image that generates a moving object stereoscopic image from the specific left viewpoint image and right viewpoint image And a generation unit 72. The moving body stereoscopic image generated by the moving body stereoscopic image generation unit 72 is displayed on the monitor 18 as well as the 3D still image, and is recorded in the memory card 20 as moving body stereoscopic image data.

  The moving object detection unit 70 extracts feature points for each of the (n + 1) left viewpoint images, and specifies feature points that are substantially common to the left viewpoint images as corresponding points. For example, in the case of FIG. 5, the corresponding point of the feature point A on the left viewpoint image at time t0 is the feature point A1 on the left viewpoint image at time t1, and A2 on the left viewpoint image at time t2. Become. In addition, the moving object detection unit 70 similarly extracts feature points for the (n + 1) right viewpoint images, and specifies feature points that are substantially common to the right viewpoint images as corresponding points.

  Then, the moving object is detected by comparing corresponding points of each of the left viewpoint image and the right viewpoint image in the order of time t1, time t2,. When a moving object is detected at time tk (k is a natural number equal to or less than n), the position information of the corresponding point and the luminance information of the surrounding pixels are recorded in the RAM 34, and the region (size) of the moving object and the moving direction X ( (See FIG. 5). Then, from this estimation result, the maximum position coordinate XmaxL0 and the minimum position coordinate XminL0 of the moving object at time t0 (both are coordinates related to the movement direction X) are obtained, and the maximum position coordinate XmaxLk and the minimum position coordinate of the moving object at time tk are obtained. XminLk (both coordinates related to the movement direction X) is obtained. Each time a moving object is detected, the moving object detection unit 70 estimates a moving object region and a moving direction, and obtains a maximum position coordinate and a minimum position coordinate with respect to the moving direction at the time of detection.

  The image selection unit 71 sets the maximum position coordinates XmaxL0, XmaxR0 of the moving object at time t0 as the reference coordinates Ls, Rs, and then sets the minimum position of the moving object in the left viewpoint image and the right viewpoint image at time tk when the moving object is detected. It is determined whether or not the coordinates are greater than or equal to the reference coordinates Ls and Rs. As a result of the determination, if both the minimum position coordinates XminLk and XminRk of the moving object are not equal to or higher than the reference coordinates Ls and Rs, the left viewpoint image and the right viewpoint image at time tk are not selected as the specific left viewpoint image and right viewpoint image.

  On the other hand, if both the minimum position coordinates XminLk and XminRk of the moving object are determined to be greater than or equal to the reference coordinates Ls and Rs, the left viewpoint image and the right viewpoint image at time tk are selected as specific left viewpoint images and right viewpoint image images. At the same time, the reference coordinates Ls and Rs are changed to the maximum position coordinates XmaxLk and XmaxRk at time tk. The selection of the specific left viewpoint image and right viewpoint image is performed every time a moving object is detected.

  For example, as illustrated in FIG. 6, by capturing a subject continuously from time t0 to time t4, five left viewpoint images and five right viewpoint images from time t0 to time t4 are obtained. The procedure for selecting a specific left viewpoint image and right viewpoint image when obtained is as follows. First, the maximum position coordinate XmaxL0 in the movement direction X from the left viewpoint image at time t0 is set as the reference coordinate Ls, and similarly, the maximum position coordinate XmaxR0 in the movement direction X from the right viewpoint image at time t0 is set as the reference coordinate Rs. To do. In FIG. 6, the coordinates of the movement direction X increase from the right side to the left side.

  Next, it is determined whether or not the minimum position coordinate XminL1 of the moving object in the left viewpoint image at time t1 is greater than or equal to the reference coordinate Ls (XmaxL0), and the minimum position coordinate XminR1 in the right viewpoint image at time t1 is the reference coordinate Rs ( XminR0) or not. Since Ls (XmaxL0)> XminL1 and Rs (XmaxR0)> XminR1 at time t1, the left viewpoint image and right viewpoint image at time t1 are not selected as specific left viewpoint images and right viewpoint image images.

  Similarly, next, in the left viewpoint image at time t2, it is determined whether or not the minimum position coordinate XminL2 with respect to the movement direction X is greater than or equal to the reference coordinate Ls (XmaxL0), and the minimum position coordinate in the right viewpoint image at time t2. Determine whether XminR2 is greater than or equal to the reference coordinate Rs (XminR0). At time t2, since Ls (XmaxL0) ≦ XminL2 and Rs (XmaxR0) ≦ XminR2, the left viewpoint image and right viewpoint image at time t2 are selected as specific left viewpoint images and right viewpoint image images. Further, the reference coordinate Ls is updated to XmaxL2, and the reference coordinate Rs is updated to XmaxR2.

  Next, in the left viewpoint image at time t3, it is determined whether or not the minimum position coordinate XminL3 with respect to the movement direction X is greater than or equal to the reference coordinate Ls (XmaxL2), and in the right viewpoint image at time t3, the minimum position coordinate XminR3 is the reference coordinate. Determine whether Rs (XminR2) or higher. At time t3, although Rs (XmaxR2) ≦ XminR3, Ls (XmaxL2)> XminL3, so the left viewpoint image and right viewpoint image at time t3 are not selected as the specific left viewpoint image and right viewpoint image.

  Next, in the left viewpoint image at time t4, it is determined whether or not the minimum position coordinate XminL4 with respect to the movement direction X is greater than or equal to the reference coordinate Ls (XmaxL2), and in the right viewpoint image at time t4, the minimum position coordinate XminR4 is the reference coordinate. Determine whether Rs (XminR2) or higher. At time t4, since Ls (XmaxL2) ≦ XminL4 and Rs (XmaxR2) ≦ XminR4, the left viewpoint image and right viewpoint image at time t4 are selected as specific left viewpoint images and right viewpoint image images.

  As described above, the left viewpoint image and the right viewpoint image at time t2 and time t4 are selected as the synthesis target images from the left viewpoint image and the right viewpoint image from time t0 to time t4. In addition, since the moving object does not overlap with the left viewpoint image and the right viewpoint image at time t0 when they are combined with the left viewpoint image and the right viewpoint image at other times, the specific left viewpoint image and the right viewpoint image are obtained. Selected.

  The moving body stereoscopic image generation unit 72 generates a left viewpoint synthesized image by synthesizing a specific left viewpoint image, and generates a right viewpoint synthesized image by synthesizing the specific right viewpoint image. Then, by further synthesizing the left viewpoint synthesized image and the right viewpoint synthesized image, a moving body stereoscopic image capable of stereoscopically viewing the state of the moving body in one image is generated.

  For example, when the left viewpoint image and the right viewpoint image at time t0, time t2, and time t4 are selected as specific left viewpoint images and right viewpoint image images as shown in FIG. 6, as shown in FIG. A left viewpoint composite image 75 that is a composite image of the left viewpoint images at time t0, time t2, and time t4 and a right viewpoint composite image 76 that is a composite image of the right viewpoint images at time t0, time t2, and time t4 are further combined. By doing so, a moving body stereoscopic image 77 in which the state of the moving body between time t0 and time t4 is stereoscopically viewable is obtained.

  Only the left viewpoint image and right viewpoint image at time t0, time t2, and time t4 selected as the specific left viewpoint image and right viewpoint image image are used to generate the moving body stereoscopic image 77. The left viewpoint image and the right viewpoint image at time t1 and time t3 that are not selected as the viewpoint image and the right viewpoint image are not used. If the left viewpoint image and right viewpoint image at time t1 that are not used and the left viewpoint image and right viewpoint image at time t0 are combined, the respective moving objects overlap (multiplexing of moving objects). Similarly, when the left viewpoint image at time t3 is combined with the left viewpoint image at time t2, multiplexing of moving objects occurs.

  Therefore, since the moving body stereoscopic image is generated without using the left viewpoint image and the right viewpoint image in which such a moving body can be multiplexed, the state of the moving body can be represented as a 3D image without causing image quality failure. Can be reliably grasped as.

  Further, even when the moving object can be multiplexed in only one of the left viewpoint image and the right viewpoint image as at time t3, the left viewpoint image and the right viewpoint image at that time are converted into moving object stereoscopic vision. By not using it for image generation, there is no risk of image quality degradation. For example, it is possible to prevent confusion between multiplexing of moving objects and parallax in both eyes.

  Also, since the size and direction of the moving object are detected each time a moving object is detected, even when the moving object moves in the middle or the size is deformed, Therefore, it is possible to accurately determine whether or not the moving object causes multiplexing.

  In addition, when generating a moving body stereoscopic image, a left viewpoint composite image obtained by performing multiframe composition processing on a specific left viewpoint image, and a right viewpoint composite image obtained by performing multiframe composition processing on a specific right viewpoint image in the same manner Therefore, it is possible to achieve a high-sensitivity and low-noise image quality in areas other than moving objects such as the background.

  Next, the flow until the moving body stereoscopic image is generated will be described with reference to the flowcharts of FIGS. First, the mode is switched to the strobe mode by the mode switch 22. Then, the shutter button 15 is fully pressed with the left-viewpoint imaging unit 12 and the right-viewpoint imaging unit 13 facing the subject (time t0). The subject is moved simultaneously with or immediately after the full-press operation. When a moving subject (moving object) is out of frame from the monitor 18 after a certain time, the shutter button 15 is fully pressed again (time tn). Thereby, a left viewpoint image and a right viewpoint image between time t0 and time tn (n is a natural number of 1 or more) are obtained.

  When the left viewpoint image and the right viewpoint image between time t0 and time tn are obtained, the moving object detection unit 70 extracts feature points from all the left viewpoint images and right viewpoint images. Next, by comparing corresponding points of the left viewpoint image and the right viewpoint image at time t0 with the left viewpoint image and the right viewpoint image at time t1, is there a moving object in the left viewpoint image and the right viewpoint image at time t1? A moving object detection process is performed to detect whether or not. If no moving object is detected, the same processing is performed on the left viewpoint image and the right viewpoint image at the next time t2. This moving object detection process is performed for each of the left viewpoint image and the right viewpoint image at each time, and is repeated until a moving object is detected.

  When a moving object is detected from the left viewpoint image and the right viewpoint image at time tk, the size of the moving object and the moving direction X are estimated using the luminance information of the corresponding points and the surrounding pixels. In addition, in the image selection unit 71, the maximum position coordinates XmaxL0, XmaxR0 of the moving object at time t0 are set to the reference coordinates Ls, Rs, and the reference coordinates Ls, Rs and the minimum position coordinates XminLk, XminRk of the moving object at time tk are set. Compare size. As a result of the size comparison, if both XminLk ≧ Ls and XminRk ≧ Rs are not satisfied, the left viewpoint image and right viewpoint image at time tk are not selected as the specific left viewpoint image and right viewpoint image image, and the next time A moving object detection process is performed for t (k + 1).

  On the other hand, when both XminLk ≧ Ls and XminRk ≧ Rs are satisfied, the left viewpoint image and right viewpoint image at time tk are selected as specific left viewpoint images and right viewpoint image images. Then, the reference coordinates Ls, Rs are changed to the maximum position coordinates XmaxLk, XmaxRk of the moving object at time tk. When the change of the reference coordinates is completed, the moving object detection process is performed at the next time t (k + 1).

  For all the left viewpoint images and right viewpoint images, after the detection of the moving object and the determination as to whether or not it is a specific left viewpoint image and right viewpoint image image, the moving object stereoscopic image generation unit 72 performs the moving object stereoscopic image. Perform the generation process. The moving object stereoscopic image generation unit 72 generates a left viewpoint synthesized image by synthesizing a specific left viewpoint image. Similarly, a right viewpoint synthesized image is generated by synthesizing a specific right viewpoint image. In addition, a moving object stereoscopic image is generated by further combining the left viewpoint composite image and the right viewpoint composite image. The generated moving body stereoscopic image is displayed on the monitor 18 and recorded in the memory card 20 as moving body stereoscopic image data.

  In the first embodiment, moving object detection processing is performed on all left viewpoint images and right viewpoint images, and among the left viewpoint image and right viewpoint image from which moving objects are detected, a specific left viewpoint image and right viewpoint image are detected. Although it is determined whether or not it corresponds to the viewpoint image, it is not necessary to be limited to this.

  For example, as shown in FIG. 10, feature points are extracted only from the right viewpoint image, and a moving object detection process is performed based on the extraction result. If a moving object is detected from the right viewpoint image at time tk as a result of detection, the minimum position coordinate XminRk of the moving object at time tk and the reference coordinate Rs are compared in size. If XminRk ≧ Rs is satisfied as a result of the size comparison, the right viewpoint image at time tk is provisionally selected as a specific right viewpoint image. In FIG. 10, the flow from feature point extraction to reference coordinate change is the same as that in FIG. 8 except that only the right viewpoint image is processed, and thus detailed description thereof is omitted.

  When the moving object detection process and the determination as to whether or not the image is a specific right viewpoint image are completed for all the right viewpoint images, as shown in FIG. 11, the specific right viewpoint image temporarily selected from all the left viewpoint images A left viewpoint image captured at the same time is selected. Then, feature points are extracted from these selected left viewpoint images (hereinafter referred to as “selected left viewpoint images”), and the size and direction of the moving object are estimated for each image using the extracted feature points. Then, using this estimation result, a specific left viewpoint image in which moving objects do not overlap when selected with the selected left viewpoint image at another time is selected from the selected left viewpoint images. The determination of whether or not the image is a specific left viewpoint image is performed on all the selected left viewpoint images.

  Note that the specific left viewpoint image selected from the selected left viewpoint images is a selected left viewpoint image in which the minimum position coordinate of the moving object in the movement direction is equal to or greater than the reference coordinate, as in the above embodiment. The setting of the reference coordinates and the comparison of the size of the minimum position coordinates and the reference coordinates performed when selecting a specific left viewpoint image are the same as in the above embodiment, and thus detailed description thereof is omitted.

  When the determination as to whether or not all the selected left viewpoint images are specific left viewpoint images is completed, a specific image captured at the same time as the specific left viewpoint image is selected from the temporarily selected specific right viewpoint images. After the right viewpoint image is selected, a moving object stereoscopic image generation process is performed.

  As illustrated in FIG. 12, in the moving object stereoscopic image generation process, a left viewpoint synthesized image is generated by synthesizing a specific left viewpoint image. Further, the right viewpoint synthesized image is generated by synthesizing the selected specific right viewpoint image. Then, a moving object stereoscopic image is generated by further combining the generated left viewpoint composite image and right viewpoint composite image.

  As described above, by generating a moving object stereoscopic image, it is possible to reduce the processing necessary for generating the image and shorten the processing time. A specific right viewpoint image is temporarily selected from all right viewpoint images, and then a specific left viewpoint image is selected from the left viewpoint images captured at the same time as the temporarily selected specific right viewpoint image. However, on the contrary, after temporarily selecting a specific left viewpoint image from all the left viewpoint images, a specific right viewpoint is selected from the right viewpoint images captured at the same time as the temporarily selected specific left viewpoint image. A viewpoint image may be selected.

  The second embodiment of the present invention differs from the first embodiment in which a specific left viewpoint image and a specific right viewpoint image are combined so that moving objects at any time have the same brightness when multi-frame combining is performed. The synthesis is performed so that the moving object at the time is emphasized more than the moving objects at the other times. That is, the composition ratio of moving objects at a specific time is set larger than the composition ratio of moving objects at other times. In the second embodiment, the same processing as in the first embodiment is performed from the extraction of the feature points to the change of the reference coordinates (see FIG. 8).

  When the detection of moving objects for all the left viewpoint images and right viewpoint images and the determination of whether or not they correspond to specific left viewpoint images and right viewpoint images are completed, the composition ratio of moving objects is increased as shown in FIG. The time tx to be input is input by the operation unit 19. Then, as shown in FIG. 14, the left viewpoint composite image 100 is obtained by multi-frame combining the specific left viewpoint image so that the composition ratio of the moving object 100 a at time tx is higher than the composition ratio of the moving object at other times. Generate. Further, the same process is performed on the right viewpoint image, thereby generating the right viewpoint composite image 101 having a high composition ratio of the moving object 101a at time tx. By synthesizing the left-viewpoint synthesized image and the right-viewpoint synthesized image, the moving object stereoscopic image 102 in which the moving object 102a at time t is emphasized more than the moving object at other times is obtained.

  In the second embodiment, only the moving object at a specific time is emphasized in the moving object stereoscopic image, but the present invention is not limited to this. For example, when displaying a moving object stereoscopic image on a monitor, the moving object to be emphasized may be changed at regular intervals so that the movement of the moving object can be understood.

  In the third embodiment of the present invention, only the specific left viewpoint image and the right viewpoint image are synthesized, and the left viewpoint image and the right viewpoint image that are not the specific left viewpoint image and the right viewpoint image are not synthesized. Unlike the left viewpoint image and the right viewpoint image that are not the specific left viewpoint image and the right viewpoint image, if the stereoscopic image does not become an unnatural stereoscopic image, a part of them may be synthesized. Good.

  For example, in FIG. 15, since the left viewpoint image and the right viewpoint image at time t2 are the specific left viewpoint image and right viewpoint image, the minimum position coordinates XminL2, XminR2 of the moving object exceed the reference coordinates Ls, Rs. ing. On the other hand, for the left viewpoint image and the right viewpoint image at time t1, both the minimum position coordinates XminL1 and XminR1 of the moving object are less than the reference coordinates Ls and Rs, and for the right viewpoint image at time t3, the moving object Although the minimum position coordinate XminR3 exceeds the reference coordinate Ls, the minimum position coordinate XminL3 of the moving object is less than the reference coordinate Rs for the left viewpoint image at time t3.

  Therefore, regarding the left viewpoint image and the right viewpoint image at time t1 in which the minimum position coordinates XminL1 and XminR1 of the moving object are both less than the reference coordinates Ls and Rs, Ls (XminL0 ) And a region exceeding Rs (XminR0) with respect to the motion direction X in the right viewpoint image at time t1.

  Then, as shown in FIG. 16, a specific left viewpoint image and a specific right viewpoint image at time t0 and time t2, and an area portion (Ls (XminL0), The left viewpoint synthesized image 120 is obtained by synthesizing the image in the region exceeding Rs (XminR0). At that time, the composition ratio of the region portion cut out from the left viewpoint image and the right viewpoint image at time t1 is set to be lower than that of the other images. The right viewpoint composite image 121 is obtained by performing the same composition process for the right viewpoint image. Then, by further synthesizing the obtained left viewpoint synthesized image 120 and right viewpoint synthesized image 121, a moving body stereoscopic image 122 is obtained.

  The moving body stereoscopic image 122 can express a sense of speed of the moving body on the moving body stereoscopic image 122 by combining a part of the moving body in an image that is not a specific left viewpoint image and right viewpoint image. it can. In addition, since the composition ratio of a part of the moving object is lower than that of other images, image quality deterioration such as difficulty in seeing the moving object does not occur.

  Note that in the third embodiment, as in time t1, a part of the moving object in the image is displayed in the moving object stereoscopic view only when the minimum position coordinate of the moving object is less than the reference coordinate for both the left viewpoint image and the right viewpoint image. Even when only one of the left viewpoint image and the right viewpoint image has a minimum position coordinate of the moving object that is less than the reference coordinates as at time t3, a part of the moving object in the image is added to the converted image. May be added to the moving body stereoscopic image.

  In the fourth embodiment of the present invention, unlike the first embodiment in which the state of one moving object can be stereoscopically viewed in the moving object stereoscopic image, the state of two or more moving objects can be stereoscopically viewed in one image. A moving object stereoscopic image is generated. Even if there are two or more moving objects, the size and direction of the moving object are detected in the left viewpoint image and the right viewpoint image at each time, and the minimum position coordinates of all the detected moving objects are greater than the reference coordinates. Determine whether or not. As a result of the determination, when the minimum position coordinates are equal to or greater than the reference coordinates for all moving objects, the left viewpoint image and the right viewpoint image at that time are set as a specific left viewpoint image and right viewpoint image.

  For example, as shown in FIG. 17, the left viewpoint image and the right viewpoint image at time t <b> 1 move in the movement direction X from the right side to the left side by comparing corresponding points with the left viewpoint image and the right viewpoint image at time t <b> 0. The moving body Z1 and the moving body Z2 moving in the moving direction Y from the left side to the right side are detected. For the moving object Z1, the minimum position coordinates XminL1, XminR1 are lower than the reference coordinates XLs, XRs in both the left viewpoint image and the right viewpoint image. Also for the moving object Z2, the minimum position coordinates YminL1, YminR1 are lower than the reference coordinates YLs, YRs in both the left viewpoint image and the right viewpoint image. Therefore, since the minimum position coordinate is lower than the reference coordinate for any moving body Z1 and Z2, the left viewpoint image and the right viewpoint image are not a specific left viewpoint image and right viewpoint image. In FIG. 17, the coordinates of the movement direction Y increase from the left side to the right side.

  On the other hand, for the left viewpoint image and right viewpoint image at time t2, the minimum position coordinates XminL2, XminR2 of the moving object Z1 exceed the reference coordinates XLs, XRs, and the minimum position coordinates YminL2, YminR2 of the moving object Z2 are the reference coordinates YLs, Since YRs is exceeded, the left viewpoint image and the right viewpoint image are a specific left viewpoint image and right viewpoint image. Then, as shown in FIG. 18, the left viewpoint composite image 130 is generated by combining the specific left viewpoint images at time t0 and time t2, and the specific right viewpoint image at initial time t0 and time t2 is combined. Thus, the right viewpoint composite image 131 is generated. Then, by further synthesizing the left viewpoint synthesized image 130 and the right viewpoint synthesized image 131, a moving body stereoscopic image 132 in which the two moving bodies Z1 and Z2 can be stereoscopically viewed in one image is generated.

  In the fifth embodiment of the present invention, a moving body stereoscopic image of a subject moving in a direction approaching or moving away from the compound eye camera 10 (depth direction) is generated. In the fifth embodiment, in determining whether the left viewpoint image and the right viewpoint image at each time correspond to the specific left viewpoint image and the right viewpoint image, the minimum position coordinate of the moving object at each time is greater than or equal to the reference coordinate. As well as whether or not the amount of parallax indicating how far the same moving object is separated between the left viewpoint image and the right viewpoint image at the same time is within an allowable range. .

  For example, as illustrated in FIG. 19, a case where the moving object moves in a direction approaching the compound eye camera 10 from time t0 to time t3 will be described. First, the maximum position coordinate YmaxL0 of the moving object moving in the movement direction Y from the left viewpoint image at time t0 is set as the reference coordinate Ls, and the maximum position coordinate XmaxR0 of the moving object moving in the movement direction X from the right viewpoint image at time t0 is set. Set to reference coordinate Rs. Further, the parallax amount D0 of the moving object is obtained from the left viewpoint image and the right viewpoint image at time t0, and the maximum allowable value Dmax is obtained from the obtained parallax amount D0.

  Next, in the left viewpoint image and the right viewpoint image at each time, it is determined whether or not the minimum position coordinate of the moving object is greater than or equal to the reference coordinate, and whether or not the parallax amount of the moving object exceeds an allowable value. At time t1, the parallax amount D1 is less than the allowable amount Dmax. However, in both the left viewpoint image and the right viewpoint image at time t1, the minimum position coordinates XminL1 and XminR1 of the moving object are less than the reference coordinates XLs and XRs, and the minimum position coordinates YminL1 and YminR1 are less than the reference coordinates YLs and YRs. . Therefore, the left viewpoint image and the right viewpoint image are not selected as the specific left viewpoint image and right viewpoint image.

  In the left viewpoint image and the right viewpoint image at time t2, the parallax amount D2 is less than the allowable amount Dmax. In both the left viewpoint image and the right viewpoint image at time t2, the minimum position coordinates XminL2 and XminR2 of the moving object exceed the reference coordinates XLs and XRs, and the minimum position coordinates YminL2 and YminR2 exceed the reference coordinates YLs and YRs. ing. Therefore, the left viewpoint image and the right viewpoint image at time t2 are selected as the specific left viewpoint image and right viewpoint image.

  At the time t3 left viewpoint image and right viewpoint image, the minimum position coordinates XminL3, XminR3 of the moving object exceed the reference coordinates XLs, XRs, and the minimum position coordinates YminL3, YminR3 exceed the reference coordinates YLs, YRs. However, the parallax amount D3 exceeds the allowable amount Dmax. Therefore, the left viewpoint image and the right viewpoint image at time t3 are not selected as the specific left viewpoint image and right viewpoint image.

  From the above, among the left viewpoint image and right viewpoint image from time t0 to time t3, the left viewpoint image and right viewpoint image at time t2 are used as the specific left viewpoint image and right viewpoint image, and left at other times. The left viewpoint image and the right viewpoint image at time t0 at which the moving objects do not overlap when combined with the viewpoint image and the right viewpoint image are also selected as the specific left viewpoint image and right viewpoint image.

  Then, as shown in FIG. 20, the left viewpoint composite image 140 is generated by combining the left viewpoint images at time t0 and time t2. Further, the right viewpoint synthesized image 141 is generated by synthesizing the right viewpoint images at time t0 and time t2. Then, by further synthesizing the left viewpoint synthesized image 140 and the right viewpoint synthesized image 141, a moving body stereoscopic image 142 capable of stereoscopically viewing the moving body moving in the depth direction with respect to the compound eye camera 10 is generated. Is done. By generating the moving body stereoscopic image 142 in this way, even if the moving body moves in the depth direction instead of the moving body moving in a plane that is a fixed distance away from the compound-eye camera 10, the state of the moving body is displayed. It can be reliably grasped as a 3D image.

  In the first to fifth embodiments, the moving object stereoscopic image is generated by the compound eye camera 10 using the left viewpoint image and the right viewpoint image acquired in the strobe mode. As shown, the left viewpoint image and the right viewpoint image recorded in the memory card 20 in the strobe mode are read through the card slot 150a of the PC 150, and the moving object in the PC 150 is read using the read left viewpoint image and right viewpoint image. The stereoscopic image generation unit 151 may generate a moving object stereoscopic image. The generated moving body stereoscopic image is displayed on a display 152 provided with a lenticular sheet (all not shown) in the image display unit.

  The PC includes a storage unit that stores a moving object stereoscopic image generation program having the same function as the second stereoscopic image generation circuit 61. The 3D image strobe generation circuit 151 generates the moving object stereoscopic image generation program. A moving object stereoscopic image is generated by executing the program.

DESCRIPTION OF SYMBOLS 10 Compound eye camera 12 Left viewpoint image pick-up part 13 Right viewpoint image pick-up part 61 2nd stereo image generation circuit 70 Moving object detection part 71 Image selection part 72 Moving object stereo image generation part 75,100,120,130,140 Left viewpoint synthesis Images 76, 101, 121, 131, 141 Right viewpoint composite image 77, 102, 122, 132, 142 Moving object stereoscopic image

Claims (10)

An image acquisition step of acquiring a plurality of first viewpoint images and second viewpoint images obtained by continuously performing simultaneous imaging for capturing a moving object that is a subject moving from two different viewpoints at the same timing; ,
A specific first viewpoint image in which moving objects do not overlap when a first viewpoint image captured at another timing is synthesized from the plurality of first viewpoint images and second viewpoint images; An image selection step for selecting a specific second viewpoint image in which moving objects do not overlap when the second viewpoint image captured at the same time as the specific first viewpoint image and captured at another timing is synthesized. When,
A stereoscopic image generation step of generating a stereoscopic image in which the state of the moving object can be stereoscopically viewed within one image by combining the selected specific first viewpoint image and the specific second viewpoint image. A 3D image generation method characterized by the above. The image selection step includes:
A first motion direction detection step for detecting a motion direction of the moving object in the first viewpoint image at each imaging timing and detecting a motion direction of the moving object in the second viewpoint image at each imaging timing;
From among the plurality of first viewpoint images and second viewpoint images, a first viewpoint image in which the coordinates of a moving object relating to a moving direction satisfy a predetermined condition is selected as the specific first viewpoint image, A first selection step of selecting, as a specific second viewpoint image, a second viewpoint image that is simultaneously imaged with the specific first viewpoint image and in which the coordinates of the moving object relating to the movement direction satisfy a predetermined condition. The 3D image generation method according to claim 1. The image selection step includes:
A second moving object detection step for detecting a moving direction of the moving object in the first viewpoint image at each imaging timing;
From the plurality of first viewpoint images, a specific first viewpoint image in which the coordinates of the moving object relating to the moving direction satisfy a predetermined condition is temporarily selected and the plurality of second viewpoint images are selected from the plurality of second viewpoint images. A second selection step of selecting a second viewpoint image captured simultaneously with the temporarily selected specific first viewpoint image;
A third moving object detecting step for detecting a moving direction of the moving object in the second viewpoint image selected in the second selecting step;
From the second viewpoint image selected in the second selection step, a specific second viewpoint image in which the coordinates of the moving object regarding the movement direction satisfy a predetermined condition is selected, and the temporarily selected specific second viewpoint image is selected. A third selection step of main-selecting a specific first viewpoint image simultaneously captured with the specific second viewpoint image from one viewpoint image;
The stereoscopic image generation step includes:
A stereoscopic image is generated by combining the specific first viewpoint image selected in the third selection step and the specific second viewpoint image selected in the third selection step. The 3D image generation method according to claim 1.   The predetermined condition is that the minimum position coordinate of the moving object at the first imaging timing is equal to or greater than a reference coordinate set from the maximum position coordinate of the moving object at the second imaging timing before the first imaging timing. The 3D image generation method according to claim 2, wherein the 3D image is generated.   5. The 3D image according to claim 1, wherein the stereoscopic image generation step generates a stereoscopic image such that a moving object at a specific imaging timing is emphasized more than a moving object at another imaging timing. Image generation method.   In the stereoscopic image generation step, in addition to the selected specific first viewpoint image and the specific second viewpoint image, the first viewpoint image not selected as the specific first viewpoint image 6. A stereoscopic image is generated by combining a part and a part of a second viewpoint image that is not selected as the specific second viewpoint image. The 3D image generation method according to item.   7. The 3D image generation method according to claim 1, wherein the stereoscopic image generation step generates a stereoscopic image in which a state of a plurality of moving objects can be stereoscopically viewed in one image. A parallax amount determination step of determining whether or not a parallax amount indicating how far the same moving object is separated in the first viewpoint image and the second viewpoint image simultaneously captured exceeds a preset allowable value; Have
The stereoscopic image generation step generates a stereoscopic image by synthesizing only images having a parallax amount less than an allowable value out of the selected specific first viewpoint image and the specific second viewpoint image. The 3D image generation method according to claim 1, wherein   9. A 3D image generation program that causes a computer to execute each step in the 3D image generation method according to claim 1. Image acquisition means for acquiring a plurality of first viewpoint images and second viewpoint images obtained by continuously performing simultaneous imaging of moving objects that are moving subjects from two different viewpoints at the same timing; ,
A specific first viewpoint image in which moving objects do not overlap when a first viewpoint image captured at another timing is synthesized from the plurality of first viewpoint images and second viewpoint images; Image selection means for selecting a specific second viewpoint image that does not overlap moving objects when the second viewpoint image captured at the same time as the specific first viewpoint image and at another timing is synthesized. When,
3D image generation means for generating a stereoscopic image in which the state of the moving object can be stereoscopically viewed in one image by synthesizing the selected specific first viewpoint image and the specific second viewpoint image. A 3D image generation apparatus characterized by that.

Images