JP2013021538A - Three-dimensional image creation device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a high quality three-dimensional image without using a 3D lens.SOLUTION: On the basis of plural temporally successively captured images, a moving subject in the images is specified, a left-eye image and a right-eye image of the moving subject for stereoscopically viewing the moving subject are respectively extracted from the plural images, an image obtained by removing the moving subject is created on the basis of the plural images as a background image, and a left-eye whole image formed by combining the left-eye image with the background image and a right-eye whole image formed by combining the right-eye image with the background image are created.

Description

  The present invention relates to an apparatus, method, and program for creating a three-dimensional image from a two-dimensional image.

  As a method for creating a pseudo three-dimensional image (also referred to as a stereoscopic image, a 3D image, a stereo image, or the like) from a two-dimensional image, for example, there is one described in Patent Document 1. This is a two-dimensional image composed of a background and a foreground image of a person, etc., in which only the foreground image is shifted horizontally and the position of the foreground image in the horizontal direction is different (right-eye / left-eye whole image). Is to create. According to this, a three-dimensional image can be obtained without using a dedicated 3D lens.

JP 7-182533 A

  However, in the method of Patent Document 1, since the same foreground image is simply shifted, the perspective of the background and the foreground image can be obtained in the completed three-dimensional image, but the foreground image itself cannot be stereoscopically viewed and is uncomfortable. It becomes a strong three-dimensional image.

The three-dimensional image creating apparatus according to the present invention is a method for specifying a moving subject in an image based on a plurality of images that are continuously captured in time, and for stereoscopically viewing the moving subject from the plurality of images. Extracting means for extracting the left eye image and right eye image of the moving subject, background image creating means for creating an image from which the moving subject is removed based on a plurality of images, and left eye for the background image And a synthesis means for creating an entire image for the left eye formed by combining the images and an entire image for the right eye formed by combining the image for the right eye with the background image.
A 3D image creation method according to the present invention is a method of creating a 3D image from a plurality of images captured continuously in time, and the step of specifying a moving subject in the image based on the plurality of images Extracting a left-eye image and a right-eye image of the moving subject for stereoscopic viewing of the moving subject from a plurality of images, and an image obtained by removing the moving subject based on the plurality of images as a background image And a step of creating an entire image for the left eye formed by combining the image for the left eye with a background image, and an entire image for the right eye formed by combining the image for the right eye with the background image. To do.
The three-dimensional image creation program according to the present invention is for causing a computer to execute the above steps.

  According to the present invention, a high-quality three-dimensional image can be obtained without using a 3D lens.

The control block diagram of the digital camera in one Embodiment of this invention. The flowchart which shows the procedure of the three-dimensional image creation process performed with CPU of a camera. The figure which shows the mode at the time of video recording. The figure which shows the continuous image obtained by video recording. The figure which shows the extraction method of the image for left eyes of a moving subject. The figure which shows the extraction method of the image for left eyes / right eyes of a moving subject. The figure explaining the creation method of the image for left eyes / right eyes of the object which moves diagonally. The figure which shows the created background image.

An embodiment when the present invention is applied to a digital camera will be described with reference to the drawings.
In FIG. 1, a photographing lens 11 is a normal (non-3D) lens having only a single optical system. The subject light flux that has passed through the photographic lens 11 is imaged by the image sensor 12, and the image signal is input to the image processing unit 13. The image processing circuit 13a constituting the image processing unit 13 performs various processes on the input imaging signal to generate image data. The image data is displayed as an image on the liquid crystal monitor 3 through processing by the display circuit 13b. The liquid crystal monitor 3 is provided on the back surface of the camera, for example.

  When the shooting mode is set, the above imaging is repeated, and an image based on the image data obtained each time is sequentially updated and displayed on the liquid crystal monitor 3. This sequential update display is called live view display or live view display, and the photographer determines the composition while viewing the display image. When a shooting instruction is given, the image is taken again, and the image data generated thereby is recorded on the recording medium 15 such as a memory card by the recording / reproducing circuit 13c as an image file that can be handled by a computer. . Note that the camera of this embodiment can also shoot moving images.

  In the reproduction mode, the image data (still image, moving image) recorded on the recording medium 15 is read by the recording / reproducing circuit 13c, and is displayed as an image on the liquid crystal monitor 3 through processing by the image processing circuit 13a and the display circuit 13b. be able to.

  In response to an input from the operation unit 17, the CPU 16 controls the image processing unit 13, the AF drive circuit 18, and other circuits (not shown) to perform various operations including shooting. The AF driving circuit 18 drives the AF motor 19 in accordance with a command from the CPU 16 and drives the focus lens of the photographing lens 11. The operation unit 17 includes a release button, a moving image button, a power button, and other plural operation members.

The camera of the present embodiment can set a three-dimensional image shooting mode. This mode creates a three-dimensional image from a two-dimensional image obtained by imaging, and comprises the following procedure.
(1) Acquisition of a plurality of images continuous in time (2) Identification of moving subject in image (3) Extraction of left-eye / right-eye image of moving subject (stereoscopic target object) (4) Creation of background image ( 5) Composition of background image and left eye / right eye image All these processes are executed by the CPU 16 (see the flowchart of FIG. 2).

  The continuous image (1) is acquired by moving image shooting. For example, when the photographer presses the recording button, the video recording may be started, and when the recording button is pressed again, the video recording may be ended, or the video shooting may be performed for a preset time by a predetermined operation. Good. Further, continuous shooting (high-speed continuous shooting is desirable) may be used instead of moving image shooting.

  FIG. 3 is a diagram showing a situation during moving image shooting, and it is assumed that a car 51 travels from the left to the right in the figure and a person 52 walks to the left beyond the side. The photographer does not move during moving image shooting and does not move the camera 100 as much as possible (does not follow the moving subject in the screen). If possible, it is desirable to shoot a moving image while fixing the camera 100 to a tripod or the like. In moving image shooting for several seconds, for example, as shown in FIG. 4, a plurality of temporally continuous images (including a car 51, a person 52, and a background) are obtained. Although only three images are shown in FIG. 4, there are actually a plurality of halfway images between these images. These images are temporarily stored in the memory.

The identification of the moving subject in (2) is processing for performing a known moving object detection on the continuous image obtained in (1) and identifying the moving subject in the image. The images are compared, and if the subject position is different, the subject is recognized as a moving subject. Here, the moving subject to be identified is a subject moving in the horizontal / horizontal direction (correctly, a subject having a lateral movement component), and is not only a subject moving right side but also diagonally forward or diagonally. A subject that moves to the back is also recognized as a moving subject. However, it is desirable to exclude moving subjects with large vertical movement components (subjects whose altitude changes greatly).
In the example of FIG. 4, the car 51 and the person 52 are recognized as moving subjects.

  The extraction of the left eye / right eye image in (3) is a process of extracting the left eye image / right eye image of the moving subject from the continuous image obtained in (1). That is, in order to obtain a three-dimensional image, an image when the object to be stereoscopically viewed is viewed with the left eye (left-eye image) and an image when viewed with the right eye (right-eye image) are required. In this embodiment, all moving subjects in the image are set as stereoscopic targets, and the following processes (3-1) and (3-2) are performed on each moving subject.

  (3-1) A portion of the moving subject is cut out from all the images in which the moving subject is captured among the continuous images obtained in (1). The clipping is performed by using a technique such as edge detection. If the target moving subject continues to move in the same direction, the plurality of cut out images are all images of the same moving subject with different viewpoints, and the cutout positions in the original image are all different. Each clipped image is temporarily stored in the memory in association with information regarding the clip position in the original image.

(3-2) From the images (cutout images) of a plurality of target moving subjects stored in (3-1), an image in which the moving subject is captured at viewpoints (angles) suitable for the left eye and the right eye, respectively. The extracted images are left eye images and right eye images, respectively.
In the example of FIG. 4, an image for the left eye / right eye of the car 51 and an image for the left eye / right eye of the person 52 are obtained.

A specific method of extracting the left eye / right eye image will be described with reference to FIG.
FIG. 5 shows an example in which an image for the left eye of the car 51 is extracted. In the figure, L0 is the position of the left eye of the person, C0 is the position of the camera (photographing lens), and C1 is the position (lateral position) on the image where the car is to be placed in the composition processing described later. Now, when the car 51 located at the C1 position is viewed with the left eye, it looks like 51A. The image of the car 51 that looks closest to 51A is the left-eye image to be extracted. This is an image of a car cut out from an image captured when the car is at an angle position (L1 position) equivalent to the angle at which the left eye sees the car 51 at the C1 position (angle L0-C1). Therefore, the amount to be obtained is the distance Wans (number of pixels) between C1 and L1 on the image,
Wans = (32.5 × (W / 2)) / (D × (tan (α / 2)))
Is obtained.

  Here, α is the horizontal angle of view of the photographic lens 11, which can be obtained from the focal length of the photographic lens. D is the distance (mm) to the moving subject (car 51), which can be obtained from the result of so-called multipoint ranging. W is the width (number of pixels) of each original image obtained by moving image shooting, and this is a numerical value determined in advance according to the image quality mode. 32.5 (mm) is half the human average eye-to-eye distance.

  When Wans is obtained, an image of the vehicle 51 cut out from the position closest to the L1 position that is distant by Wans to the right from the position C1 where the vehicle 51 should be placed (the cut-out in which information indicating the position closest to the L1 position is associated) Image) is extracted as a left-eye image.

  Similarly for the right-eye image, the distance between C1 and R1 (see FIG. 6) is obtained by the above equation, and the cut-out image of the car 51 at the position closest to the R1 position is extracted as the right-eye image.

  Here, in order to realize a stereoscopic view without a sense of incongruity using the left-eye / right-eye images selected by the above method, it is necessary that the moving subject moves linearly at a constant speed during moving image shooting in (1). desirable. In this regard, as described above, since moving image shooting may be performed in a very short time, there is a high possibility that the moving subject is moving linearly at a constant speed within the short time.

  By the way, it is assumed that the left-eye image and the right-eye image for stereoscopic viewing are subject images having the same shape. The car 51 does not change its shape during movement, but the person 52 walks while moving their limbs, so the shape (posture) during movement is not constant. Accordingly, the shapes of the left-eye / right-eye images (cut-out images) selected in (3-2) above are compared, and if the two shapes are significantly different, either the left or the right is selected again. Since the shape change when the person walks is periodic, the shape of the left-eye / right-eye image can be made uniform by reselection. For example, the left-eye image is used as it is, and the right-eye image is almost the same shape as the left-eye image, and a cut-out image associated with the information at the position closest to the R1 position is selected again. Note that in the case of an object that moves irregularly, the shape of the left-eye / right-eye image cannot be made uniform, and such an object may be excluded from the target of stereoscopic vision.

  Also, it is assumed that the left-eye image and the right-eye image for stereoscopic viewing are the same size. There is no problem with a moving subject that moves to the side, but in the case of a moving subject that moves diagonally forward or diagonally backward, the left-eye image and the right-eye image obtained by the above method are not the same size. Therefore, as (3-3), the size of the image for the left eye and the image for the right eye finally selected in the above (3-2) are compared. Add a process with the same size.

  As an example of (3-3), as illustrated in FIG. 7, consider that a moving subject 53 that moves from the left back to the right front (or vice versa) along the horizontal plane is disposed at the FC position. The left-eye image of the moving subject 53 is an image of the moving subject 53 positioned on the right side (near) from the FC position, and the right-eye image is an image of the moving subject 53 positioned on the left side (far) from the FC position. Become. Therefore, the left-eye image is too large as a clipped image to be arranged at the FC position, and the right-eye image is too small. Therefore, the left-eye image is reduced and the right-eye image is enlarged, so that the sizes of both are made uniform. In the case of a moving subject moving from the right back to the left front (or vice versa), the left-eye image is enlarged and the right-eye image is reduced contrary to the above.

  Here, the lateral arrangement position (the C1 position and the FC position) of the moving subject that is a stereoscopic target object may be automatically determined by the CPU 16 or may be selected by the photographer. When the camera determines, for example, the center of the moving range of the moving subject in the moving image taken in (1) is set as the arrangement position of the moving subject. According to this, for example, a moving subject that completely crosses the screen, such as the car 51 in FIG. 4, is arranged at the center of the image. The determined arrangement position is reflected in the processing of (3-2).

On the other hand, when the photographer decides the arrangement position of the moving subject, before the processing of (3-2), any image of the moving image captured in (1) is displayed on the liquid crystal monitor 3, and a pointing device or a cursor is displayed. When the photographer designates a position using, the position (lateral position) is stored as the arrangement position of the moving subject. The arrangement position can be specified for each moving subject. The stored arrangement position is reflected in the process (3-2).
Note that the arrangement position of the moving subject in the vertical direction is naturally determined from the original image.

The creation of the background image in (4) is a process of creating an image of only the background (background image) excluding the moving subject by synthesizing the continuous images obtained in (1) (see FIG. 8). In a certain image, there is no background image data for the portion where the moving subject is located, but a background image can be obtained by supplementing the data for that portion from another image (an image in which the moving subject is not shown). .
Note that the order of the process (3) and the process (4) may be reversed.

  The synthesis process of (5) is similar to the process of creating the left-eye whole image by synthesizing the left-eye images of all the moving subjects created in (3) with the background image created in (4) (4 ) And the background image created in (3) are combined with the right-eye images of all the moving subjects created in (3) to create an entire right-eye image. The arrangement position of each moving subject image in the composition is a position determined by the CPU 16 or the photographer as described above.

  This completes the three-dimensional image. The combined left-eye / right-eye whole images are combined into one file, for example, as a three-dimensional image file, and the three-dimensional image file is recorded on the recording medium 15. The moving image of (1) and the cutout image of (3-1) may be deleted, but the moving image may be recorded.

  As described above, in the present embodiment, the three-dimensional image is created by extracting the left-eye / right-eye images of the moving subject from the plurality of continuous images obtained by moving image shooting or continuous shooting. As a matter of course, the moving subject itself can be stereoscopically viewed, and a high-quality (uncomfortable) three-dimensional image can be created compared to the conventional method. Moreover, since the left-eye / right-eye images are extracted for each of the plurality of moving subjects in consideration of the distance to the subject, the three-dimensional effect of each moving subject and the sense of distance between the moving subjects are also appropriate. can do. In addition, since the left-eye / right-eye image is extracted in consideration of the angle of view of the photographing lens, a high-quality three-dimensional image can be obtained regardless of the lens used. Furthermore, a subject whose shape changes during movement and a subject that moves obliquely laterally can be stereoscopically viewed without any problem.

  The perspective with the moving subject can be adjusted by shifting the horizontal position of the background (including the foreground) left and right. By acquiring the distance of each part of the background by distance measurement and changing the shift amount and the shift direction of each background part accordingly, a more authentic three-dimensional image can be obtained.

  Although the above has been described with respect to the camera, the continuous image of (1) may be taken into a computer (such as a personal computer) and the processing after (2) may be performed by the computer. In this case, each parameter (view angle and subject distance information) used in the process (3-2) may be obtained from meter data embedded in the captured image.

  In place of the process of (3), the photographer may select the left-eye / right-eye image of the moving subject while viewing each image, or the photographer can select either the left-eye / right-eye image. When selected, the CPU may extract the other.

11 photographing lens 12 image sensor 16 CPU
51 Moving subject (car)
52 Moving subject (person)

Claims (9)

A specifying means for specifying a moving subject in an image based on a plurality of images taken continuously in time;
Extraction means for extracting a left-eye image and a right-eye image of the moving subject for stereoscopically viewing the moving subject from the plurality of images,
Background image creation means for creating an image from which the moving subject is removed as a background image based on the plurality of images;
And a synthesis means for creating an entire image for the left eye formed by combining the image for the left eye with the background image, and an entire image for the right eye formed by combining the image for the right eye with the background image. 3D image creation device.   The extraction means performs processing for extracting the left-eye image and right-eye image for each identified moving subject, and the combining means moves the left-eye whole image and the right-eye whole image, respectively. The three-dimensional image creating apparatus according to claim 1, wherein the three-dimensional image creating device is created for each subject.   3. The three-dimensional image creation apparatus according to claim 1, wherein the extraction unit extracts the left-eye image and the right-eye image based on a distance to the moving subject.   The three-dimensional image creation apparatus according to claim 1, wherein the extraction unit extracts the left-eye image and the right-eye image based on an angle of view at the time of imaging.   5. The left eye image and the right eye image are extracted by the extraction unit so that the shapes of the moving subjects of the left eye image and the right eye image are substantially the same. 6. The three-dimensional image creation apparatus described in 1.   When the size of the extracted left-eye image and right-eye image is different, the synthesis unit further includes a scaling unit that changes a scaling ratio of the left-eye image and the right-eye image so as to have substantially the same size. 6. The three-dimensional image creation apparatus according to claim 1, wherein the left-eye image and the right-eye image after the scaling ratio change are used for the synthesis.   The three-dimensional image creation apparatus according to claim 1, further comprising an imaging unit that acquires the plurality of temporally continuous images by imaging. A method of creating a three-dimensional image from a plurality of images that are continuously captured in time,
Identifying a moving subject in the image based on the plurality of images;
Extracting a left-eye image and a right-eye image of the moving subject for stereoscopic viewing of the moving subject, respectively, from the plurality of images;
Creating a background image from which the moving subject is removed;
And a step of creating an overall image for the left eye formed by combining the image for the left eye with the background image, and an entire image for the right eye formed by combining the image for the right eye with the background image. Original image creation method.   A three-dimensional image creation program for causing a computer to execute each step according to claim 8.

Images