JP2012165247A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a stereoscopic image with a simple configuration.SOLUTION: An image processing device 1 comprises: display means 15 that reproduces and displays an image; image composition means 16 that, based on respective feature amounts of images of plural frames continuously photographed, combines the images of the plural frames to compose one image; display control means 16 that causes the display means 15 to reproduce and display the image composed by the image composition means 16; an operation member 20 for selecting a predetermined area in the composed image during reproduction display; and image extraction means 16 that extracts, from among the images of the plural frames, two images which are images of two frames respectively including the area indicated by a selection operation signal from the operation member 20 and which correspond to a predetermined base line length.

Description

  The present invention relates to an image processing device, a photographing device, and an image processing program.

  A technique for generating a stereoscopically panoramic image is known (see Patent Document 1). In the prior art, two consecutive frame images are compared, the magnitude of the optical flow at each pixel on two reference lines that are equidistant from the center line that bisects the reference frame image is calculated, and the optical flow The width of the slit image is determined based on the size, the slit image is cut out from the reference frame image, and the slit images for generating the panorama image for the right eye and the left eye are combined, respectively, for the right eye and the left eye Each panorama image is generated.

JP-A-11-164326

  In the prior art, there is a problem that the processing load required for the process leading to the clipping of the slit image and the joining process after the clipping is large.

  An image processing apparatus according to the present invention combines a display unit for reproducing and displaying an image, and a combination of a plurality of frames based on each feature amount of a plurality of frames that have been continuously captured to synthesize a single image. Composition means, display control means for reproducing and displaying the image synthesized by the image composition means on the display means, an operation member for selecting a predetermined region of the synthesized image being reproduced and displayed, and a selection operation signal from the operation member Image extraction means for extracting two images corresponding to a predetermined baseline length from a plurality of frames.

  According to the present invention, a stereoscopic image can be obtained with a simple configuration.

It is the figure which looked at the electronic camera carrying the image processing apparatus by one embodiment of this invention from the front. It is the figure which looked at the electronic camera of FIG. 1 from the back. It is a block diagram which illustrates the composition of an electronic camera. It is a figure which shows the electronic camera in parallel movement. It is a figure which shows the electronic camera which moves in parallel to the direction different from FIG. It is a figure which illustrates the image group from the image 1 to the image N image | photographed continuously. It is a figure which illustrates the synthesized image reproduced and displayed on the LCD monitor. It is a figure which illustrates the synthesized image and the frame which were reproduced and displayed on the LCD monitor. It is a figure which illustrates the image of 2 frames extracted. It is a flowchart explaining the flow of the process which CPU performs for acquisition of a stereoscopic vision image. It is a figure which illustrates a computer apparatus.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a front view of an electronic camera 1 equipped with an image processing apparatus according to an embodiment of the present invention. A photographing optical system 11 is disposed in front of the electronic camera 1. On the upper surface of the electronic camera 1, a power switch 20a, a photographing button 20b, and a mode switching dial 20c are provided. FIG. 2 is a view of the electronic camera 1 of FIG. 1 as viewed from the back. On the back of the electronic camera 1, an LCD monitor 15, a cross key switch 20d, a playback switch 20e, and a deletion switch 20f are provided.

  FIG. 3 is a block diagram illustrating the configuration of the electronic camera 1. In FIG. 1, an electronic camera 1 includes a photographing optical system 11, an image sensor 12, an image processing unit 13, an SDRAM 14, an LCD monitor 15, a CPU 16, a nonvolatile memory 17, and a card interface (I / F). 18, a communication interface (I / F) 19, an operation member 20, and a shake sensor 21.

  The CPU 16, the nonvolatile memory 17, the card interface 18, the communication interface 19, the image processing unit 13, the SDRAM 14, and the LCD monitor 15 are connected via a bus 25.

  The photographing optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens, and forms a subject image on the light receiving surface of the image sensor 12. In order to simplify FIG. 1, the photographing optical system 11 is shown as a single lens.

  The imaging element 12 is configured by a CMOS image sensor or the like in which light receiving elements are two-dimensionally arranged on the light receiving surface. The image sensor 12 photoelectrically converts an image of the light beam that has passed through the photographing optical system 11 to generate a digital image signal. The digital image signal is input to the image processing unit 13. The image processing unit 13 performs various types of image processing (color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment processing, etc.) on the digital image data.

  The LCD monitor 15 is composed of a liquid crystal panel or the like. The LCD monitor 15 displays an image, an operation icon, a menu screen, and the like according to an instruction from the CPU 16. The SDRAM 14 temporarily stores digital image data in a pre-process and post-process of image processing by the image processing unit 13 and is used when a program is executed by the CPU 16. The nonvolatile memory 17 is configured by a flash memory or the like. Since the stored contents are retained even when the power is turned off, the program executed by the CPU 16 is stored.

  The CPU 16 controls the operation performed by the electronic camera 1 by executing a program stored in the nonvolatile memory 17. The CPU 16 also performs AF (autofocus) operation control and automatic exposure (AE) calculation. In the AF operation, for example, an in-focus position of a focusing lens (not shown) is obtained based on the contrast information of the through image. The through image refers to a monitor image that is repeatedly acquired by the image sensor 12 at a predetermined time interval (for example, 60 frames / second) before the release operation.

  The card interface 18 has a connector (not shown), and a storage medium 30 such as a memory card is connected to the connector. The card interface 18 writes data to the connected storage medium 30 and reads data from the storage medium 30. The storage medium 30 is configured by a memory card incorporating a semiconductor memory, a hard disk drive, or the like.

  For example, the communication interface 19 performs communication using an TCP / IP protocol with an external device connected to a connector (not shown). Through this communication, commands and data from an external device are received, and image data, audio data, and the like stored in the storage medium 30 are transmitted to the external device. The operation member 20 includes operation members 20a to 20f such as a photographing button 20b and a cross key switch 20d as well as the power switch 20a described above. The operation member 20 sends an operation signal corresponding to each operation such as a photographing operation, a mode switching operation, and various selection operations to the CPU 16. The shake sensor 21 is configured by, for example, an acceleration sensor. The shake sensor 21 detects acceleration generated in the pitch direction and the yaw direction, and sends a detection signal to the CPU 16.

The present embodiment is characterized in that a stereoscopic image (so-called 3D image) is obtained with the electronic camera 1. For this reason, the following description will be focused on obtaining stereoscopic images. First, an outline of a procedure for acquiring a stereoscopic image performed in the present embodiment will be described. In the present embodiment, the electronic camera 1 repeats imaging at a predetermined frame rate, for example, 60 frames / second, and extracts two frames of images from which a stereoscopic effect is obtained from the acquired series of images. This acquisition procedure is roughly divided into the following five stages.
Stage 1. Continuous shooting stage 2.1 Combined images (preview image)
Stage 3. 3. Display preview image Region selection operation acceptance stage: Extracting 5.2 frames of images

<Stage 1>
In “stage 1”, the user causes the electronic camera 1 to perform continuous shooting. At this time, the user translates the electronic camera 1 that is acquiring a series of images as illustrated in FIG. FIG. 4 is a diagram illustrating the electronic camera 1 during translation. The user directs the photographing optical system 11 of the electronic camera 1 toward the main subject and translates it on a plane on which the distance from the main subject is substantially constant. The direction in which the electronic camera 1 is translated may be a spiral shape illustrated in FIG. 5 or a zigzag shape. FIG. 5 is a diagram showing the electronic camera 1 that translates in a direction different from that in FIG. As illustrated in FIG. 6, the electronic camera 1 that is moving in parallel primarily stores N image groups of subsequently captured images 1 to N in the SDRAM 14. In FIG. 6, image 1 is the first frame image after the start of moving image shooting, and image N is the last frame image before the end of moving image shooting.

  The CPU 16 causes the SDRAM 14 to temporarily store the acceleration signal detected by the shake sensor 21 at the time of shooting each frame as additional information of each frame image. By storing the acceleration signal, the relative position information of the electronic camera 1 when the image of each frame was captured (from the position where (m-1) frame was captured to the position where (m) frame was captured) The moving direction and moving amount of the electronic camera 1 can be calculated.

<Stage 2>
The CPU 16 of the electronic camera 1 combines one image by connecting N image groups that are primarily stored in the SDRAM 14. For example, the CPU 16 of the electronic camera 1 extracts feature amounts from image data in a predetermined area of the image 1 (for example, a predetermined range including the center of the screen), and generates template information for performing a known template matching process. The template information is, for example, reduced image data having a low resolution (about 1/9 size of an image that can be acquired by the image sensor 12).

  The CPU 16 detects a region similar to the subject of the image 1 from the image 2 by performing template matching processing using the template information on the data of the image 2. CPU16 extracts the feature-value from the image data within the predetermined range containing the said detection area among the images 2, and produces | generates template information newly. The CPU 16 detects a region similar to the subject of the image 2 from the image 3 by performing a template matching process on the data of the image 3. Thereafter, similarly, by performing template matching processing between the images of the frames preceding and following the image N, a region similar to the subject of the image of the previously acquired frame is detected from the image of the frame acquired later. The CPU 16 represents a subject area that overlaps between images as a representative image (for example, a frame image acquired previously), and combines the subject areas that do not overlap between images as appropriate to synthesize one image. By combining in this way, the combined image has a wider range than the image range of each frame. In this description, the composite image is called a preview image.

<Stage 3>
The CPU 16 reproduces and displays the preview image (composite image) obtained in “Step 2” on the LCD monitor 15. FIG. 7 is a diagram illustrating a composite image 71 reproduced and displayed on the LCD monitor 15. In this description, for example, an example in which an animal 72 shown in FIG. 7 is a main subject will be described.

<Stage 4>
CPU16 receives selection operation by a user. For example, the user selects an area including the main subject 72 in the composite image 71 by operating the cross key switch 20d and the reproduction switch 20e. This selection range corresponds to the range of a stereoscopic image (so-called stereo image). Based on the selection operation signal, the CPU 16 displays a frame 81 indicating the selection area so as to overlap the composite image 71. FIG. 8 is a diagram illustrating a composite image 71 and a frame 81 reproduced and displayed on the LCD monitor 15.

<Stage 5>
The CPU 16 detects an image including the main subject 72 from the N image groups. For example, the CPU 16 extracts feature amounts from image data within a selection range surrounded by a frame 81 in the composite image 71, and generates template information. The template information is the same as that described above. The CPU 16 detects a frame image having a subject area similar to the subject in the frame 81 by performing template matching processing using the template information for N image groups.

  Assuming that p frames having a subject area similar to the subject in the frame 81 are detected, the CPU 16 extracts two frames of images that can ensure a parallax corresponding to the baseline length from the p images. The baseline length is, for example, 65 mm corresponding to the distance (eye width) between the left and right eyes of a person. The CPU 16 calculates the shooting position of each frame based on the additional information described above, and extracts two frames of images whose shooting positions are 65 mm apart in the horizontal direction. FIG. 9 is a diagram illustrating two extracted frames. Image X is a frame image corresponding to the right eye, and image Y is a frame image corresponding to the left eye. The stereoscopic parallax direction obtained from the extracted two-frame images is the horizontal direction.

  A flow of processing executed by the CPU 16 for acquiring the stereoscopic image described above will be described with reference to a flowchart illustrated in FIG. When the on-operation signal of the power switch 20a is input, the CPU 16 activates a program that executes the processing shown in FIG.

  In step S10 of FIG. 10, the CPU 16 proceeds to step S20 when an operation signal indicating depression of the photographing button 20b is input. In step S20, the CPU 16 determines whether or not the “stereo shooting mode” is set. For example, when an operation signal indicating the “stereo shooting mode” is input from the mode switching dial 20c constituting the operation member 20, the CPU 16 makes a positive determination in step S20 and proceeds to step S30. If the operation signal indicating the “stereo shooting mode” is not input from the mode switching dial 20c, the CPU 16 makes a negative determination in step S20 and proceeds to step S130. In step S130, the CPU 16 performs a shooting process in another shooting mode (for example, a still image shooting mode) corresponding to the operation signal from the mode switching dial 20c.

  In step S <b> 30, the CPU 16 starts the moving image capturing and storing process in the “stage 1”, and proceeds to step S <b> 40. In step S40, the CPU 16 determines whether or not the photographing is finished. The CPU 16 makes a positive determination in step S40 when the operation signal for pressing the shooting button 20b is input again, ends the moving image shooting and saving process, and proceeds to step S50. CPU16 makes negative determination in step S40, and returns to step S30, when the pressing-down operation signal of imaging | photography button 20b is not input again. When returning to step S30, the moving image capturing and storing process is continued. Through the above processing, the N image groups from image 1 to image N, which are continuously captured, are temporarily stored in the SDRAM 14.

  In step S50, the CPU 16 performs a preview image generation process (synthesizing process) in the “stage 2”, and proceeds to step S60. In step S60, the CPU 16 performs the preview image display process in the “stage 3” and proceeds to step S70. In step S70, the CPU 16 adjusts the inclination of the preview image. For example, the CPU 16 rotates the composite image 71 illustrated in FIG. 7 in accordance with an operation signal from the cross key switch 20d, and displays the rotated composite image on the LCD monitor 15. The rotation center is the approximate center of the composite image 71 and is rotated about the approximate center point. The reason for tilting the preview image in this way is to make it possible to set the stereoscopic parallax direction obtained from the two-frame image extracted in step S110 to be different from the horizontal direction. When the CPU 16 adjusts the inclination of the preview image as described above, the process proceeds to step S80. When the operation signal from the cross key switch 20d is not input, the CPU 16 skips step S70 and proceeds to step S80.

  In step S80, the CPU 16 accepts the region selection operation in the “stage 4” and proceeds to step S90. This area corresponds to the range of a stereoscopic image (so-called stereo image). As illustrated in FIG. 8, the position and size of the frame 81 indicating the selected area on the display screen of the LCD monitor 15 are manipulated by the user. To be decided. For example, the user moves the cursor 82 by operating the cross key switch 20d. For example, the CPU 16 stores coordinates indicating the position of the cursor 82 being displayed at that time in response to a pressing operation signal of an OK switch (not shown). When the CPU 16 stores the coordinates of the two points, the rectangle having the two points as a diagonal is defined as the frame 81. The area surrounded by the frame 81 is the selection range.

  In step S90, the CPU 16 calculates a baseline length and proceeds to step S100. In the present embodiment, the base line length is 65 mm as described above. In step S100, the CPU 16 determines whether or not the selection range is a range in which a stereoscopic image (so-called stereo image) can be generated. The CPU 16 makes an affirmative determination in step S100 and proceeds to step S110 when there is an image that can ensure the parallax corresponding to the baseline length for the selection range. If there is no image that can ensure the parallax corresponding to the baseline length for the selected range, the CPU 16 makes a negative determination in step S100 and returns to step S80. To return to S80, for example, “Please select an area again”. Is displayed on the LCD monitor 15 to prompt the user to perform another operation.

  In step S110, the CPU 16 extracts the two-frame images in the “stage 5”, and proceeds to step S120. As a result, a two-frame image having a subject area similar to the subject in the selection range surrounded by the frame 81 and capable of securing a parallax corresponding to the baseline length is extracted. For example, the CPU 16 displays two frames of images alternately at a predetermined display rate instead of the preview image being displayed on the LCD monitor 15. The user can check the stereoscopic image displayed on the LCD monitor 15.

  In step S120, the CPU 16 sends an instruction to the card interface 18, stores the extracted two-frame image data in the storage medium 30 as stereo image data, and ends the process of FIG.

According to the embodiment described above, the following operational effects can be obtained.
(1) The electronic camera 1 synthesizes a single image by combining a plurality of frames based on the LCD monitor 15 that reproduces and displays the images and the feature quantities of the continuously shot frames. CPU 16, CPU 16 that reproduces and displays the synthesized image on LCD monitor 15, operation member 20 for selecting a predetermined area from the synthesized image being reproduced and displayed, and an area indicated by a selection operation signal from operation member 20 Since the CPU 16 that includes two frames each including two images corresponding to a predetermined baseline length is extracted from a plurality of frames, a stereoscopic image can be obtained with a simple configuration.

(2) In the electronic camera 1 of the above (1), the image file storing the data of the plurality of frames includes first information indicating the shooting position of the electronic camera 1 detected at the time of shooting each image, Based on the first information, the CPU 16 extracts two frames of images shot at two shooting positions that are separated by the baseline length in the horizontal direction. Thereby, a stereoscopic image can be obtained without using a twin-lens camera.

(3) Since the electronic camera 1 of (1) or (2) further includes the CPU 16 that generates the second information for associating the two frames of images extracted by the CPU 16, it is not necessary to process each image. Thereby, it is possible to obtain a stereoscopic image while keeping a plurality of frames taken continuously.

(4) In the electronic camera 1 of the above (3), the CPU 16 displays a stereoscopic image using two frames of image data on the LCD monitor 15 based on the second information, so a file dedicated to the stereoscopic image is generated. Therefore, the stereoscopic image can be confirmed without reading the file dedicated to the stereoscopic image.

(5) The electronic camera 1 includes an image sensor 12 that captures a subject image, a shake sensor 21 that detects a shooting position of the electronic camera 1, image data captured by the image sensor 12, and a shooting position detected by the shake sensor 21. The CPU 16 generates an image file including information indicating the image, the LCD monitor 15 that displays a reproduction image based on the image data, and the images of the plurality of frames based on the respective feature amounts of the images of the plurality of frames that are subsequently captured A CPU 16 for combining one image in combination, a CPU 16 for reproducing and displaying the combined image on the LCD monitor 15, an operation member 20 for selecting a predetermined area of the combined image being reproduced and displayed, and an operation member 20 images of two frames each including the area indicated by the selection operation signal from 20 and captured at two shooting positions that are separated from each other by the baseline length in the horizontal direction based on the information An image of two frames so and a CPU16 to be extracted from among the plurality of frames of images, it is possible to obtain a stereoscopic image with a simple configuration.

(Modification 1)
In the above description, the camera position information (the moving direction and amount of movement of the electronic camera 1 from the position where the (m-1) frame is captured to the position where the (m) frame is captured) is used as the detection signal by the shake sensor 21. Based on this, the relative position is calculated. Instead, a configuration may be adopted in which radio waves from GPS satellites are received and calculated as absolute positions.

(Modification 2)
Although an example in which N image groups of subsequently captured images 1 to N are temporarily stored in the SDRAM 14 has been described, it may be stored in the storage medium 30 as a moving image data file. The shooting position information at the time of shooting each frame is included in the stored content as additional information of the image of each frame, similar to the above-described example. By configuring so that a preview image can be generated based on moving image data stored in the storage medium 30, it is possible to select an image capable of stereoscopically viewing an arbitrary subject from the captured moving image data. become.

(Modification 3)
Further, when data of N image groups from successively captured images 1 to N is stored, the data may be stored as one file or a plurality of files. If you want to save multiple files separately, for example, save them in the same folder, or add information that correlates them to each other, so that you can see that it is a series of images that were taken continuously. .

(Modification 4)
If the movement amount of the electronic camera 1 from the position where the (m-1) frame is captured to the position where the (m) frame is captured among the images 1 to N that are continuously captured, m) Saving of frame images may be omitted.

(Modification 5)
Although the example in which the base line length is 65 mm has been described, the base line length value may be configured to be changeable.

(Modification 6)
In step S110 of FIG. 10, even when there is no image of a frame whose shooting position is 65 mm apart in the horizontal direction, if there are an image whose shooting position is 60 mm away and an image that is 70 mm away, these shootings are performed. A composite image generated based on an image that is 60 mm apart and an image that is 70 mm apart may be used as one of the extracted images. As in the second modification, when selecting an image capable of stereoscopically viewing an arbitrary subject from the captured moving image data, it is difficult to recapture the moving image. Even if there is no frame image corresponding to an appropriate baseline length, an image that can be viewed stereoscopically can be appropriately selected.

(Modification 7)
In the embodiment described above, the example in which the image processing apparatus is mounted on the electronic camera 1 has been described. However, the image processing apparatus may be configured by a personal computer. In the flowchart illustrated in FIG. 10, the image processing apparatus may be configured by causing the computer apparatus 100 illustrated in FIG. 11 to execute a program that performs the processes after step S50. When the program is used by being loaded into the personal computer 100, the program is loaded into the data storage device of the personal computer 100 and then used as an image processing device by executing the program.

  The loading of the program to the personal computer 100 may be performed by setting a storage medium 104 such as a CD-ROM storing the program in the personal computer 100, or to the personal computer 100 by a method via the communication line 101 such as a network. You may load. When passing through the communication line 101, the program is stored in the hard disk device 103 of the server (computer) 102 connected to the communication line 101. The program can be supplied as various forms of computer program products such as provision via the storage medium 104 or the communication line 101.

  According to the modification 7 demonstrated above, the effect similar to the said embodiment is acquired. In addition to the configuration using a personal computer, a digital photo frame, a projector, or the like may be configured to generate the above-described stereo image.

  The above description is merely an example, and is not limited to the configuration of the above embodiment.

DESCRIPTION OF SYMBOLS 1 ... Electronic camera 12 ... Image pick-up element 13 ... Image processing part 14 ... SDRAM
15 ... LCD monitor 16 ... CPU
17 ... Nonvolatile memory 30, 104 ... Storage medium 100 ... Personal computer

Claims (6)

Display means for reproducing and displaying images;
Image combining means for combining the images of the plurality of frames and combining the images of the images based on the feature amounts of the images of the plurality of frames continuously captured;
Display control means for reproducing and displaying the image synthesized by the image synthesis means on the display means;
An operation member for selecting a predetermined region of the composite image being reproduced and displayed;
Image extraction means for extracting two images corresponding to a predetermined baseline length from the plurality of images, each of which is a two-frame image including a region indicated by the selection operation signal from the operation member;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image file storing the image data of the plurality of frames includes first information indicating the shooting position of the shooting device detected at the time of shooting each image,
The image processing device is characterized in that the image extraction device extracts two frames of images shot at two shooting positions that are separated from the baseline length in the horizontal direction based on the first information. The image processing apparatus according to claim 1 or 2,
An image processing apparatus, further comprising: information generation means for generating second information for associating two frames extracted by the image extraction means. The image processing apparatus according to claim 3.
The display control means causes the display means to display a stereoscopic image using the image data of the two frames based on the second information. Imaging means for capturing a subject image;
Position detecting means for detecting the photographing position of the apparatus;
A file generating unit that generates an image file including image data captured by the imaging unit and information indicating a shooting position detected by the position detecting unit;
Display means for displaying a reproduced image based on the image data;
Image combining means for combining the images of the plurality of frames and combining the images of the images based on the feature amounts of the images of the plurality of frames continuously captured;
Display control means for reproducing and displaying the image synthesized by the image synthesis means on the display means;
An operation member for selecting a predetermined region of the composite image being reproduced and displayed;
A plurality of images of two frames each including a region indicated by a selection operation signal from the operation member, each imaged at two shooting positions separated by a base line length in the horizontal direction based on the information. Image extracting means for extracting from the frame images;
An imaging apparatus comprising: A first process for combining a plurality of frames of images and combining the images based on the feature amounts of the images of the frames shot continuously;
A second process for reproducing and displaying the image synthesized in the first process on a display device;
A third process for accepting an operation for selecting a predetermined region of the composite image being reproduced and displayed;
A fourth process of extracting two images corresponding to a predetermined baseline length from the plurality of images, each of which is a two-frame image including the region indicated by the selection operation signal received in the third process;
An image processing program for causing a computer to execute.

Images