JP4995966B2 - Image processing apparatus, method, and program - Google Patents

Description

  The present invention relates to a list display of reduced images of multi-view images and single-view images.

  In a general stereoscopic image display system, an image drawn with the right eye as a viewpoint is individually displayed for the right eye, and an image drawn with the left eye as a viewpoint is individually presented for the left eye. Usually, in such two images with the right eye and the left eye as viewpoints (hereinafter referred to as multi-viewpoint images), an object located far from the viewpoint looks almost parallel to both eyes. Since they are almost the same, they are drawn at almost the same position. On the other hand, objects close to the viewpoint are drawn at positions shifted from each other because the lines of sight from both eyes are non-parallel and look different. When an image with such parallax (hereinafter referred to as a parallax image) is viewed with the right eye and the left eye, the observer perceives a sense of perspective with respect to the object.

  With regard to the generation of parallax images, a photorealistic method of photographing a subject using two left and right cameras and viewing the obtained parallax images three-dimensionally using an optical illusion and computer graphics technology are used. In addition, there is a three-dimensional computer graphics (three-dimensional CG) display that generates a parallax image by visual field conversion of model data with the left and right eyes as viewpoints. The three-dimensional computer graphics display is commonly called a virtual three-dimensional display or a pseudo three-dimensional display.

  In the photorealistic method, a parallax image is stereoscopically displayed using a parallax barrier type or lenticular lens type 3D monitor. In the parallax barrier type, a parallax barrier having a pattern in which light transmitting portions and light shielding portions are alternately arranged at a predetermined pitch is generated on the parallax barrier display layer, and left and right images are displayed on the lower image display surface. The strip-shaped image fragments indicating “” are alternately arranged and displayed to enable stereoscopic viewing of the parallax image.

  Alternatively, the realistic method employs a lenticular method using a lenticular lens sheet, an integral photography method using a microlens array sheet, a holography method using an interference phenomenon, and the like.

  In general computer graphics, an object defined in a three-dimensional space is represented by a set of a plurality of polygons. Usually, the simplest triangle is often used as the polygon. Each polygon that composes an object is coordinated from its own object coordinates to three-dimensional coordinates based on the position of the observer's viewpoint, that is, visual field coordinates, using mathematical methods such as vectors and matrices. Conversion is performed. Further, for display on the display device, coordinate transformation (perspective transformation) for perspectively projecting a polygon represented by visual field coordinates onto a two-dimensional plane is performed. The final polygon position data after these series of coordinate transformations is performed, the Z value indicating the distance from the viewpoint in the direction perpendicular to the display (Z axis), and the position coordinates in the display surface X value and Y value representing

  Regarding the method for determining the color values of the pixels constituting each polygon, a virtual light source is arranged at the visual field coordinates, and the intensity of light emitted from this light source to each vertex constituting the polygon is physically calculated. There is a method of obtaining the light intensity at each pixel position by linear interpolation from the values in the rasterizing process in the polygon.

Alternatively, as disclosed in Patent Document 1, there is a method of pasting an image representing a texture or special pattern of an object surface called a texture on a polygon surface. In this case, texture coordinates in addition to position coordinates are given to each vertex of the polygon constituting the object at the stage of object coordinates before coordinate conversion. As in the case of the light intensity described above, the texture coordinates are linearly interpolated for each pixel position in the polygon rasterization process, and the pixel color value is determined according to the obtained texture coordinates.

  A graphics image viewed from a specific viewpoint can be generated by the above procedure, but in order to generate a parallax image for stereoscopic viewing as described above, the viewpoint position is set to the right eye and the left eye. A multi-viewpoint image is generated by drawing the same object twice, replacing each position of the eye. An expensive system includes a display device capable of realizing the above-described realistic technique, and generates a multi-viewpoint image.

  On the other hand, conventionally, when a multi-viewpoint image and a normal image without parallax are mixed and stored in a storage device, there is a technique for displaying these reduced images in a distinguishable manner. The electronic device disclosed in Patent Document 2 includes a display device that can realize a photorealistic technique. When displaying a list of thumbnail images in a planar manner, a symbol “3D” is added to a thumbnail image corresponding to a three-dimensional image. In addition, the thumbnail image corresponding to the two-dimensional image is represented by a circle or an ellipse, and the symbol “2D” is added, and the thumbnail image is represented by a triangle, the thumbnail created from the two-dimensional image and the thumbnail created from the three-dimensional image Can be distinguished.

JP 2002-24850 A JP 2004-120165 A

  In Patent Document 2, 2D images and 3D image thumbnails are represented by symbols such as “3D” and “2D”, and shapes such as a circle, an ellipse, and a triangle. Hard to do.

  Further, in Patent Document 2, it is essential to arrange a display device that can display a three-dimensional image by a realistic method, and display switching between a two-dimensional image and a three-dimensional image is essential. There is a disadvantage that the cost becomes high.

  Therefore, the present invention realizes an apparatus that allows a user to easily and intuitively identify a representative image of a multi-viewpoint image from representative images of a captured image with a simple configuration.

  An image processing apparatus according to the present invention inputs an image input unit that inputs a multi-viewpoint image captured from two or more viewpoints and an image including a single-viewpoint image captured from a single viewpoint, and is input to the image input unit. A storage unit that stores an image, a reduced image generation unit that generates a first reduced image obtained by reducing the image stored in the storage unit, and an image stored in the storage unit is a multi-viewpoint image or a single-viewpoint image An image identification unit for identifying the image and a first reduced image corresponding to the image identified by the image identification unit as a multi-viewpoint image are subjected to three-dimensional computer graphic processing to create a second reduced image And a first reduction created by the reduced image creation unit as a representative image of the image identified by the image identification unit as a single-viewpoint image in response to an instruction to display a list of images stored in the storage unit. List images and Includes a display unit for image identification section lists the second reduced image created by the image processing unit as a representative image of images identified as multi-viewpoint image.

  According to the present invention, reduced images subjected to three-dimensional computer graphic processing are displayed as a list of representative images of multi-view images, and simple reduced images are displayed as a list of representative images of single-view images. Since the representative image of the multi-viewpoint image is subjected to three-dimensional computer graphic processing, the difference from a simple reduced image can be seen at a glance. Further, it is not necessary to use a special photorealistic method for displaying a representative image of a multi-viewpoint image, and the cost can be reduced.

  An image processing apparatus according to the present invention inputs an image input unit that inputs a multi-viewpoint image captured from two or more viewpoints and an image including a single-viewpoint image captured from a single viewpoint, and is input to the image input unit. A storage unit that stores images, a reduced image creation unit that creates a reduced image obtained by reducing the image stored in the storage unit, and whether the image stored in the storage unit is a multi-viewpoint image or a single-viewpoint image An image identification unit that performs image recognition, an image identified by the image identification unit as a multi-viewpoint image, a face detection unit that detects the presence or absence of a subject face area, and a list display of images stored in the storage unit In response, a reduced image of the image identified by the image identifying unit as a single-viewpoint image and a list of reduced images as representative images of the image identified by the image identifying unit as a multi-viewpoint image and the face detection unit did not detect the subject face area And image identification But provided with a display unit identify and face detection unit and the multi-view image is to list the predetermined sample reduced image subjected to the three-dimensional computer graphics as a representative image of the image detected subject face area.

  According to the present invention, a predetermined sample image subjected to three-dimensional computer graphic processing as a representative image of a multi-view image with a face detected is simply a reduced image as a representative image of a single-view image or a multi-view image with no face detected. Will be listed. Since a predetermined sample that is a representative image of the multi-viewpoint image is subjected to three-dimensional computer graphic processing, the difference between the representative image of the multi-viewpoint image with a face and a simple reduced image can be seen at a glance. In addition, since a predetermined sample is used, it is not necessary to perform three-dimensional computer graphic processing on each multi-viewpoint image in order to display a representative image of the multi-viewpoint image.

  An image processing apparatus according to the present invention inputs an image input unit that inputs a multi-viewpoint image captured from two or more viewpoints and an image including a single-viewpoint image captured from a single viewpoint, and is input to the image input unit. A storage unit that stores images, a reduced image creation unit that creates a reduced image obtained by reducing the image stored in the storage unit, and whether the image stored in the storage unit is a multi-viewpoint image or a single-viewpoint image An image identification unit that performs identification, a flash emission identification unit that identifies the presence or absence of flash emission at the time of shooting, and a list display of images stored in the storage unit. In response, the image identification unit displays a list of reduced images as representative images identified as single-view images, the image identification unit identifies multi-view images, and the flash emission identification unit identifies that there was no flash emission. A three-dimensional computer is used as a representative image of an image that is displayed as a list of reduced images created by the reduced image creating unit as a representative image of the image, the image identifying unit identifies as a multi-viewpoint image, and the flash emission identifying unit identifies flash emission. And a display unit for displaying a list of predetermined reduced sample images with graphics.

  According to the present invention, a reduced image subjected to three-dimensional computer graphic processing as a representative image of a multi-viewpoint image that was flashed at the time of shooting is used as a representative image of a single-viewpoint image or a multi-viewpoint image that was not flashed at the time of shooting. A list of simple reduced images is displayed. Since the representative image of the multi-viewpoint image having flash emission at the time of shooting is subjected to three-dimensional computer graphic processing, the difference from a simple reduced image or the representative image of the multi-viewpoint image having no flash emission at the time of shooting can be seen at a glance. Further, it is not necessary to use a special photorealistic method for displaying a representative image of a multi-viewpoint image, and the cost can be reduced.

  The image recognition unit further includes a face detection unit that detects the presence or absence of a subject face area from the image identified as a multi-viewpoint image, and the image processing unit detects the subject detected by the face detection unit for the first reduced image An area corresponding to the face area is set, and a second reduced image is created by performing three-dimensional computer graphic processing on the set area, and the display unit is an image identified by the image identification unit as a multi-viewpoint image. Among them, the face detection unit displays a list of first reduced images as representative images of images in which the subject face area has not been detected, and the second reduction as a representative image of images in which the face detection unit has detected the subject face area. A list of images may be displayed.

  Image processing is performed only on an important part called a face, and a substantial pseudo-stereoscopic image can be displayed as a representative image with less processing.

  An image processing apparatus according to the present invention inputs an image input unit that inputs a multi-viewpoint image captured from two or more viewpoints and an image including a single-viewpoint image captured from a single viewpoint, and is input to the image input unit. A storage unit that stores an image, a reduced image generation unit that generates a first reduced image obtained by reducing the image stored in the storage unit, and an image stored in the storage unit is a multi-viewpoint image or a single-viewpoint image An image identifying unit for identifying the image, a distance identifying unit for identifying whether a subject in the image identified by the image identifying unit as a multi-view image is at a long distance or a short distance, and an image identifying unit for identifying the multi-view image. And an image processing unit that creates a second reduced image by performing a three-dimensional computer graphic process on the first reduced image created from the image identified by the distance identifying unit as being close to the subject; One of the images stored in the storage unit In response to the display instruction, the image identification unit displays a list of the first reduced images created by the reduced image creation unit as representative images of the images identified as single-view images, and the image identification unit identifies the images as multi-view images. And a list of the first reduced images created by the reduced image creation unit as a representative image of the images identified by the distance identification unit as being at a long distance, and the image identification unit identifies the multi-viewpoint image and identifies the distance A display unit that displays a list of second reduced images created by the image processing unit as a representative image of an image that the unit has identified as being close to the subject.

  According to the present invention, since the three-dimensional computer graphic processing is performed only on the multi-viewpoint image having the subject at a short distance, the processing is not performed on the meaningless image even when the subject is at a long distance. I'll do it.

  An image processing method according to the present invention includes a step of inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint, and a step of storing the input image. A step of creating a first reduced image obtained by reducing the stored image, a step of identifying whether the stored image is a multi-view image or a single-view image, and an image identified as a multi-view image A single-viewpoint image is generated in response to an instruction to create a second reduced image by performing a three-dimensional computer graphic process on the first reduced image corresponding to, and to display a list of stored images. And displaying a list of first reduced images as representative images of the images identified as, and displaying a list of second reduced images as representative images of the images identified as multi-view images.

  An image processing method according to the present invention includes a step of inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint, and a step of storing the input image. A step of creating a reduced image obtained by reducing the stored image, a step of identifying whether the stored image is a multi-view image or a single-view image, and an image identified as a multi-view image. A step of detecting the presence / absence of a face area and a reduction of a single-viewpoint image identified as a single-viewpoint image and a multi-viewpoint image in response to an instruction to display a list of stored images, and no subject face area are detected A list of reduced images is displayed as a representative image of the captured image, and a three-dimensional computer graphic is applied as a representative image of the image that is identified as a multi-viewpoint image and the subject face area is detected. Comprising the steps of displaying a list of predetermined sample reduced image, a.

An image processing method according to the present invention includes a step of inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint, and a step of storing the input image. Shooting a reduced image obtained by reducing the stored image, a step of identifying whether the stored image is a multi-view image or a single-view image, and an image identified as a multi-view image. A step of identifying the presence or absence of flash emission at the time, and in response to an instruction to display a list of stored images, a list of reduced images is displayed as a representative image of the images identified as single-view images, and identified as multi-view images And a list of reduced images as representative images of the images identified as having no flash emission, the representative images of the images identified as being multi-viewpoint images and identified as having flash emission, and List the predetermined sample reduced image subjected to the three-dimensional computer graphics Te comprising the steps, a.

  A step of detecting the presence / absence of a subject face region from an image identified as a multi-viewpoint image, and setting a region corresponding to the subject face region for the first reduced image, and a three-dimensional computer in the set region A step of creating a second reduced image by performing graphic processing, and among the images identified as multi-viewpoint images, a list of the first reduced images is displayed as representative images of images in which the subject face area has not been detected. And a step of displaying a list of second reduced images as representative images of the images in which the subject face area has been detected.

  An image processing method according to the present invention includes a step of inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint, and a step of storing the input image. Generating a first reduced image obtained by reducing the stored image; identifying whether the stored image is a multi-view image or a single-view image; and A step of identifying whether the subject is at a long distance or a short distance; and a three-dimensional computer graphic process for a first reduced image created from the image identified as a multi-viewpoint image and identified as the subject is at a short distance To generate a second reduced image, and in response to an instruction to display a list of stored images, the first reduced image is used as a representative image identified as a single-viewpoint image. List the first reduced image as a representative image of the images that are displayed as a list, identified as multi-view images and identified as a subject at a long distance, identified as multi-view images and identified as a short distance Displaying a list of the second reduced images as representative images of the obtained images.

  A program for causing a computer to execute this image processing method is also included in the present invention.

  According to the present invention, an image subjected to three-dimensional computer graphic processing as a representative image of a multi-view image is a single-view image, a multi-view image with no face detected, or a representative image of a multi-view image in which a subject is at a long distance. A list of simple reduced images is displayed. Since a predetermined sample that is a representative image of the multi-viewpoint image is subjected to three-dimensional computer graphic processing, the difference between the representative image of the multi-viewpoint image with a face and a simple reduced image can be seen at a glance. Further, it is not necessary to use a special photorealistic method for displaying a representative image of a multi-viewpoint image, and the cost can be reduced.

Camera block diagram Flowchart of list image display processing according to the first embodiment Diagram showing an example of texture mapping The figure which shows an example of the thumbnail displayed by the list image display process which concerns on 1st Embodiment. Flowchart of list image display processing according to second embodiment The figure which shows an example of the thumbnail displayed by the list image display process which concerns on 2nd Embodiment. Flowchart of list image display processing according to third embodiment The figure which shows an example of the thumbnail displayed by the list image display process which concerns on 3rd Embodiment. Flowchart of list image display processing according to the fourth embodiment The figure which shows an example of the thumbnail displayed by the list image display process concerning 4th Embodiment Flowchart of list image display processing according to fifth embodiment The figure which shows an example of the thumbnail displayed by the list image display process which concerns on 5th Embodiment PC block diagram

<First Embodiment>
FIG. 1 shows an electrical configuration of the compound eye camera 1. The lens optical axes L1 and L2 of the first and second imaging units 2a and 2b are arranged in parallel so as to be parallel or at a predetermined angle.

  The first imaging unit 2a includes a first zoom lens 11a, a first diaphragm 12a, a first focus lens 13a, and a first image sensor 14a arranged along the lens optical axis L1. A diaphragm control unit 16a is connected to the first diaphragm 12a, and a timing generator (TG) 18a is connected to the first image sensor 14a. The operations of the first aperture 12a and the first focus lens 13a are controlled by the photometric distance measuring CPU 19a. The operation of the TG 18a is controlled by the main CPU 62.

The camera 1 is provided with an operation unit 70 for performing various operations when the user uses the camera 1. The operation unit 70 includes a power switch for turning on the power for operating the camera 1, a mode dial for selecting auto shooting and manual shooting, and a cross key for setting and selecting various menus or zooming. , A flash emission switch, and an information position designation key for executing or canceling the menu selected by the cross key. By appropriately operating the operation unit 70, turning on / off the power, switching between various modes (shooting mode, reproduction mode, etc.), zooming, and the like are performed.

  Further, the camera 1 includes a main CPU 62, an EEPROM 146, a YC / RGB conversion unit 147, and a display driver 148. The main CPU 62 controls the entire camera 1. The EEPROM 146 stores solid data and programs unique to the camera 1.

  The YC / RGB conversion unit 147 converts the color video signal YC generated by the YC processing units 35a and 35b into RGB signals of three colors and outputs them to the image display LCD 10 via the display driver 148.

In response to a zoom operation from the input operation unit 9, the first zoom lens 11a moves along the lens optical axis L1 to the NEAR side (feed-out side) or the INF side (retract-side) to change the zoom magnification. This movement is driven by a motor (not shown). The aperture 12a performs exposure adjustment by changing the aperture value (aperture value) during AE (Auto Exposure) operation to limit the light flux.
The focus lens 13a is moved to the NEAR side or the INF side along the lens optical axis L1 during AF (Auto Focus) operation to change the focus position and perform focus adjustment. This movement is driven by a motor (not shown).

  When the half-pressed state of the still image release switch is detected, the main CPU 62 obtains distance measurement data from the distance image sensors 51a and 51b, respectively. The main CPU 62 adjusts focus, aperture, etc. based on the obtained distance measurement data.

  The first image sensor 14a receives subject light imaged by the first zoom lens 11a, the first diaphragm 12a, and the first focus lens 13a, and accumulates photocharges corresponding to the amount of received light in the light receiving element. The photoelectric charge accumulation / transfer operation of the first image sensor 14a is controlled by the TG 18a, and the electronic shutter speed (photo charge accumulation time) is determined by the timing signal (clock pulse) input from the TG 18a. The first image sensor 14a acquires an image signal for one screen every predetermined period in the photographing mode.

  The second image pickup unit 2b has the same configuration as the first image pickup unit 2a, and is a second image in which the second zoom lens 11b, the second diaphragm 12b, the second focus lens 13b, and the timing generator (TG) 18b are connected. It is configured by the sensor 14b. These operations are controlled by the main CPU 62. The first imaging unit 2a and the second imaging unit 2b basically operate in conjunction with each other, but can also be operated individually. A CCD type or CMOS type image sensor is used as the first and second image sensors 14a and 14b.

  The imaging signals output from the first and second image sensors 14a and 14b are input to the A / D converters 30a and 30b, respectively. The A / D converters 30a and 30b convert the input image data from analog to digital. Through the A / D converters 30a and 30b, the imaging signal of the first image sensor 14a is first image data (right-eye image data), and the imaging signal of the second image sensor 14b is second image data (left-eye image). Data).

The image signal processing circuits 31a and 31b respectively perform various image processing such as gradation conversion, white balance correction, and γ correction processing on the first and second image data input from the A / D converters 30a and 31b. The buffer memories 32a and 32b include image signal processing circuits 31a and 31.
The first and second image data subjected to various image processes in b are temporarily stored.

  The photometry / ranging CPUs 19a and 19b calculate AF evaluation values and AE evaluation values from the first and second image data stored in the buffer memories 32a and 32b, respectively. The AF evaluation value is calculated by integrating high-frequency components of the luminance value for the entire area or predetermined area (for example, the central portion) of each image data, and represents the sharpness of the image. The high-frequency component of the luminance value is a sum of luminance differences (contrast) between adjacent pixels in a predetermined area. In addition, the AE evaluation value is calculated by integrating the luminance values over the entire area or a predetermined area (for example, the central portion) of each image data, and represents the brightness of the image. The AF evaluation value and the AE evaluation value are respectively used in an AF operation and an AE operation that are executed during an imaging preparation process described later.

  The image data stored in the buffer memories 32a and 32b is converted into luminance signals (Y signals) and color difference signals (Cr and Cb signals) by YC processing units 35a and 35b, respectively, and predetermined processing such as gamma correction is performed. Applied. The processed YC signals are stored in the work memories 128a and 128b, respectively.

  The YC signals of the first and second image data stored in the work memories 128a and 128b are read by the controller 34 to the YC / RGB conversion unit 147, respectively. The YC / RGB converter 147 converts the YC signal of the first and second image data into a predetermined format video signal (for example, an NTSC color composite video signal), and then performs stereoscopic display on the image display LCD 10. To the stereoscopic image data. When the LCD 10 is used as an electronic viewfinder in the shooting mode, the stereoscopic image data synthesized by the YC / RGB conversion unit 147 is displayed on the LCD 10 as a through image via the LCD driver 148.

  The compression / decompression processing circuits 36a and 36b respectively apply JPEG for still images, MPEG2, MPEG4, H.264 for moving images to the first and second image data stored in the work memories 128a and 128b, respectively. Compression processing is performed according to a predetermined compression format such as H.264. The media controller 37 records each image data compressed by the compression / decompression processing circuits 36 a and 36 on a memory card 38 or other recording media connected via the I / F 39.

  When the first and second image data recorded in the memory card 38 in this way are reproduced and displayed on the LCD 10, each image data in the memory card 38 is read by the media controller 37, and the compression / decompression processing circuits 36a and 36 are used. The YC / RGB conversion unit 147 converts the image data into stereoscopic image data, and then displays the reproduced image on the LCD 10 via the LCD driver 148.

  The LCD 10 is a normal monitor that does not employ a photorealistic technique, and is used as an electronic viewfinder at the time of image photographing, and displays image data obtained by photographing at the time of image reproduction.

  The main CPU 62 comprehensively controls the overall operation of the compound eye camera 1. In addition to the release switches 5a and 5b and the operation unit 70, the main CPU 62 is connected to an EEPROM 146 which is a nonvolatile memory. The EEPROM 146 stores various control programs and setting information. The main CPU 62 executes various processes based on this program and setting information.

  In addition, an optical system control instruction unit 127 is connected to the main CPU 62, and the imaging magnifications of the first imaging unit 2a and the second imaging unit 2b are changed according to a zoom operation to the optical system control instruction unit 127. .

  The release switch 5a has a two-stage push switch structure. If the release switch 5a is lightly pressed (half-pressed) during the shooting mode, AF operation and AE operation are performed, and shooting preparation processing is performed. In this state, when the release switch 5a is further pressed (fully pressed), shooting processing is performed, and the first and second image data for one screen are transferred from the frame memory 32 to the memory card 38 and recorded. .

  In the AF operation, the main CPU 62 controls the first and second focus lenses 13a and 13b and moves them in predetermined directions, respectively, and the AF evaluation value calculated from each of the first and second image data obtained sequentially. This is done by finding the maximum value. In the AE operation, after the AF operation is completed, the aperture values of the first and second apertures 12a and 12b and the electronic shutter speeds of the first and second image sensors 14a and 14b are calculated based on the calculated AE evaluation values. This is done by setting.

  In addition, the camera 1 is provided with an operation LCD display 103 for assisting the operation.

  Further, the camera 1 has a configuration in which the power supply battery 68 is detachable. The power supply battery 68 is composed of a rechargeable secondary battery such as a nickel-cadmium battery, a nickel metal hydride battery, or a lithium ion battery. The power supply battery 68 may be a single-use primary battery such as a lithium battery or an alkaline battery. The power supply battery 68 is electrically connected to each circuit of the camera 1 by being loaded into a battery storage chamber (not shown).

  The first imaging unit 2a and the second imaging unit 2b are respectively provided with an interval / convergence angle detection circuit 4a, 4b for detecting an interval / convergence angle between the first imaging unit 2a and the second imaging unit 2b, and an interval / congestion. Lens interval / convergence angle storage circuit 6 for storing the angle of convergence detected by the angle detection circuits 4a and 4b, and interval / convergence angle drive circuits 3a and 3b for changing the convergence by rotating the optical axis with a drive motor or the like by changing the baseline length. And are provided.

  Further, the camera 1 includes an interval / convergence angle control circuit 5 that controls changes in the convergence angle of the interval / convergence angle driving circuits 3a and 3b based on the intervals / convergence angles detected by the interval / convergence angle detection circuits 4a and 4b. Is provided.

  The charging / light emission control units 138a and 138b are supplied with electric power from the power supply battery 68 in order to cause the strobes 44a and 44b to emit light, respectively, and charge a flash light emitting capacitor (not shown) or emit light from the strobes 44a and 44b. Control.

  The charge / light emission control units 138a and 138b receive various signals such as half-press and full-press operation signals of the release switches 5a and 5b, and signals indicating the light emission amount and the light emission timing from the main CPU 62 and the photometry / ranging CPUs 19a and 19b. In response to the capture, current supply control to the strobes 44a and 44b is performed so that a desired light emission amount can be obtained at a desired timing.

  The vertical / horizontal shooting switching button 40 is a button for instructing whether to perform vertical shooting or horizontal shooting. The vertical / horizontal shooting detection circuit 166 detects whether shooting is performed in vertical shooting or horizontal shooting depending on the state of this button.

  The 2D / 3D mode switching flag 168 is set with a flag indicating the 2D mode or the 3D mode.

The distance light-emitting elements 52a and 52b are each composed of a light-emitting diode (LED) for irradiating a projection spot to the same subject captured by the first imaging unit 2a and the second imaging unit 2b.

  The distance image pickup devices 51a and 51b are image pickup devices dedicated to distance measurement that acquire subject images irradiated with the light projection spots by the distance light emitting devices 52a and 52b, respectively.

  The distance driving / control circuit 54 performs control to synchronize the light emission timings of the distance light emitting elements 52a and 52b with the distance imaging elements 53a and 53b.

  Analog image signals obtained by the imaging operations of the distance imaging elements 53a and 53b are converted into digital image data by the distance measurement A / D conversion units 55a and 55b, respectively, and are output to the distance information processing circuit 57.

  The distance information processing circuit 57 calculates the distance from the input digital image data to the subject captured by the first imaging unit 2a and the second imaging unit 2b. This is based on the principle of so-called triangulation. The distance information calculated by the distance information processing circuit 57 is stored in the distance information storage circuit 58.

  The face detection unit 150 detects a face area that is an area including the face portion of the person who is the subject from the image data stored in the buffer memory 32 a or the buffer memory 32 b or the image data stored in the buffer memory 42.

  The method for detecting the face area is not particularly limited, and various methods can be adopted. For example, the technique disclosed in Japanese Patent Application Laid-Open No. 9-101579 by the present applicant can be applied. This technology determines whether or not the hue of each pixel of the captured image is included in the skin color range and divides it into a skin color region and a non-skin color region, and detects edges in the image to detect each pixel in the image. The location is classified as an edge portion or a non-edge portion. Then, an area surrounded by pixels that are located in the skin color area and classified as a non-edge part and surrounded by pixels that are determined to be edge parts is extracted as a face candidate area, and the extracted face candidate area becomes a human face. It is determined whether the region is a corresponding region, and is detected as a face region based on the determination result. In addition to this, a face region can also be detected by a method described in Japanese Patent Application Laid-Open Nos. 2003-209683 and 2002-199221.

  The multi-viewpoint image is not necessarily acquired by the compound eye camera 1 as described above, and may be acquired by continuous shooting using a motion stereo method using a monocular camera.

  Hereinafter, the flow of the list image display process executed by the main CPU 62 will be described with reference to the flowchart of FIG. A program for defining this processing is stored in the EEPROM 146. In response to an instruction of “display thumbnail image list” from the operation unit 70, all the multi-viewpoint images stored in the memory card 38 are collectively performed.

  In S1, the directory of the memory card 38 is read.

  In S2, it is determined whether the image file exists in the directory. If an image file exists, the process proceeds to S3. If the image file does not exist, the process ends.

  In S3, the header portion of the image file is read out. Then, it is determined whether or not identification information indicating that the image file stores a multi-viewpoint image is detected. If a multi-viewpoint image is stored, the process proceeds to S5. If a normal image is stored, the process proceeds to S8.

In S5, a thumbnail image is created from the multi-viewpoint image stored in the image file. The specific method may be created by thinning out only the left-eye image, only the right-eye image, or both, as in paragraph 0040 of Patent Document 2.

  In S6, a 3D thumbnail image is created by performing 3D CG processing on the created thumbnail image. For example, as shown in FIG. 3, a thumbnail image is displayed on a polygon PG of a three-dimensional shape model corresponding to a main subject and other various objects (FIG. 3 shows a mug but may be a person) existing in the image. Texture mapping for pasting the corresponding region of th is performed to obtain a three-dimensional CG thumbnail image X. This specific method is the same as that of Patent Document 1. Alternatively, a 3D CG process may be performed on the original multi-viewpoint image, and a 3D CG thumbnail image may be created by thinning out the 3D CG process. However, in terms of processing load, it is better to create a thumbnail first. Yes.

  Note that other 3D computer graphic techniques may be applied to the thumbnail image to create a 3D CG thumbnail image.

  In S 7, the created three-dimensional CG thumbnail image is stored in a list image buffer secured in the buffer memory 41.

  In S8, a thumbnail image is created from a single viewpoint normal image stored in the image file. The specific method may be created by thinning out a single viewpoint image in the same manner as a normal thumbnail image. This thumbnail is called a normal thumbnail image.

  In S9, the created normal thumbnail image is stored in the list image buffer secured in the buffer memory 41.

  In S10, it is determined whether or not a three-dimensional CG thumbnail image or a normal thumbnail image has been created for all images on the memory card 38. If it has been created, the process proceeds to S11. If the creation has not been completed, the process returns to S3 to continue the creation.

  In S11, the 3D CG thumbnail image and the normal thumbnail image in the buffer memory 41 are displayed in a list on the LCD 10. The order may be the order of the names and recording dates of the image files from which the thumbnail images are created. That is, the 3D CG thumbnail image and the normal thumbnail image are not distinguished from each other, and the 3D CG thumbnail image and the normal thumbnail image are mixedly arranged in one screen of the LCD 10.

  Note that the processing of S3 to S9 may be performed every time an image is newly recorded.

  FIG. 4 shows an example of the three-dimensional CG thumbnail image X and the normal thumbnail image Y displayed as a list on the LCD 10. Since the 3D CG thumbnail image X is subjected to 3D CG graphic processing and has a three-dimensional effect, the difference from the normal thumbnail Y is apparent at first glance, and a special device that realizes a realistic method is used. Even without this, the viewer can easily recognize that the image is a reduced image of a multi-viewpoint image.

Second Embodiment
FIG. 5 shows another example of the list image display process executed by the main CPU 62. Here, the thumbnail image of the multi-viewpoint image showing the face is replaced with a special sample.

  S21 to S24 are the same as S1 to S4.

In S25, the face detection unit 150 is instructed to extract a face from one or both of the multi-viewpoint images. In response to a command from the main CPU 62, the face detection unit 150 attempts to detect a face area from the left-eye image or the right-eye image stored in the image file.

  In S26, it is determined whether or not the face detection unit 150 has detected the face area. If the face area can be detected, the process proceeds to S27. If the face area cannot be detected, the process proceeds to S29.

  In S 27, a sample three-dimensional thumbnail image representing that the face is a multi-viewpoint image is taken out from the EEPROM 146.

  In S28, the extracted sample is stored in the list image buffer. The sample 3D thumbnail image does not have to be the same as the face of the actual subject, and may be a sample person's face, and the face or body is preliminarily provided with a 3D graphic stereoscopic effect. Yes.

  S29 is the same as S5 described above. In S30, the thumbnail images created in S29 are stored in the list image buffer. This thumbnail image is called a stereoscopic image reduced image, and is distinguished from a normal thumbnail image.

  S31 to S34 are the same as S8 to S11 described above.

  FIG. 6 shows an example of sample 3D thumbnail images S and normal thumbnail images Y displayed as a list on the LCD 10 as a result of the execution of S34.

  Since the sample thumbnail image S has been subjected to three-dimensional CG graphic processing, the difference from the normal thumbnail Y and the stereoscopic image reduced image Z is obvious at a glance, and it is easy to identify the stereoscopic image with the face as the subject. is there.

<Third Embodiment>
FIG. 7 shows another example of the list image display process executed by the main CPU 62. Here, the thumbnail image of the multi-viewpoint image captured by flash is replaced with a special sample.

  S41 to S44 are the same as S1 to S4.

  In S45, flash emission information is acquired from the image file, and the on / off status of the strobes 44a and 44b at the time of image recording is identified. The flash emission information is, for example, information stored in the “Flash” tag of the Exif file.

  In S46, it is determined whether or not the strobes 44a and 44b are turned on and off based on the flash emission information. If the strobes 44a and 44b are on at the time of shooting, the process proceeds to S47, and if off, the process proceeds to S49.

  In S 47, a sample three-dimensional thumbnail image representing a multi-viewpoint image is taken out from the EEPROM 146. This sample is given a three-dimensional effect by three-dimensional CG processing as in the second embodiment. Note that the design of this sample need not be the face of a specific person.

  In S48, the extracted three-dimensional thumbnail image of the sample is stored in the list image buffer. The sample three-dimensional thumbnail image does not have to be the same as the face of the actual subject, but is a thumbnail image that displays in 3D graphics that a person's face is shown.

  S49 to S54 are the same as S29 to S34 described above.

  FIG. 8 shows an example of sample 3D thumbnail images W and normal thumbnail images Y displayed as a list on the LCD 10 as a result of the execution of S54.

  Since the sample thumbnail image W has been subjected to three-dimensional CG processing, the difference from the normal thumbnail Y and the stereoscopic image reduced image Z is obvious at a glance, and it is easy to identify the stereoscopic image with the face as the subject. .

  In addition, it is not necessary to uniformly display a sample even in a dark image that is difficult to effectively use as a multi-viewpoint image, and the processing becomes efficient.

<Fourth embodiment>
FIG. 9 shows another example of the list image display process executed by the main CPU 62. Here, the thumbnail image of the multi-viewpoint image showing the face is replaced with a thumbnail image that has been subjected to the three-dimensional CG process only for the face area of the multi-viewpoint image.

  S61 to S66 are the same as S21 to S26. However, in S66, if it is determined that there is a face, the process proceeds to S67, and if it is determined that there is no face, the process proceeds to S69.

  In S67, a face area is extracted from the multi-viewpoint image showing the face, and the three-dimensional CG process is performed on the face area. For example, a face area is extracted from a multi-viewpoint image showing a face, and the multi-viewpoint image is thinned out at a predetermined thinning rate. An area corresponding to the face area extracted from the original multi-viewpoint image is set in the reduced image after thinning. Then, a polygon peculiar to the face in the set area is created, and texture mapping is performed by pasting the face area after thinning on the polygon to obtain a thumbnail image on which only the face area has been subjected to 3D CG processing. In this method, three-dimensional CG processing is performed after creation of a thinned image.

  Alternatively, a polygon unique to the person's face is created from the detected face itself, texture mapping is performed to attach the face area to the polygon, and a three-dimensional CG process is performed on only the face area on the multi-viewpoint image. Make a CG image. Then, the provisional CG image is thinned out at a predetermined thinning rate to obtain a thumbnail image in which only the face area is subjected to the three-dimensional CG process. This method creates a thinned image after the three-dimensional CG processing.

  The above texture mapping method needs to start with creation of a polygon. If a precise polygon is created, the processing load is large. The following is a simple polygon creation method. First, face components such as a face outline, eyes, nose, and mouth are detected from the detected face region, and polygon dividing lines are determined according to the positions of the detected face components. For example, a line that starts from the central part (between eyebrows) and reaches the tip of the nose along the nasal muscles, a line that circulates both eyes, a line that circulates the mouth, and a contour of the face are polygon dividing lines. When texture mapping is performed on this polygon, a simple three-dimensional CG thumbnail image in which the mouth and nose are raised and the periphery of the eye is depressed is obtained.

  In S68, the created three-dimensional CG thumbnail image is stored in the list image buffer.

  S69 to S74 are the same as S29 to S34 described above.

  FIG. 10 shows three-dimensional thumbnail images S, normal thumbnail images Y corresponding to face-detected multi-viewpoint images displayed as a list on the LCD 10 as a result of executing S74, and multi-viewpoint images with no face detected. An example of a corresponding thumbnail image Z is shown.

  Since the sample thumbnail image U is subjected to the 3D CG graphic processing only for the face area, the difference from the normal thumbnail Y or the reduced image Z of the simple viewpoint image is apparent at first glance, and the sample thumbnail image U is a three-dimensional image with the face as the subject. Image identification is easy. In addition, since the three-dimensional CG graphic processing is performed on the face area using the original image, the outline of the face of the original image can be understood three-dimensionally.

<Fifth Embodiment>
FIG. 11 shows another example of the list image display process executed by the main CPU 62. Here, only a thumbnail image of a multi-viewpoint image obtained by photographing a subject at a short distance is subjected to three-dimensional CG processing.

  S81 to S84 are the same as S41 to S44.

  In S85, distance information is acquired from the image file, and the distance from the camera 1 to the subject at the time of image recording is identified. The distance information is, for example, information stored in the “SubjectDistance” tag of the Exif file, and is stored by the distance information storage circuit 58.

  In S86, based on the distance information D, it is determined whether or not the subject is at a position farther than a predetermined distance (for example, D> 10 m). If the subject is a long distance, the process proceeds to S89. If the subject is not a long distance, that is, if the object is a short distance, the process proceeds to S87.

  In S87, as in S6, a three-dimensional CG thumbnail image is created.

  In S88, the created three-dimensional CG thumbnail image is stored in the list image buffer secured in the buffer memory 41.

  S89 to S94 are the same as S69 to S74 described above.

  FIG. 12 shows an example of the thumbnail image V that is displayed as a list of the three-dimensional thumbnail image T, the normal thumbnail image Y, and the subject that has not been subjected to the three-dimensional CG processing on the LCD 10 as a result of the execution of S94.

  Since the sample thumbnail image T has been subjected to three-dimensional CG processing, the difference from the normal thumbnail Y or the thumbnail image V of an image with a far subject is apparent at a glance.

  In addition, since the 3D CG process is performed only on the multi-viewpoint image in which the subject is at a short distance at the time of shooting, it is not necessary to perform the 3D CG process uniformly even to the far object image even if the process is performed. , Process becomes more efficient.

<Sixth Embodiment>
FIG. 13 is a block diagram of the personal computer 100. The personal computer 100 mainly includes a central processing unit (CPU) 102 that controls the operation of each component, a main memory 106 that stores a control program for the device and a work area when the program is executed, and an operating system ( OS), a hard disk device 108 in which various application software including an image editing program for editing a viewpoint image so as to conform to a device driver of a peripheral device connected to the personal computer 100, a user image, and the like are stored, and a CD- ROM device 110, display memory 116 for temporarily storing display data, monitor device 118 such as a CRT monitor or a liquid crystal monitor for displaying images, characters, etc. from image data, character data, etc. from this display memory 116, keyboard 120 and a mouse 12 as a position input device A mouse controller 124 that detects the state of the mouse 122 and outputs signals such as the position of the mouse pointer on the monitor device 118 and the state of the mouse 122 to the CPU 102, and a communication interface 132 connected to the network 30 such as the Internet. A card interface 112 having a card insertion slot for inserting and removing the memory card 38, a bus 104 for connecting the above-described components, and a camera connection I / F 134 for USB connection with the camera 1.

  Application software for image editing processing stored in the hard disk device 108 can be installed in the personal computer 100 by setting a CD-ROM in which the application software is recorded in the CD-ROM device 110 of the personal computer 100. .

  The monitor device 118 displays an image similarly to the LCD 10.

  The hard disk device 108 stores a multi-viewpoint image or a normal image received from the compound eye camera 1 via the camera connection I / F 134 or a multi-viewpoint image or a normal image captured from the memory card 38 via the card I / F 112. The

  The CPU 102 performs the list image display processing of the first to fifth embodiments described above for the multi-viewpoint image and the normal image stored in the hard disk device 108. This process may be executed for each image individually as each image is stored in the hard disk device 108, or all images stored in the hard disk device 108 in accordance with instructions from the keyboard 120 and mouse 122. May be performed collectively.

  That is, the list image display process is not necessarily performed by the camera 1 and can be realized by other apparatuses.

  1: camera, 2a: first imaging unit, 2b: second imaging unit, 10: LCD, 62: main CPU, 38: memory card, 100: personal computer, 102: CPU, 108: hard disk device

Claims (3)

An image input unit for inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint;
A storage unit for storing an image input to the image input unit;
A reduced image creating unit that creates a reduced image obtained by reducing the image stored in the storage unit;
An image identification unit for identifying whether the image stored in the storage unit is a multi-viewpoint image or a single-viewpoint image;
A face detection unit that detects the presence or absence of a subject face region for the image identified by the image identification unit as a multi-viewpoint image;
In response to an instruction to display a list of images stored in the storage unit, a reduced image of the image identified by the image identifying unit as a single-viewpoint image, and the image identifying unit identifies it as a multi-viewpoint image and detects the face A list of the reduced images as representative images of images in which the subject face area has not been detected, and a representative of images in which the image identification unit has identified the multi-viewpoint image and the face detection unit has detected the subject face area A display unit for displaying a list of predetermined reduced sample images subjected to three-dimensional computer graphics in the subject face area as images;
An image processing apparatus comprising: Inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint;
Storing the input image; and
Creating a reduced image obtained by reducing the stored image;
Identifying whether the stored image is a multi-viewpoint image or a single-viewpoint image;
Detecting the presence or absence of a subject face area for the image identified as the multi-viewpoint image;
In response to an instruction to display a list of the stored images, as a reduced image of an image identified as the single-viewpoint image and a representative image of an image identified from the multi-viewpoint image and the subject face area is not detected The reduced images are displayed in a list, and a predetermined sample reduced image in which a three-dimensional computer graphic is applied to the subject face area is displayed as a list as a representative image that is identified from the multi-viewpoint image and that detects the subject face area. And steps to
An image processing method including: Computer
Inputting an image including a multi-viewpoint image captured from two or more viewpoints and a single-viewpoint image captured from a single viewpoint;
Storing the input image; and
Creating a reduced image obtained by reducing the stored image;
Identifying whether the stored image is a multi-viewpoint image or a single-viewpoint image;
Detecting the presence or absence of a subject face area for the image identified as the multi-viewpoint image;
In response to an instruction to display a list of the stored images, as a reduced image of an image identified as the single-viewpoint image and a representative image of an image identified from the multi-viewpoint image and the subject face area is not detected The reduced images are displayed in a list, and a predetermined sample reduced image in which a three-dimensional computer graphic is applied to the subject face area is displayed as a list as a representative image that is identified from the multi-viewpoint image and that detects the subject face area. And steps to
An image processing program for executing

Images