JP2014090252A - Image processing device and control method for the same, image pickup device and control method for the same and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent images from having a missing part due to change of parallax in stereoscopic vision display, thereby enabling reproduction with no discomfort for appreciation.SOLUTION: An image processing device generates left-eye image data and right-eye image data for stereoscopic vision display by using composite image data of a left-eye image and a right-eye image and parallax map data representing a parallax amount distribution thereof. It is determined whether any enlarged-display possible area exists in the right-and-left direction of an image, and processing of enlarging an image-display area is executed when the area concerned exists (see image example 1304 or 1306). Parallax enlarging processing is executed in conformity with the enlarged image-display area, and parallax map data which has been changed in parallax amount are generated. The parallax map data are dynamically varied in conformity with the image-display area, thereby generating and reproducing left-eye image data and right-eye image data that have no image missing part (see image example 1303).

Description

  The present invention relates to an image processing technique for generating stereoscopic image data from image information and parallax information.

Conventionally, a stereo camera capable of capturing an image for 3D (three-dimensional) display or stereoscopic display has been proposed. With the widespread use of digital imaging devices such as digital cameras and digital video cameras, an original image for generating a stereoscopic image has been taken with a digital imaging device.
As a mechanism for viewing a stereoscopic video, it is common to display a “left-eye image” corresponding to an image of an object viewed with the left eye and a “right-eye image” corresponding to an image viewed with the right eye. In the parallax barrier method, the lenticular method, and the like, there are known a method of dividing an image to be viewed by parallax, a method of causing different images to enter the left eye and the right eye through filters having different characteristics on the left and right.

On the other hand, there is a method for simultaneously capturing images from different viewpoints as a method for capturing an image that allows viewing of a stereoscopic image. Patent Document 1 discloses an apparatus having a microlens and a plurality of divided photodiodes that form one pixel portion in a solid-state imaging device. Stereo image data can be acquired by utilizing the fact that each photodiode receives light from a different pupil plane. That is, the first image signal is obtained from one output of the photodiode pair and the second image signal is obtained from the other output, and then each data of the left-eye image and the right-eye image is generated.
Patent Document 2 discloses a method for calculating an output parallax map with output elements having output values corresponding to shifts to be applied to respective pixels of the first image in order to calculate the second image. Yes. This calculation is based on an input parallax map with each input element having an input value, and generating a second image using the first image and the output parallax map to render a multi-view image. Is possible.
The three-dimensional image quality control device disclosed in Patent Document 3 can perform parallax enlargement by changing the parallax amount between the left and right images, and can change the pop-out amount of the image.

JP-A-58-24105 Special table 2008-518317 JP-A-8-317429

By the way, when viewing a stereoscopic video, if the parallax is expanded so that a more stereoscopic effect can be obtained, a portion beyond the image display area may be lost (see FIG. 12 for this phenomenon). Will be described later). For this reason, there is a possibility that the viewer may feel uncomfortable.
An object of the present invention is to perform reproduction without discomfort during viewing in stereoscopic display by preventing image loss due to parallax change.

  In order to solve the above problems, an apparatus according to the present invention is an image processing apparatus that acquires image data for stereoscopic display and data on the amount of parallax thereof, and generates image data for left eye and image data for right eye. Control means for controlling processing for enlarging the image display area related to the image data, parallax amount changing means for changing the parallax amount in accordance with the enlarged image display area, the image data, and the parallax Data generating means for generating the left-eye image data and the right-eye image data using the parallax amount data changed by the amount changing means is provided.

  According to the present invention, it is possible to prevent a viewer from feeling uncomfortable by preventing image loss due to parallax change.

FIG. 7 is a diagram schematically illustrating an entire configuration of an imaging device in order to describe an embodiment of the present invention in conjunction with FIGS. 2 to 6. It is a figure which shows the structural example of the pixel of an image pick-up element. It is the conceptual diagram showing a mode that the light beam which came out of the exit pupil of the imaging lens injects into an image pick-up element. It is a figure which shows the structural example of an imaging device. It is a figure explaining the production | generation process example of parallax map data. It is a figure which shows the structural example of an image file typically. It is a figure explaining a parallax expansion process in order to explain a 1st embodiment of the present invention in conjunction with Drawing 8 thru / or Drawing 10, Drawing 12, and Drawing 13. It is a flowchart explaining an image reproduction process. It is a flowchart explaining the example of a stereo image reproduction | regeneration processing. 10 is a flowchart illustrating an example of an image display area enlargement process and a data generation process. It is a flowchart explaining the weighting process example of parallax map data in order to demonstrate 2nd Embodiment of this invention combined with FIG. 12, FIG. It is a figure which illustrates the synthesized image, the image for left eyes, and the image for right eyes produced by 3D reproduction | regeneration processing. It is a figure which shows the example of a display of the stereo image on the screen of a display part.

The configuration and processing common to the embodiments of the present invention will be described below with reference to FIGS. The image processing apparatus according to the embodiment includes an image data generation unit and an image display control unit, and a case where the image processing apparatus is applied to an imaging apparatus will be exemplified below. That is, the image processing apparatus is provided in the imaging apparatus, and the image reproduction process is performed by processing the imaging data by the imaging means. In addition to this, there is a system configuration in which an imaging apparatus is connected to an image processing apparatus as an external apparatus. In this case, the image processing apparatus acquires image data from the imaging apparatus by wired or wireless communication and performs image reproduction processing. Hereinafter, stereoscopic display is abbreviated as 3D display, and two-dimensional display is abbreviated as 2D display.
FIG. 1 is a diagram schematically illustrating a configuration example of an imaging element used in the imaging apparatus of the present embodiment. The image sensor 100 includes a pixel array 101, a vertical selection circuit 102 that selects a row in the pixel array 101, and a horizontal selection circuit 104 that selects a column in the pixel array 101. The readout circuit 103 reads out the signal of the pixel selected by the vertical selection circuit 102 among the pixels in the pixel array 101. The reading circuit 103 includes a memory for accumulating signals, a gain amplifier, an AD converter, and the like for each column.

  The serial interface (SI) unit 105 determines an operation mode of each circuit according to an instruction from an external circuit. The vertical selection circuit 102 sequentially selects a plurality of rows in the pixel array 101 and extracts pixel signals to the readout circuit 103. The horizontal selection circuit 104 sequentially selects a plurality of pixel signals read by the reading circuit 103 for each column. In addition to the components shown in FIG. 1, the imaging device 100 includes a timing generator that provides timing signals to the vertical selection circuit 102, the horizontal selection circuit 104, the readout circuit 103, and the like, a control circuit, and the like. Detailed description thereof will be omitted.

  FIG. 2 is a diagram illustrating a configuration example of pixels of the image sensor 100. FIG. 2A schematically shows the configuration of one pixel. FIG. 2B shows the arrangement of the pixel array 101. A pixel 201 illustrated in FIG. 2A includes a microlens 202 as an optical element and a plurality of photodiodes (hereinafter abbreviated as PD) as light receiving elements.

  FIG. 2A illustrates an example in which two pixels, the left PD 203 and the right PD 204, are provided in one pixel, but three or more (for example, four or nine) PDs may be used. The PD 203 photoelectrically converts the received light beam and outputs a left-eye image signal. The PD 204 photoelectrically converts the received light beam and outputs a right eye image signal. The pixel 201 includes, for example, a pixel amplification amplifier that extracts a PD signal to the readout circuit 103, a row selection switch, a PD signal reset switch, and the like in addition to the illustrated components.

  In order to provide a two-dimensional image, the pixel array 101 is arranged in a two-dimensional array like a large number of pixels 301 to 304 shown in FIG. In the pixels 301 to 304, PDs 301L, 302L, 303L, and 304L correspond to the PD 203 in FIG. PDs 301R, 302R, 303R, and 304R correspond to the PD 204 in FIG. That is, each pixel has a first photoelectric conversion unit (PD 203) that outputs image data for the left eye and a second photoelectric conversion unit (PD 204) that outputs image data for the right eye.

Next, light reception of the imaging element 100 having the pixel configuration illustrated in FIG. FIG. 3 is a conceptual diagram showing a state where a light beam emitted from the exit pupil of the photographing lens is incident on the image sensor 100.
The pixel array 401 includes a microlens 402, a color filter 403, and PDs 404 and 405. PD 404 and PD 405 correspond to PD 203 and PD 204 in FIG.

  In FIG. 3, for each microlens 402, the optical axis 409 is the center of the light beam emitted from the exit pupil 406 of the photographing lens. The light emitted from the exit pupil 406 enters the image sensor 100 with the optical axis 409 as the center. Partial areas 407 and 408 are areas of the exit pupil 406 of the photographing lens. Rays 410 and 411 are the outermost rays among the light passing through the partial region 407. Light rays 412 and 413 are the outermost light rays among the light passing through the partial region 408.

  Among the light beams emitted from the exit pupil 406, with the optical axis 409 as a boundary line, the upper light beam in FIG. 3 is incident on the PD 405, and the lower light beam is incident on the PD 404. That is, PD 404 and PD 405 receive light beams from different regions of the exit pupil of the photographing optical system.

  FIG. 4 shows a configuration example of the imaging apparatus. Hereinafter, an application example to a digital camera will be described with reference to FIG. The lens unit 501 constituting the imaging optical system forms an image of light from the subject on the imaging element 505. The image sensor 505 corresponds to the image sensor 100 shown in FIG. 1 and has the pixel configuration shown in FIG.

  The lens driving device 502 performs zoom control, focus control, aperture control, and the like. The mechanical shutter 503 is controlled by a shutter driving device 504. The image sensor 505 photoelectrically converts the subject image formed by the lens unit 501 into an image signal. The imaging signal processing circuit 506 performs various corrections, predetermined pixel interpolation processing, and color conversion processing on the image signal output from the imaging element 505. The timing generator 507 outputs a timing signal necessary for the image sensor 505 and the image signal processing circuit 506.

  A system control unit 509 is a control unit that performs various calculations and controls the entire imaging apparatus, and performs processing by a CPU (central processing unit) (not shown) executing a program. The system control unit 509 generates data for a left-eye image and a right-eye image used for 3D display based on the combined image data generated by the image combining circuit 513 and the disparity map data generated by the disparity map generating circuit 514. Control processing. Also, the system control unit 509 performs playback control of the left-eye image and the right-eye image, so that the user can view a stereoscopic image on the display screen. Note that the system control unit 509 can also realize phase difference AF (autofocus) by detecting a phase difference based on the left-eye image data and the right-eye image data.

  The storage unit 508 includes a memory that temporarily stores image data. A recording medium control interface unit (hereinafter, the interface is abbreviated as I / F) 510 records or reads image data or the like on the recording medium 511. The recording medium 511 that can be attached to and detached from the imaging apparatus is a semiconductor memory or the like. The external I / F unit 512 is operated by the user to give an instruction to the camera, and an operation signal is sent to the system control unit 509. The display unit 521 displays various information and captured images in accordance with display data from the display control circuit 522. The display control circuit 522 performs 2D display control and 3D display control.

  The imaging signal processing circuit 506 performs processing for distributing imaging data output from the imaging element 505 into left-eye image data and right-eye image data. The storage unit 508 stores the output data of the imaging signal processing circuit 506, the synthesized image data generated by the image synthesis circuit 513, and the parallax map data generated by the parallax map generation circuit 514.

The image synthesis circuit 513 generates synthesized image data obtained by synthesizing the left-eye image data and the right-eye image data. The image composition circuit 513 performs an averaging process on the image data output from the image sensor 505 for each pixel, and selectively obtains only necessary PD signals for processing. In the example shown in FIG. 2B, the pixel signal of PD30xL is extracted from the pixel signals output from PD30xL and PD30xR (x = 1 to 4), and left-eye image data is generated. Further, the pixel signal of the PD 30xR is taken out, and right-eye image data is generated. When the output signal of PD30xL is expressed as S30xL and the output signal of PD30xR is expressed as S30xR, composite image data is generated from a signal obtained by averaging the two signals, (S30xL + S30xR) / 2. Note that, instead of the averaging process, composite image data may be generated from a signal obtained by calculating an addition signal (S30xL + S30xR) and performing dynamic range adjustment.
The parallax map generation circuit 514 calculates a parallax amount for each pixel based on the paired left-eye image data and right-eye image data. A parallax map indicating the parallax amount distribution is created from the parallax amount data obtained by the parallax amount calculation processing. For example, the parallax map generation circuit 514 calculates the positional deviation amount of the subject image in the left-eye image or the right-eye image as the parallax amount with reference to the position of the subject image in the composite image. Alternatively, the positional deviation amount of the right-eye image (or left-eye image) may be calculated as the parallax amount using the left-eye image (or right-eye image) as a reference.
The compression / decompression circuit 520 compresses the image data into JPEG image data or the like according to a predetermined image compression method (for example, adaptive discrete cosine transform (ADCT) or the like). JPEG (Joint Photographic Experts Group) image data is data that can be handled by many applications. The compressed image data is written in the storage unit 508. The compression / decompression circuit 520 performs a process of decompressing the image data read from the storage unit 508 and writing the decompressed image data to the storage unit 508.

  Next, the camera operation at the time of shooting in the present embodiment will be described. When the main power supply is turned on, the power supply of the control system circuit unit is turned on. Further, the power of the imaging processing system circuit such as the imaging signal processing circuit 506 is turned on, and the camera enters a standby state. The system control unit 509 determines whether the user has operated the release button. When there is a shooting operation start operation instruction, the system control unit 509 performs arithmetic processing related to focus state detection and focus adjustment operations based on output data from the image sensor 505. In this calculation process, the in-focus position corresponding to the distance from the camera to the subject is calculated, and the driving amount of the focus lens is calculated. The system control unit 509 performs drive control of the lens unit 501 via the lens driving device 502 in accordance with the calculation result, and determines whether or not it is in focus.

  If the system control unit 509 determines that it is not in focus, the system control unit 509 executes focus state detection processing again by driving control of the lens unit 501. The calculation for obtaining the distance to the subject and the in-focus position may be performed by using a dedicated distance measuring device (not shown) in addition to the method of calculating from the data from the image sensor 505. The system control unit 509 starts the photographing operation after determining the focused state. When the photographing operation is completed, the imaging signal processing circuit 506 processes the image signal output from the imaging element 505, and the system control unit 509 performs control to write the image data in the storage unit 508.

Next, image processing in this embodiment will be described.
The image sensor 505 outputs all image signals from the PD. In the example shown in FIG. 2B, image signals are sequentially output in the order of PDs, that is, 301L, 301R, 302L, 302R, 303L, 303R, 304L, and 304R. The imaging signal processing circuit 506 acquires image data from the output signal of the imaging element 505 and performs data separation processing into left-eye image data and right-eye image data. The left-eye image data is image data obtained by selectively processing only the outputs (left PD output) of PDs 301L, 302L, 303L, and 304L in FIG. The right-eye image data is image data obtained by selectively processing only the outputs (right PD output) of PDs 301R, 302R, 303R, 304R, etc. in FIG. The left-eye image data and the right-eye image data are stored in the storage unit 508 separately.

  The left-eye image data and right-eye image data stored in the storage unit 508 are sent to the image composition circuit 513 to generate composite image data. The generated composite image data is stored in the storage unit 508. The image processing performed by the image composition circuit 513 is, for example, addition composition processing for obtaining an addition average value for each pixel with respect to the image data for the left eye and the image data for the right eye. Even when the image data read from the image sensor 505 is photographed with different shapes of the subject for the left-eye image and the right-eye image, the shape of the subject is interpolated by the image composition processing, and image data with the correct shape is obtained. Generated. Note that the imaging signal processing circuit 506 may synthesize the left-eye image and the right-eye image after image processing.

  Next, the parallax map generation circuit 514 generates parallax map data using the left-eye image data and the right-eye image data. The generated parallax map data is stored in the storage unit 508. Information stored in the parallax map will be described with reference to FIG.

  FIG. 5 is a diagram schematically showing a captured image. FIG. 5A shows three subjects 602 to 604 having different distances from the camera in the image frame 601. An arrow Z in FIG. 5B represents the depth direction, and a subject 602 represents a subject farthest from the camera. A subject 604 represents a subject closest to the camera, and a subject 603 represents a subject located at a distance between the subjects 602 and 604.

  FIG. 5C shows a stereo image obtained when the composition shown by the image frame 601 in FIG. The left-eye image shown in the image frame 605 includes images 607L, 608L, and 609L corresponding to the subjects 602, 603, and 604, respectively. The right eye image shown in the image frame 606 includes images 607R, 608R, and 609R corresponding to the subjects 602, 603, and 604, respectively. A one-dot chain line in the figure represents the positional relationship between the subject images in the image frames 605 and 606, and the shift amount of the subject in the left-eye image and the right-eye image is defined as the parallax amount. The parallax amount 610 represents the positional deviation amount between the left-eye image and the right-eye image regarding the subject 602, that is, the parallax between the images 607L and 607R. Similarly, the parallax amount 611 represents the positional deviation amount of the images 609L and 609R with respect to the subject 604. The subject 603 shows a state in which the amount of parallax is zero between the left-eye image and the right-eye image.

When obtaining the parallax amount, the parallax map generation circuit 514 detects the positions of the subject images included in the left-eye image in the image frame 605 and the right-eye image in the image frame 606, and obtains the shift amount between the subject images. To calculate the amount of parallax. Specifically, when left-eye image data and right-eye image data are input to the parallax map generation circuit 514, parallax amounts 610 and 611 are calculated. The amount of parallax when the composite image is used as a reference is one half of the amounts of parallax 610 and 611.
Note that the parallax map data includes a form in which the image data is handled by being included in one image file, and a form in which both are handled as being associated with each other as separate files. In the present embodiment, an example in which an image file includes parallax map data will be described.

  FIG. 6 shows an example of the structure of a DCF image file including a parallax map generated at the time of shooting. The DCF (Design Rule for Camera File System) is an image file format for handling image data with a common specification in a digital camera. The image file has a DCF image data part 701 and a parallax map part 702. The DCF image data part 701 includes a DCF header part 703, a thumbnail image part 704, and a JPEG image part 705. The parallax map unit 702 includes a thumbnail image parallax map unit 706 and a JPEG image parallax map unit 707.

The DCF header portion 703 is an area for storing DCF header information, and is given a predetermined data size in advance. The DCF header information includes the following supplementary information and offset information of an area for storing each image data (a symbol is shown in parentheses).
Metadata A (708): shooting information and parameters related to image data stored in the JPEG image portion 705.
Offset value B (709): Offset data up to the top position of the thumbnail image portion 704.
Offset value C (710): Offset data up to the top position of the JPEG image portion 705.
Offset value D (711): Offset data to the top position of the thumbnail image parallax map unit 706.
Offset value E (712): Offset data up to the top position of the JPEG image parallax map unit 707.
The offset values B to E are relative position information from the reference position of the DCF header portion 703 to each image portion, and the start position of each image data in the image file is specified by these values.

  The JPEG image portion 705 stores JPEG image data obtained by combining the right eye image and the left eye image. The disparity map unit 707 stores JPEG image disparity map data. The thumbnail image portion 704 stores thumbnail image data that has been resized by thinning out the JPEG image data stored in the JPEG image portion 705. The parallax map unit 706 stores parallax map data generated in accordance with the size of the thumbnail image.

Next, the parallax enlargement process will be described with reference to FIG.
In FIG. 7, each subject is represented by a circle and a horizontally long rectangle. FIG. 7A is a side view showing the positional relationship 801 between subjects in the depth direction, and the front view shows the positional relationship 802 between subjects when viewed from the front. The data 803 exemplifies an image file including composite image data and parallax map data of a left-eye image and a right-eye image obtained when a subject is photographed from the front. The numerical sequence represents the amount of parallax.
An image 804 illustrated in FIG. 7B indicates an image corresponding to data obtained by combining the data of the left-eye image and the right-eye image acquired from the JPEG image unit 705 (see FIG. 6) or the thumbnail image unit 704. . A disparity map 805 indicates data of a disparity map acquired from the disparity map units 706 and 707 (see FIG. 6). In the parallax map 805, the parallax value of the background portion is set to zero, the parallax value of the rectangular subject is set to 1, and the parallax value of the circular subject is set to 2. This value is not limited to a positive integer, and may be a negative integer or a real number.

  For the stereoscopic image before the parallax expansion, the parallax map generation circuit 514 uses the data of the image 804 and the parallax map 805 to generate data for the left-eye image 806 and the right-eye image 807. The parallax map generation circuit 514 performs a parallax amount changing process on the data of the parallax map 805 by performing a weighting process described later. An example of the parallax map when the parallax map 808 changes the parallax amount by two times by weighting processing is illustrated. The parallax map generation circuit 514 executes the parallax enlargement process using the image 804 obtained by combining the data of the left-eye image and the right-eye image and the data of the parallax map 808. Thereby, the image for left eye 809 and the image for right eye 810 after the amount of parallax is changed are generated, respectively.

[First Embodiment]
Next, an image reproduction process according to the first embodiment of the present invention will be described with reference to FIGS. 8 to 10, FIG. 12, and FIG. 8 to 10 are flowcharts for explaining image reproduction processing. FIG. 12 illustrates a left-eye image, a right-eye image, and a composite image of both images generated by the image reproduction process. FIG. 13 shows a display example when 3D display is performed on the screen of the display unit 521.

  Each process illustrated in FIG. 8 is realized by the CPU of the system control unit 509 reading and executing an image processing program from the memory. In step S <b> 901, the system control unit 509 determines the operation state of the image playback button from the signal acquired from the external I / F unit 512. If it is determined that the image playback button has not been operated, the process of S901 is repeated. If it is determined that the image playback button has been operated, the process proceeds to S902. In step S902, an image file reading process is executed. The image file saved in the recording medium 511 is read, and the data is stored in the storage unit 508. In the next step S903, analysis processing of the image file read into the storage unit 508 is executed. In the analysis processing, the structure of the DCF image data described with reference to FIG. 6 is analyzed, and thereby, access to each image data becomes possible.

Next, S904 is processing for determining whether the image file is a stereoscopic image file from the analysis result in S903. In this process, it is determined whether a left-eye image and a right-eye image can be prepared. In the present embodiment, in the case of a file having a structure in which the parallax map and the composite image data described with reference to FIG. 6 are recorded, the system control unit 509 determines that the file is a stereoscopic image file, and advances the process to S905. If it is determined that the file is not a stereoscopic image file, the process proceeds to S906.
In step S905, it is determined whether or not the stereoscopic image playback setting is set. The reproduction setting is performed so that the user can arbitrarily select whether to reproduce an image in 2D display or to reproduce an image in 3D display. The setting operation may be set in advance by the user using the operation unit on the setting screen. If it is determined in S905 that the stereoscopic image playback setting is not set, the process proceeds to S906, and the 2D playback process is executed. An image 1301 shown in FIG. 12 shows a state in which a circular subject image is displayed in 2D.

  On the other hand, if it is determined in S905 that the stereoscopic image playback setting is set, the process proceeds to S907, and the 3D playback process is executed. Details of the 3D playback processing will be described later with reference to FIG. Next, S908 is a process for determining whether or not the image display has been completed. If the display of all the images has not been completed, the process returns to S902 and the next image reproduction process is continued. When the image display is completed, a series of processing is terminated.

Next, the 3D playback process shown in S907 of FIG. 8 will be described with reference to FIG. This process is executed under the control of the system control unit 509.
First, in S1001, image data (see JPEG image portion 705 in FIG. 6) acquisition processing is executed. In the present embodiment, a process of acquiring composite image data from the image file analyzed in S903 of FIG. 8 is performed. In the next step S1002, a process for acquiring parallax map data of an image (see the parallax map unit 707 in FIG. 6) is executed. In S1003, a weighting process is performed on the disparity map data acquired in S1002. By expanding the parallax through the weighting process, the user's eyes can have a more stereoscopic effect during 3D display. An image example 1302 in FIG. 12 shows a composite image, a left-eye image, and a right-eye image before parallax expansion. An image example 1303 shows a state in which a missing portion is generated in the image after parallax enlargement. An image example 1401 in FIG. 13 shows a display example when an image before parallax enlargement is displayed in 3D on the screen of the display unit 521, and an image example 1402 shows an image display example after parallax enlargement.

If a portion that exceeds the image display area due to the parallax enlargement is lost from the image displayed in the 3D display before the parallax enlargement, the viewer may feel uncomfortable. Therefore, in this embodiment, as shown in image examples 1304 to 1306 in FIG. 12, processing for enlarging the image display area is executed.
In S1004 of FIG. 9, left-eye image data and right-eye image data are generated based on the image data acquired in S1001 and the parallax map data after the parallax expansion processed in S1003, and reproduction processing is performed.

Next, the reproduction processing in S1004 of FIG. 9 will be described with reference to FIG.
S1101 is a determination process regarding whether or not there is an area that can be enlarged and displayed on the left side of the image display area. If it is determined that there is an area that can be enlarged, the process proceeds to S1102, and if it is determined that there is no such area, the process proceeds to S1103.

  In S1102, the enlargement process is performed by changing the parameter for the left area of the image display area. The area enlargement process is performed by the display control circuit 522 according to a control command from the system control unit 509. An image example 1306 in FIG. 12 shows a state in which the image display area is enlarged to the left. Then, the process proceeds to S1103. S1103 is a process for determining whether there is an area that can be enlarged and displayed on the right side of the image display area. If it is determined that there is an area that can be enlarged, the process proceeds to S1104. If it is determined that there is no such area, the process proceeds to S1105. In S1104, the enlargement process is performed by changing the parameter for the right region of the image display area. An image example 1305 in FIG. 12 shows a state in which the image display area is enlarged to the right side. When the enlargement processing of the image display areas on the left and right sides is performed, a state shown in an image example 1304 in FIG. 12 is obtained. Thereafter, the process proceeds to S1105.

  In S1105, a data generation process for the image for the left eye is performed from the image data acquired in S1001 in FIG. 9 and the parallax map data generated in S1003. Data generation processing is performed using the compression / decompression circuit 520 under the control of the system control unit 509. In step S 1106, the right eye image data generation process is performed using the compression / decompression circuit 520. In S1107, the left-eye image data generated in S1105 and the right-eye image data generated in S1106 are sent from the display control circuit 522 to the display unit 521, and the image is displayed in 3D on the screen.

  In the first embodiment, when performing parallax enlargement processing and performing 3D playback, if there is an area that can be enlarged and displayed on the left or right side of the image display area, the image display area is enlarged. By preventing the missing portion of the image from occurring when the stereoscopic display image is corrected, a stereoscopic effect without any sense of incongruity during viewing can be obtained.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, the same configurations as those in the first embodiment are omitted by using the reference numerals already used, and the differences will be mainly described below.
In the first embodiment, when the image display area can be enlarged only in one of the left and right directions in the image display area, a missing portion may occur in the image as shown in the display example 1403 in FIG. That is, an image is missing and displayed on the side where the image display area cannot be enlarged. Therefore, in the second embodiment, description will be given of a process that enables 3D display by performing parallax expansion without causing a missing part of an image by changing the content of the weighting process in S1003 of FIG.

A 3D playback process according to the present embodiment will be described with reference to FIGS.
FIG. 11 is a flowchart illustrating an example of weighting processing for parallax map data.
In step S1201, the system control unit 509 determines whether there is an area that can be enlarged and displayed only on the left side of the image display area. If it is determined that there is an area that can be enlarged and displayed only on the left side, the process proceeds to S1202, and if there is no such area, the process proceeds to S1203. In S1202, parallax map data is weighted in the left direction of the image.

In step S1203, the system control unit 509 determines whether there is an area that can be enlarged and displayed only on the right side of the image display area. If it is determined that there is an area that can be enlarged and displayed only on the right side, the process proceeds to S1204, and if there is no such area, the process proceeds to S1205. In step S1204, parallax map data is weighted in the right direction of the image.
In step S1205, the parallax map data is weighted equally in the left-right direction of the image.

  In the second embodiment, the content of the parallax map data weighting process is changed according to the determination result of the area that can be enlarged and displayed. As a result, when only the right side can be magnified and displayed (see image example 1305 in FIG. 12) or only the left side can be magnified and displayed (see image example 1306 in FIG. 12), the stereoscopic display image can be modified to correct the image. Missing can be prevented. In the display example 1404 shown in FIG. 13, the parallax map data is corrected and 3D display is performed so that a missing portion does not occur on the left side or the right side of the image.

  In the above example, the case where there is an area that can be enlarged and displayed only on the left side and the case that there is an area that can be enlarged and displayed only on the right side have been described. In addition, when there are areas that can be enlarged and displayed on both the left side and the right side of the image, the size of the area that can be enlarged and displayed may be different between the left side and the right side. In this case, weighting processing of the left and right parallax map data is performed so that the image after parallax enlargement falls within the range in which display enlargement is possible. That is, the parallax map data is corrected in accordance with the size of each area that can be enlarged and displayed on the left side and the right side so that the image is not lost.

[Other embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

501: Lens unit 505: Imaging device 506: Imaging signal processing circuit 509: System control unit 513: Image synthesis circuit 514: Parallax map generation circuit 520: Compression / decompression circuit 521: Display unit 522: Display control circuit

Claims (7)

An image processing apparatus that acquires image data for stereoscopic display and data on the amount of parallax thereof, and generates image data for left eye and image data for right eye,
Control means for controlling processing for enlarging an image display area related to the image data;
Parallax amount changing means for changing the parallax amount in accordance with the enlarged image display area;
An image processing apparatus comprising: data generation means for generating the left-eye image data and the right-eye image data using the image data and the parallax amount data changed by the parallax amount changing means.   When there is an area in which the image display area can be displayed by enlarging the image display area to the left or right, the control means controls a process of enlarging the image display area to a range including the area, and the parallax amount changing means The image processing apparatus according to claim 1, wherein the parallax amount is changed in accordance with a direction in which the image display area is enlarged.   The data generation means uses the combined image data obtained by combining the left-eye image data and the right-eye image data, and the parallax amount data changed according to the image display area by the parallax amount changing means. The image processing apparatus according to claim 1, wherein the left-eye image data and the right-eye image data are generated. The imaging element has a plurality of photoelectric conversion units that receive and photoelectrically convert light beams that have passed through different regions of the exit pupil of the imaging optical system for each microlens, and image data for stereoscopic display by the imaging element, and An imaging device that generates left-eye image data and right-eye image data from the parallax amount data,
Control means for controlling processing for enlarging an image display area related to the image data;
Parallax amount changing means for changing the parallax amount in accordance with the enlarged image display area;
An imaging apparatus comprising: data generation means for generating the left-eye image data and the right-eye image data using the image data and the parallax amount data changed by the parallax amount changing means. A control method executed by an image processing apparatus that acquires image data for stereoscopic display and data on the amount of parallax thereof and generates image data for left eye and image data for right eye,
An enlargement processing step for performing processing for enlarging an image display area related to the image data;
A parallax amount changing step for changing the parallax amount according to the enlarged image display area;
A control method for an image processing apparatus, comprising: a data generation step of generating the left-eye image data and the right-eye image data using the image data and the changed parallax amount data. The imaging element has a plurality of photoelectric conversion units that receive and photoelectrically convert light beams that have passed through different regions of the exit pupil of the imaging optical system for each microlens, and image data for stereoscopic display by the imaging element, and A control method executed by an imaging device that generates left-eye image data and right-eye image data from the parallax amount data,
An enlargement processing step for performing processing for enlarging an image display area related to the image data;
A parallax amount changing step for changing the parallax amount according to the enlarged image display area;
A method for controlling an imaging apparatus, comprising: a data generation step of generating the left-eye image data and the right-eye image data using the image data and the changed parallax data. An image processing program that is executed by a computer in an image processing device that acquires image data for stereoscopic display and data on the amount of parallax thereof and generates image data for left eye and image data for right eye,
An enlargement processing step for performing processing for enlarging an image display area related to the image data;
A parallax amount changing step for changing the parallax amount according to the enlarged image display area;
An image processing program comprising a data generation step of generating the left-eye image data and the right-eye image data using the image data and the changed parallax data.

Images