JP2013090129A - Image processing apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a parallax image in which a stereoscopic effect is emphasized by performing image shift processing when generating a virtual viewpoint image on the basis of LR images as a three-dimensional image.SOLUTION: An image processing apparatus includes: an optimum shift amount calculation part for calculating an optimum value of an image shift amount within a comfortable parallax range by inputting an L image which is an image signal for a left eye and an R image applied to three-dimensional image display and parallax information of the LR images; and a virtual viewpoint image generation part for generating a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR images. The optimum shift amount calculation part calculates an optimum shift amount within a range which is equal to or less than a preset crosstalk amount by using correspondence data between a parallax amount and a stereoscopic effect and correspondence data between the parallax amount and the crosstalk amount and the virtual viewpoint image generation part generates a final virtual viewpoint image for output by performing image shift processing in accordance with the optimum shift amount calculated by the optimum shift amount calculation part.

Description

  The present disclosure relates to an image processing device, an image processing method, and a program. In particular, the present invention relates to an image processing apparatus, an image processing method, and a program for generating a multi-viewpoint image to be applied to three-dimensional (3D) image display.

In a three-dimensional (3D) image display process, a naked-eye type 3D display device that can perceive a stereoscopic image without wearing glasses has been put into practical use. The naked-eye 3D display device includes, for example, a lenticular sheet and a parallax barrier (parallax barrier) on a display surface, and controls an image entering the left eye and the right eye according to a viewing position.
That is, control is performed such that a left viewpoint image corresponding to an image observed from the left eye is observed on the left eye, and a right viewpoint image corresponding to an image observed from the right eye is observed on the right eye.

  However, with such a method, a correct stereoscopic view can be obtained only at a limited viewing position with respect to the display. Accordingly, when the user's observation position is different from the specified position, the right eye image (right viewpoint image) enters the left eye and the left eye image (left viewpoint image) enters the right eye. Crosstalk in which the left eye image and the right eye image are mixed occurs.

  In order to solve this problem, not only the standard left viewpoint image and right viewpoint image corresponding to one regular observation position, but also a new viewpoint that is set so that crosstalk does not occur when observed at other observation positions. The structure which produces | generates and displays the image from is proposed.

In addition to the original set of left-viewpoint image and right-viewpoint image, other virtual viewpoint images are generated as multi-viewpoint images, and the optimal left-viewpoint image according to the user's observation position from these multi-viewpoint images And a right-viewpoint image set can be selected according to the observation position, and image display with reduced backsight and crosstalk is performed.
That is, by observing a different pair of left and right viewpoint images depending on the user's observation position, even when the user's observation position is changed, the left and right eyes of the observer are placed at the observation position, respectively. The corresponding left viewpoint image and right viewpoint image are observed.

  Specifically, based on the original two viewpoint images input to the display device or the image processing apparatus, that is, two viewpoint images of a left viewpoint image (L image) and a right viewpoint image (R image) for 3D image display. Thus, a viewpoint image of a virtual viewpoint other than these two viewpoints is generated. For example, ten different viewpoints including the original LR image are generated.

  3D that suppresses crosstalk in which left viewpoint image and right viewpoint image are mixed at various observation positions by combining two optimal images corresponding to the user's observation position with respect to the display from the generated multi-viewpoint images. Images can be displayed and observed.

  For example, in Patent Document 1 (Japanese Patent Laid-Open No. 2006-115198), an original left viewpoint image (L image) and a right viewpoint image (R image) are input, and parallax detection is executed from these two images. A method for generating images of a plurality of virtual viewpoints based on detected parallax information is disclosed. Specifically, the parallax is detected from two original 3D images of the input left viewpoint image (L image) and the right viewpoint image (R image), and the input LR image is determined based on the crosstalk amount and the fusion parallax range. A method for determining different virtual viewpoint positions is disclosed.

  However, when a virtual viewpoint image is generated by the process described in Patent Document 1, there is a problem that even if a virtual viewpoint image with a small parallax is generated, a sufficient stereoscopic effect cannot be enjoyed.

In particular, when two virtual viewpoint images corresponding to the viewpoints between the input original LR images, that is, two virtual viewpoint images interpolated between the input LR images are applied and observed, the parallax of these virtual viewpoint images is It becomes smaller than the parallax of the LR image, and it is difficult to obtain a stereoscopic effect.
However, if a configuration for performing processing for adjusting the parallax is added, the cost increases, image quality degradation due to parallax adjustment, and the like, and the quality of the generated virtual viewpoint image decreases, and the image cannot be observed. May be displayed.

There is a close relationship between the virtual viewpoint position and the image quality.
For example,
The viewpoint position of the input L image is 0.0,
The viewpoint position of the input R image is 1.0,
Then,
The relationship between the newly generated virtual viewpoint image and the image quality has the following characteristics.
(Feature 1) At a virtual viewpoint position between 0.0 and 1.0 between the L image (0.0) and the R image (1.0),
The virtual viewpoint image of 0.5, which is the center position of the LR image, has the lowest image quality compared to other virtual viewpoint positions.
(Feature 2) At a virtual viewpoint position of 0.0 or less, 1.0 or more on the left side of the L image or on the right side of the R image,
The further away from the L or R image, the lower the video quality.

Such a relationship between the virtual viewpoint position and the image quality is attributed to, for example, the accuracy of parallax detection, the amount of the occlusion area included in the image, and the like.
Since the original input left eye image can be used as it is at the viewpoint position = 0.0, and the original input right eye image can be used as it is at the viewpoint position = 1.0, the image quality is highest.

JP 2006-115198 A

  The present disclosure solves, for example, the above-described problem, and performs image processing for generating a high-quality multi-viewpoint image based on a left viewpoint image (L image) and a right viewpoint image (R image) for 3D images An object is to provide an apparatus, an image processing method, and a program.

  Further, the present disclosure provides a virtual viewpoint image generated by applying an input LR image in a configuration in which multi-viewpoint image generation processing based on a left viewpoint image (L image) and a right viewpoint image (R image) for a 3D image is performed. The present invention provides an image processing apparatus, an image processing method, and a program capable of emphasizing a three-dimensional effect without applying a large processing process to the above.

The first aspect of the present disclosure is:
A left viewpoint image input unit for inputting a left viewpoint image (L image) which is an image signal for the left eye applied to three-dimensional image display;
A right viewpoint image input unit for inputting a right viewpoint image (R image) that is an image signal for the right eye applied to three-dimensional image display;
The parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image) is input, and the image within the comfortable parallax range An optimum shift amount calculation unit for calculating an optimum value of the shift amount;
A virtual viewpoint image generation unit that generates a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image;
The optimal shift amount calculation unit uses the correspondence data between the parallax amount and the stereoscopic effect, and the correspondence data between the parallax amount and the crosstalk amount, so that the optimum shift amount within a predetermined range of the crosstalk amount or less is used. To calculate
The virtual viewpoint image generation unit
An image processing device that performs image shift processing according to the optimal shift amount calculated by the optimal shift amount calculation unit on a virtual viewpoint image generated based on an input LR image to generate a final output virtual viewpoint image It is in.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the optimum shift amount calculation unit uses correspondence data between the parallax amount and the stereoscopic effect, and correspondence data between the parallax amount and the crosstalk amount, A pixel-corresponding evaluation value S for each pixel is calculated, and an image-unit evaluation value S for each image is calculated by adding the pixel-corresponding evaluation values s of the constituent pixels of the entire image. The shift amount is calculated as the optimum shift amount.

  Furthermore, in an embodiment of the image processing apparatus of the present disclosure, the optimum shift amount calculation unit uses correspondence data between the parallax amount and the stereoscopic effect, and correspondence data between the parallax amount and the crosstalk amount, A pixel-corresponding evaluation value s corresponding to each pixel is calculated, and the pixel-corresponding evaluation value s of the constituent pixels of the entire image is multiplied by a weighting factor that performs weighting for each image area, and is added according to each shift amount. The image unit evaluation value S is calculated, and the shift amount that maximizes the image unit evaluation value S is calculated as the optimum shift amount.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the optimum shift amount calculation unit changes the shift amount within the comfortable parallax range, with a range within which the crosstalk amount is equal to or less than a predetermined value as a comfortable parallax range. Then, a pixel-corresponding evaluation value s corresponding to each shift amount is calculated, and further, an image unit evaluation value S corresponding to the shift amount is calculated by adding the pixel-corresponding evaluation value s of the constituent pixels of the entire image. The shift amount that maximizes the unit evaluation value S is calculated as the optimum shift amount.

  Furthermore, in an embodiment of the image processing device according to the present disclosure, the optimum shift amount calculation unit changes the shift amount within the comfortable parallax range, with a range within which the crosstalk amount is equal to or less than a predetermined value as a comfortable parallax range. The pixel-corresponding evaluation value s corresponding to each shift amount is calculated, and each of the pixel-corresponding evaluation values s of the constituent pixels of the entire image is multiplied and added by a weighting coefficient that performs weighting in units of image areas. The image unit evaluation value S corresponding to the shift amount is calculated, and the shift amount that maximizes the image unit evaluation value S is calculated as the optimum shift amount.

  Furthermore, in an embodiment of the image processing device of the present disclosure, the image processing device further includes a parallax estimation unit that generates parallax information from the left viewpoint image (L image) and the right viewpoint image (R image). The virtual viewpoint image generation unit receives the left viewpoint image (L image), the right viewpoint image (R image), the parallax information, and the optimum shift amount, and inputs the viewpoint other than the viewpoint of the input LR image. A virtual viewpoint image including the viewpoint image is generated.

Furthermore, the second aspect of the present disclosure is:
An image processing method executed in an image processing apparatus,
Via the image input unit, a left viewpoint image (L image) that is an image signal for the left eye applied to a three-dimensional image display and a right viewpoint image (R image) that is an image signal for the right eye are input,
An optimal shift amount calculation unit inputs parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image). An optimal shift amount calculation process for calculating an optimal value of the image shift amount within the comfortable parallax range;
The virtual viewpoint image generation unit executes a virtual viewpoint image generation process for generating a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image,
The optimum shift amount calculation process includes:
A process of calculating an optimum shift amount within a range equal to or less than a predetermined crosstalk amount by using correspondence data between a parallax amount and a stereoscopic effect and correspondence data between a parallax amount and a crosstalk amount.
The virtual viewpoint image generation process includes:
Executed as a process for generating a final output virtual viewpoint image by performing an image shift process according to the optimal shift amount calculated by the optimal shift amount calculation unit on the virtual viewpoint image generated based on the input LR image There is an image processing method.

Furthermore, the third aspect of the present disclosure is:
A program for executing image processing in an image processing apparatus;
Processing for inputting a left viewpoint image (L image) that is an image signal for the left eye to be applied to the three-dimensional image display and a right viewpoint image (R image) that is an image signal for the right eye via the image input unit When,
An optimal shift amount calculation unit inputs parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image). An optimal shift amount calculation process for calculating an optimal value of the image shift amount within the comfortable parallax range;
The virtual viewpoint image generation unit executes a virtual viewpoint image generation process for generating a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image;
In the optimum shift amount calculation process,
Using the correspondence data between the parallax amount and the stereoscopic effect and the correspondence data between the parallax amount and the crosstalk amount, a process for calculating an optimum shift amount within a predetermined range of the crosstalk amount or less is executed.
In the virtual viewpoint image generation process,
The virtual viewpoint image generated based on the input LR image is subjected to image shift processing according to the optimum shift amount calculated by the optimum shift amount calculation unit, and the final output virtual viewpoint image is generated. It is in the program to let you.

  In addition, the program of this indication is a program which can be provided with the storage medium and communication medium which are provided with a computer-readable format with respect to the general purpose system which can execute various program codes, for example. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the computer system.

  Other objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present disclosure described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, it is possible to generate a parallax image with enhanced stereoscopic effect by performing image shift processing when generating a virtual viewpoint image based on an LR image as a three-dimensional image.
Specifically, an L image and an R image, which are image signals for the left eye applied to 3D image display, are input, and parallax information of the L image, the R image, and the LR image is input, and a comfortable parallax range is input. An optimal shift amount calculation unit that calculates an optimal value of the image shift amount, and a virtual viewpoint image generation unit that generates a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image. The optimum shift amount calculation unit uses the correspondence data between the parallax amount and the stereoscopic effect and the correspondence data between the parallax amount and the crosstalk amount to calculate an optimum shift amount within a predetermined range of the crosstalk amount or less. Then, the virtual viewpoint image generation unit performs an image shift process according to the optimum shift amount calculated by the optimum shift amount calculation unit, and generates a final output virtual viewpoint image.
With this process, when generating a virtual viewpoint image based on an LR image as a three-dimensional image, it is possible to generate a parallax image with enhanced stereoscopic effect by performing an image shift process.

It is a figure explaining the structural example of an image processing apparatus. It is a figure explaining the structural example of an image processing apparatus. It is a figure explaining the example of a process of a parallax estimation part. It is a figure explaining an example of the production | generation process of a virtual viewpoint image. It is a figure explaining an example of the production | generation process of a virtual viewpoint image. It is a figure explaining an example of the production | generation process of a virtual viewpoint image. It is a figure explaining the process which the optimal shift amount calculation part performs. It is a figure explaining the process which the optimal shift amount calculation part performs. It is a figure explaining the process which the optimal shift amount calculation part performs. It is a figure explaining the process which the optimal shift amount calculation part performs.

The details of the image processing apparatus, the image processing method, and the program of the present disclosure will be described below with reference to the drawings. The explanation will be made according to the following items.
1. 1. Overview of overall configuration and processing of image processing apparatus 2. Details of disparity information and processing of the disparity estimation unit 3. Basic virtual viewpoint image generation processing based on the input LR image executed by the virtual viewpoint image generation unit 4. Process of optimum shift amount calculation unit 5. Processing of virtual viewpoint image generation unit Summary of composition of this disclosure

[1. Overview of overall configuration and processing of image processing apparatus]
First, an overall configuration of the image processing apparatus according to the present disclosure and an outline of processing will be described with reference to FIG. Then, the detail of the process of each structure part is demonstrated.
As shown in FIG. 1, the image processing apparatus 100 includes a left viewpoint image (L image) input unit 101, a right viewpoint image (R image) input unit 102, a parallax information input unit 103, an optimum shift amount calculation unit 104, a virtual viewpoint. An image generation unit 105 and a display control unit 106 are provided, and an image generated by the image processing apparatus 100 is output to the display unit 110.
In the configuration illustrated in FIG. 1, the display unit 110 is illustrated as an external configuration of the image processing apparatus 100. However, the display unit 110 may be included in the image processing apparatus 100.

FIG. 1 shows a main configuration of the image processing apparatus. In addition to the configuration shown in the figure, the image processing apparatus 100 includes a control unit having a program execution function such as a CPU for executing data processing control, A program executed in the control unit, a storage unit storing various parameters, and an input unit for inputting parameters, image data, and the like.
For example, a control part performs the process demonstrated below according to the program previously stored in the memory | storage part.

The left viewpoint image (L image) input unit 101 and the right viewpoint image (R image) input unit 102 include a left viewpoint image (L image) for displaying a three-dimensional (3D) image generated in advance and a right viewpoint image ( R image).
The left viewpoint image (L image) corresponds to an image observed from the left eye, and the right viewpoint image (R image) corresponds to an image observed from the right eye.

However, these two images are two standard LR images. That is, in an autostereoscopic 3D display device provided with, for example, a lenticular sheet or a parallax barrier (parallax barrier) on the display surface, an LR image that is observed as a correct 3D image when the display is observed from the front center position. is there.
When the user's observation position is different from the specified position, the left eye has an image for the right eye (right viewpoint image) and the right eye has an image for the left eye (left viewpoint image). Crosstalk occurs in which the left viewpoint image and the right viewpoint image are mixed.

  In order to solve this problem, the image processing apparatus 100 performs various processing based on an input LR image corresponding to one normal observation position, that is, a standard left viewpoint image (L image) and a right viewpoint image (R image). An image from a new viewpoint (virtual viewpoint) that does not generate crosstalk when observed at various observation positions is generated.

The parallax information input unit 103 inputs parallax information between two images of an input LR image, that is, a left viewpoint image (L image) and a right viewpoint image (R image).
The parallax information corresponds to a shift (pixel shift in the left-right direction) between images of the same subject included in the input LR image, and is information corresponding to the distance of the subject. Specifically, for example, it is configured by parallax information (subject distance information) of each pixel unit or pixel region unit.

The optimum shift amount calculation unit 104
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Disparity information from the disparity information input unit 103,
Each piece of information is input, and the optimum amount of parallax is calculated according to a predetermined algorithm. Details of this processing will be described later.

The virtual viewpoint image generation unit 105
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Disparity information from the disparity information input unit 103,
Optimal parallax amount from the optimal shift amount calculation unit 104,
Each of these pieces of information is input to generate a virtual viewpoint image.

  The virtual viewpoint image generation unit 105 executes, for example, a virtual viewpoint position determination process according to the optimal parallax amount input from the optimal shift amount calculation unit 104, and generates a virtual viewpoint image corresponding to the determined virtual viewpoint position.

The virtual viewpoint image generation unit 105 generates and outputs a total of N viewpoint images in which images of other viewpoints are added to the two viewpoint images of the input LR image.
The virtual viewpoint image generated by the virtual viewpoint image generation unit 105 is output and displayed on the display unit 110 via the display control unit 106.

  The display image generated by the image processing device of the present disclosure is a display image in a naked-eye 3D display device that allows a user to view a stereoscopic image without wearing glasses.

The display unit 110 is a display unit that performs naked-eye 3D display. The display unit 110 includes, for example, a lenticular sheet or a parallax barrier (parallax barrier) on the display surface, and is capable of controlling an image entering the left eye and the right eye according to a viewing position. Part.
The display control unit 106 outputs the N viewpoint images generated by the virtual viewpoint image generation unit 105 to the display unit 110.
The display control unit 106 generates display information according to the display configuration of the display unit 110.

Note that the image processing apparatus 100 can also be configured as an imaging apparatus such as a camera including an imaging unit, or a display apparatus such as a PC or a television, and when configured as these apparatuses, functions corresponding to the respective apparatuses. It is set as the structure provided with.
For example, in the case of a camera, an imaging unit that captures an LR image as an image from a different viewpoint is used, and a multi-viewpoint image is generated using the LR image input from the imaging unit.

The configuration of the image processing apparatus illustrated in FIG. 1 is configured to input parallax information corresponding to an input LR image via the parallax information input unit 103 in addition to the input of the LR image.
In this way, the configuration may be such that the disparity information corresponding to the input LR image is generated inside the image processing apparatus, instead of the configuration in which the disparity information generated in advance is input.
A configuration example of such an image processing apparatus is shown in FIG.

2 includes a left viewpoint image (L image) input unit 101, a right viewpoint image (R image) input unit 102, a parallax estimation unit 115, an optimum shift amount calculation unit 104, and a virtual viewpoint image generation unit 105. The display control unit 106, and the image generated by the image processing device 120 is output to the display unit 110.
A difference from the image processing apparatus 100 illustrated in FIG. 1 is that the disparity estimation unit 115 is included instead of the disparity information input unit 103 illustrated in FIG.
Other configurations are the same as those shown in FIG.

The parallax estimation unit 115 receives a left viewpoint image (L image) and a right viewpoint image (R image), and generates parallax information based on these images.
The parallax information corresponds to a shift (pixel shift in the left-right direction) between images of the same subject included in the input LR image, and is information corresponding to the distance of the subject. Specifically, for example, data having parallax information (subject distance information) for each pixel unit or pixel region unit is generated.

Other configurations are the same as those shown in FIG.
The optimum shift amount calculation unit 104
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Parallax information from the parallax estimation unit 115,
Each piece of information is input, and the optimum amount of parallax is calculated according to a predetermined algorithm. Details of this processing will be described later.

The virtual viewpoint image generation unit 105
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Parallax information from the parallax estimation unit 115,
Optimal parallax amount from the optimal shift amount calculation unit 104,
Each of these pieces of information is input to generate a virtual viewpoint image.

[2. Disparity information and details of processing of disparity estimation unit]
Next, details of the disparity information and the processing of the disparity estimation unit will be described.
1 inputs disparity information corresponding to an input LR image from the outside, and the image processing device 120 illustrated in FIG. 2 generates disparity information corresponding to the input LR image in the disparity estimation unit 115. To do.
The parallax estimation unit 115 receives a left viewpoint image (L image) and a right viewpoint image (R image), and generates parallax information based on these images.
As described above, the parallax information corresponds to the shift between the images of the same subject included in the standard LR image (pixel shift in the left-right direction), and is information corresponding to the distance of the subject. Specifically, for example, data having parallax information (subject distance information) for each pixel unit is generated.

The acquisition of the parallax information is executed by the following existing method, for example.
(A) Block matching-based disparity information acquisition processing (b) DP (dynamic programming) matching-based disparity information acquisition processing (c) Segmentation-based disparity information acquisition processing (d) Learning-based disparity information acquisition processing (e) Disparity information acquisition processing by a combination of methods For example, disparity information is acquired by any one of the methods (a) to (e).

The block matching-based disparity information acquisition process will be briefly described with reference to FIG.
As shown in FIG. 3, it is the original standard image that was input.
(A) Left viewpoint image (L image);
(B) Right viewpoint image (R image),
Using these input LR images, for example, a pixel region (block) 121 of the L image is selected, and a block similar to the selected block is detected from the R image. That is, a block (matching block) that is determined to be a shooting area of the same subject is selected from the LR image. Furthermore, the positional deviation (the number of pixels in the horizontal direction, etc.) of the matching block between the LR images is measured.

In FIG. 3, for example, the corresponding pixel in the R image of the pixel of interest LP = (5, 3) in the pixel region (block) 121 of the L image is RP = (7, 3).
In this case, the parallax d (5, 3) between the LR images at the pixel position (x, y) = (5, 3) of the L image is calculated as follows.
Parallax d (5,3) = (7,3) − (5,3) = (2,0),
That is,
The parallax d of the pixel position (x, y) = (5, 3) of the L image is 2 pixels [pixel].

The displacement of the block varies according to the distance of the subject photographed in the block.
That is, the block displacement corresponds to the subject distance, and this displacement information is acquired as parallax information.
An example of the parallax information expression format is a depth map (distance image or parallax map). The depth map (parallax map) is an image in which the parallax (subject distance) of each pixel unit of the L image and the R image, for example, is expressed by the luminance of the pixel unit. For example, the high luminance region is a close subject (close to the camera), The low luminance area indicates a subject that is far (far from the camera). That is, it is an image in which the subject distance is indicated by luminance.

[3. Basic virtual viewpoint image generation process based on input LR image executed by virtual viewpoint image generation unit]
Next, basic virtual viewpoint image generation processing based on the input LR image executed by the virtual viewpoint image generation unit 105 will be described.
The virtual viewpoint image generation unit 105
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Parallax information from the parallax estimation unit 103,
Reliability information from the reliability calculation unit 104,
Each of these pieces of information is input to generate a virtual viewpoint image.

For example, a predetermined number (for example, 10) of virtual viewpoints are determined, and virtual viewpoint images corresponding to these virtual viewpoints are generated. The virtual viewpoint image is generated using the input standard LR image. That is, the left viewpoint image (L image) and the right viewpoint image (R image) that are input images are used.
A specific example of virtual viewpoint image generation processing will be described with reference to FIG.

FIG. 4 shows an original left viewpoint image (L image) 131 and right viewpoint image (R image) 132 input to the image processing apparatus, and a virtual viewpoint image 133 generated based on these LR images. .
The left viewpoint image (L image) 131 is an image observed from the left eye viewpoint position at the standard position, and the right viewpoint image (R image) 132 is an image observed from the right eye viewpoint position at the standard position.
The viewpoint position of the left viewpoint image (L image) 131 is 0.0,
The viewpoint position of the right viewpoint image (R image) 132 is 1.0.

FIG. 4 shows a processing example in the case where an observation image from a viewpoint position = 0.3 between the viewpoint positions 0.0 to 1.0 is generated as a virtual viewpoint image 133, for example.
In the left viewpoint image (L image) 131 and the right viewpoint image (R image) 132, the same subject (apple) is photographed at different positions. In the L image and the R image, the position of the same subject is different because the viewpoint position is different.
When generating the virtual viewpoint image 103 with the viewpoint position = 0.3 between the viewpoint position = 0.0 and the viewpoint position 1.0, the position of the subject (apple) is set by linear interpolation. By changing the subject position along the straight line L1 shown in FIG. 4, it is possible to generate the virtual viewpoint image by determining the subject position of the virtual viewpoint image at each virtual viewpoint.
Thus, the virtual viewpoint image of each virtual viewpoint position is generated by the linear interpolation process based on the input LR image.

Note that when generating a virtual viewpoint image, it can be generated by a process of blending two images using both of the input LR images.
Alternatively, it is possible to generate only one image by processing only the L image or only the R image to shift the subject position according to the virtual viewpoint position.
Alternatively, a process of generating a virtual viewpoint image using only the L image at a virtual viewpoint position close to the L image side and generating a virtual viewpoint image using only the R image at a position close to the R image may be performed. Good.

An example of a process for determining the pixel value of the virtual viewpoint image 131 based on the blend process of the input LR image will be described with reference to FIG.
In FIG.
Pixel P (x, y) 141 of the input left viewpoint image (L image) where the viewpoint position = 0,
The corresponding pixel 142 of the pixel P of the L image in the input right viewpoint image (R image) where the viewpoint position = 1.
The corresponding pixel 143 of the pixel P of the L image in the virtual viewpoint image with the viewpoint position = Φ,
These are shown.
In addition,
Φ = 0-1
It is.

When the parallax of the pixel P (x, y) 141 of the left viewpoint image (L image) is d (x, y) [pixel],
The pixel position of the corresponding pixel 143 of the pixel P (x, y) of the L image in the virtual viewpoint image is
It is assumed that the pixel is Q (x + Φ · d (x, y), y).
That is, the pixel value of the pixel Q (x + Φ · d (x, y), y) in the virtual viewpoint image is set to the pixel value of the pixel P (x, y) 141 of the left viewpoint image (L image).
As described above, the pixel value of each pixel of the virtual viewpoint image is set based on the parallax information of the pixel of the left viewpoint image (L image).
By this process, pixels that are not filled in the virtual viewpoint image are processed by applying the right viewpoint image (R image), interpolation processing by pixel values of adjacent pixels, processing by interpolation with pixels of the same coordinates of the left viewpoint image, etc. To determine the pixel value.

In FIG.
Horizontal line 151 of the left viewpoint image (L image),
Horizontal line 152 of right viewpoint image (R image)
Horizontal line 153 of the virtual viewpoint image,
Is shown.
The arrow shown in the figure is a line connecting the pixel position of the left viewpoint image (L image) and the pixel position of the right viewpoint image (R image) applicable to determine the pixel value of the horizontal line 153 of the virtual viewpoint image. is there.

In the horizontal line 153 of the virtual viewpoint image shown in FIG.
1 = region where the pixel value is set by the constituent pixel value of the horizontal line 151 of the left viewpoint image (L image) 2 = region where the pixel value is set by the constituent pixel value of the horizontal line 153 of the right viewpoint image (R image) 3 = Other area

Thus, the setting of the pixel value of the virtual viewpoint image is executed by the following three processes, for example.
1. For each pixel of the left viewpoint image (L image), a corresponding pixel position is obtained at the output viewpoint position, and the pixel value of the left viewpoint image (L image) is interpolated at that pixel position.
2. For each pixel of the right viewpoint image (R image), the corresponding pixel position is obtained at the output viewpoint position, and the pixel value of the right viewpoint image (R image) is interpolated at that pixel position.
3. The pixels of the output viewpoint image that are not interpolated by the processes 1 and 2 are subjected to interpolation processing based on adjacent pixels.

The processing described with reference to FIGS. 5 and 6 is basic processing for generating an image from a virtual viewpoint different from these LR images based on the input LR images.
Based on this basic processing, the virtual viewpoint image generation unit 105 of the image processing apparatus according to the present disclosure further applies the optimum shift amount information input from the optimum shift amount calculation unit 104 to execute image shift. A final virtual viewpoint image in which the stereoscopic effect is emphasized is generated.

[4. About processing of the optimum shift amount calculation unit]
Next, processing executed by the optimum shift amount calculation unit 104 will be described.
The optimum shift amount calculation unit 104
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Disparity information from the disparity information input unit 103 or the disparity estimation unit 115,
Each piece of information is input, and the optimum amount of parallax is calculated according to a predetermined algorithm.

Specifically, the optimum shift amount calculation unit 104 is based on the disparity information input from the outside in the configuration of FIG. 1 and the disparity information input from the disparity estimation unit 115 in the configuration shown in FIG.
Crosstalk amount,
Depth (≈ subject distance recognized by the observer)
Considering these pieces of information, a process for determining the optimum shift amount for the virtual viewpoint image is executed.

The optimum shift amount calculation unit 104 obtains, for example, the data shown in FIG. 7 stored in the storage unit in advance from the storage unit, and calculates the optimum parallax amount based on this data.
The data shown in FIG. 7 will be described.
FIG. 7 shows an example of data related to one pixel at one corresponding point in the input LR image.
For example, with respect to a certain pixel of an image for which parallax is set, that is, a pixel with parallax: x = d, a change in the depth sensation D (= object distance recognized by the observer) when various shift amounts Δd are set, It is a figure which shows the change of the talk amount C. FIG.
As described above, the crosstalk is a reverse viewing or left viewpoint image in which an image for the right eye (right viewpoint image) enters the left eye and an image for the left eye (left viewpoint image) enters the right eye. And the right viewpoint image. Therefore, a smaller crosstalk amount is a preferable image.

“Parallax x = d + Δd” on the horizontal axis
“Vertical D” and “Crosstalk C” on the vertical axis
These relational data are shown.
The shift amount indicates, for example, a shift process for moving a pixel to the left in the horizontal direction as a shift process in the-direction and a shift process for moving the pixel in the right direction as a shift process in the + direction.
Further, a range in which the crosstalk amount is equal to or less than a predetermined value is defined as a comfortable parallax range.
This comfortable parallax range is, for example, as shown in the figure.
Parallax x = x1 to x2
Set this range.

A correspondence relationship between the parallax and the sense of depth D (= the subject distance recognized by the observer) will be described with reference to FIG.
In FIG.
(A) Parallax <0
(B) Parallax = 0
(C) Parallax> 0
The observation state of the LR image having the parallax of these three modes is shown.

A display surface 301 shown in the figure is a display surface of the display.
On the display surface, images of the same subject of the left viewpoint image (L image) and the right viewpoint image (R image) are displayed with their positions shifted in the horizontal direction, and parallax occurs, and the observation position is on the display surface. It will be set back and forth.

  In the case of parallax = 0 in FIG. 8B, the display is the same as the two-dimensional display, and the image of the same subject of the left viewpoint image (L image) and the right viewpoint image (R image) is positioned on the display surface. Displayed without deviation. In this case, the observation position 312 of the display object is on the display surface.

  On the other hand, when the parallax <0 in FIG. 8A, the left viewpoint image (L image) is shifted to the left side and the right viewpoint image (R image) is shifted to the right side and displayed on the display surface. This “deviation” of the display position becomes parallax, and in this case, the observation position of the display object is a position far from the display surface. The observation position 311 shown in FIG.

  On the other hand, when the parallax> 0 in FIG. 8C, the left viewpoint image (L image) is shifted to the right side and the right viewpoint image (R image) is shifted to the left side and displayed on the display surface. This “displacement” of the display position becomes parallax. In this case, the observation position of the display object is closer to the display surface. The observation position 313 shown in FIG.

Thus, the depth feeling D (= the subject distance recognized by the observer) is changed according to the parallax.
As understood from the three data shown in FIG. 8, it is the position of the corresponding pixel of the LR image displayed on the display surface 301 that determines the parallax.
For example, by shifting the display positions of the left viewpoint image (L image) and the right viewpoint image (R image) shown in FIG. 8A in the horizontal direction, the settings in FIG. 8B and the settings in FIG. Is possible. That is, the sense of depth can be changed by the shift process.
The correspondence between the shift processing and the feeling of depth is a line of the feeling of depth D shown in FIG.

Also, for example, the depth range can be expanded by a shift process on one virtual viewpoint image generated based on the input LR image.
This example will be described with reference to FIG.

FIG. 9 shows the following three LR image parallax ranges (depth ranges).
(1) Parallax range (depth range) of input LR image
(2) Parallax range (depth range) of virtual viewpoint image by viewpoint interpolation processing
(3) The parallax range (depth range) of the shift image generated by executing the shift process on the virtual viewpoint image by the viewpoint interpolation process of (2) above.

  FIG. 9B shows a parallax range (depth range) in a virtual viewpoint image in which the viewpoint is set on the inner side than the viewpoint of the input LR image. This parallax range is smaller than the parallax range of the input LR image. That is, a virtual viewpoint image obtained by viewpoint interpolation is an image with a reduced depth range.

As shown in FIG. 9 (3), the depth range of the shifted image generated by executing the image shift on the virtual viewpoint image shown in FIG. 9 (2) is expanded.
That is, the observer can feel a greater distance between the farthest subject and the nearest subject included in the image.

As shown in FIG.
When the subject distance from the position of the eyes of the observer is Ld,
The subject distance Ld is calculated by the following equation.
Ld = (Ls · de) / (de + x)
However,
Ls: Distance between the observer's eyes and the display surface de: Distance between the observer's left and right eyes x: d + Δd
d: parallax in input LR image Δd: shift amount

As described with reference to FIGS. 8 to 10, the depth range expansion effect can be obtained by performing image shift processing on an image having a predetermined parallax.
However, depending on the shift amount, the crosstalk amount may increase, and a comfortable 3D image may not be observed. As described above, the amount of crosstalk C is the reverse viewing or the left eye image in which the right eye image (right viewpoint image) enters the left eye and the left eye image (left viewpoint image) enters the right eye. This is a phenomenon in which the viewpoint image and the right viewpoint image are mixed.
As shown in FIG. 7, the crosstalk amount C tends to increase as the absolute value of parallax increases, that is, as the distance from the position where parallax = 0 is increased.
However, the amount of crosstalk varies depending on the display that displays the 3D image.

The optimum shift amount calculation unit 104 has the data shown in FIG.
Correspondence between parallax (x) and depth (D) that can be adjusted by shift processing of the shift amount (Δd),
Correspondence between parallax (x) and crosstalk amount (C) that can be adjusted by shift processing of shift amount (Δd),
The shift amount in the comfortable parallax range is determined in consideration of these correspondence data.
Note that the data shown in FIG.
Correspondence data between the parallax x and the sense of depth D;
Correspondence data between the parallax x and the crosstalk amount C;
These relational data are measured or calculated in advance and stored in the storage unit of the image processing apparatus.
In addition, a parallax range in which a predetermined sense of depth D is obtained and the crosstalk amount is equal to or less than a predetermined value is defined as a comfortable parallax range. For example, the range of parallax x = x1 to x2 shown in FIG. 7 is set as a comfortable parallax range, and this comfortable parallax range information is stored in the storage unit.

The optimum shift amount calculation unit 104 stores these data, that is,
Correspondence data between the parallax x and the sense of depth D;
Correspondence data between the parallax x and the crosstalk amount C;
Comfortable parallax range information: parallax x = x1 to x2
These data are acquired from the storage unit, and the optimum shift amount is determined.

The shift amount determination process is performed according to the following algorithm.
(Step 1) An evaluation value s is calculated for each pixel using a predetermined evaluation function.
(Step 2) The evaluation value s of each pixel is summed to calculate an evaluation value S for one frame.
(Step 3) Based on the settable shift amount Δd within the predetermined comfortable parallax range shown in FIG. 7, where Δd is the horizontal shift amount of one of the LR images to be executed for parallax adjustment of the image An evaluation value S (Δd) for each image frame is calculated.
(Step 4) Among the evaluation values S (Δd) in units of image frames calculated according to the settable shift amount Δd within the comfortable parallax range, the shift amount Δd ′ that is the maximum evaluation value is set as the optimum shift amount. .

The optimal shift amount calculation unit 104 generates an optimal shift amount Δd ′ according to the above algorithm and outputs it to the virtual viewpoint image generation unit 105.
The virtual viewpoint image generation unit 105 generates a final virtual viewpoint image by executing horizontal shift processing to which the optimal shift amount Δd ′ is applied to the virtual viewpoint image generated based on the input LR image.

The evaluation function used in the above (Step 1) will be described. The evaluation value s for each pixel is calculated by the following evaluation function, for example.
s = α · D (d + Δd) −β · C (d + Δd)
In addition,
α and β are preset weighting factors D are depth sense data (see FIG. 7),
C is the amount of crosstalk (see FIG. 7),
d is parallax information,
Δd is a variable indicating the shift amount,
It is.
With the settings shown in FIG.
Parallax x = d + Δd,
The variable Δd is set in the range of x = x1 to x2, and the evaluation value s corresponding to the value of each shift amount Δd is calculated.

The disparity information d is disparity information for each pixel, and is input from the disparity information input unit 1103 in the configuration of FIG. 1 and input from the disparity estimation unit 115 in the configuration of FIG.
The depth sensation data D and the crosstalk amount C are calculated based on the relation data shown in FIG. 7 acquired from the storage unit based on the input parallax information d.

Δd is a variable and is set within a predetermined comfortable parallax range shown in FIG.
An evaluation value s (Δd) for each pixel of the image frame based on each shift amount Δd is calculated, and an addition value of the evaluation value s (Δd) for the pixels of the entire image of one frame is calculated as an evaluation value S for each image frame. Calculate as (Δd).

Note that a simple addition value of the evaluation value s (Δd) in pixel units may be calculated as the evaluation value S (Δd) in image frame units. However, the evaluation value of each pixel is further weighted. An addition value of the weighted evaluation values may be calculated as an evaluation value for the entire image frame.
As the weighting, for example, a setting for increasing the weight for the attention area, for example, the central portion of the image or the person area is effective. Specifically, it is possible to perform processing of setting a large number of weight examples in the attention area by using “saliency map” of “Visual Attention” and the like.

  In this way, among the evaluation values S (Δd) for each image frame calculated according to the shift amount Δd that can be set within the comfortable parallax range, the shift amount Δd ′ that is the maximum evaluation value is determined as the optimum shift amount. To do.

  The optimal shift amount calculation unit 104 thus generates the optimal shift amount Δd ′ and outputs it to the virtual viewpoint image generation unit 105. The virtual viewpoint image generation unit 105 generates a shift image in which the optimal shift amount Δd ′ is applied as a horizontal shift amount with respect to the virtual viewpoint image generated based on the LR image, and uses this as a virtual viewpoint image to be finally output. .

  By such an image shift, it is possible to generate a virtual viewpoint image that realizes a predetermined sense of depth within a comfortable parallax range in which the crosstalk amount is suppressed to a predetermined value or less.

  The optimal shift amount calculation unit 104 is a shift that is the shift amount within the comfortable parallax range shown in FIG. 7 and has the maximum evaluation value S for each image obtained as a total of the evaluation values s of the constituent pixels of the image frame. The quantity Δd ′ is determined.

  By the shift process using this shift amount Δd ′, it is possible to generate and output a virtual viewpoint image within a comfortable parallax range and having a sufficient sense of depth.

[5. About processing of virtual viewpoint image generation unit]
Next, a specific processing example of the virtual viewpoint image generation unit 105 illustrated in FIGS. 1 and 2 will be described.
The virtual viewpoint image generation unit 105
From the left viewpoint image (L image) input unit 101, an L image,
R image from the right viewpoint image (R image) input unit 102,
Parallax information from the parallax estimation unit 103,
Optimal shift amount from the optimal shift amount calculation unit 104,
Each of these pieces of information is input to generate a virtual viewpoint image.

The virtual viewpoint image generation unit 105 applies the L image from the left viewpoint image (L image) input unit 101 and the R image from the right viewpoint image (R image) input unit 102 to generate a virtual viewpoint image. First, a predetermined number (N) of virtual viewpoint images are generated according to the processing described above with reference to FIGS.
Further, a shift process is performed on the virtual viewpoint image. That is, a shift process in which the optimal shift amount Δd ′ input from the optimal shift amount calculation unit 104 is applied as the horizontal shift amount of the constituent pixels of the virtual viewpoint image is executed, and this is set as a virtual viewpoint image to be output.

  By this processing, it is possible to generate and output a virtual viewpoint image having a sufficient depth feeling within a comfortable parallax range.

  Note that, as described above, the display image generated by the image processing device of the present disclosure is a display image in a naked-eye 3D display device that allows a user to view a stereoscopic image without wearing glasses.

The display unit 110 illustrated in FIGS. 1 and 2 is a display unit that performs naked-eye 3D display. The display unit 110 includes, for example, a lenticular sheet and a parallax barrier (parallax barrier) on the display surface, and enters the left eye and the right eye depending on the viewing position. This is a display unit capable of controlling an image.
The display control unit 106 outputs the virtual viewpoint image generated by the virtual viewpoint image generation unit 105 to the display unit 110.
The display control unit 106 generates display information according to the display configuration of the display unit 110.

Note that the image processing device can be configured as an imaging device such as a camera provided with an imaging unit, a display device such as a PC or a television, and when configured as these devices, functions corresponding to each device are provided. It is set as the structure provided.
For example, in the case of a camera, an imaging unit that captures an LR image as an image from a different viewpoint is used, and a multi-viewpoint image is generated using the LR image input from the imaging unit.

[6. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) a left viewpoint image input unit that inputs a left viewpoint image (L image) that is an image signal for the left eye applied to three-dimensional image display;
A right viewpoint image input unit for inputting a right viewpoint image (R image) that is an image signal for the right eye applied to three-dimensional image display;
The parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image) is input, and the image within the comfortable parallax range An optimum shift amount calculation unit for calculating an optimum value of the shift amount;
A virtual viewpoint image generation unit that generates a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image;
The optimal shift amount calculation unit uses the correspondence data between the parallax amount and the stereoscopic effect, and the correspondence data between the parallax amount and the crosstalk amount, so that the optimum shift amount within a predetermined range of the crosstalk amount or less is used. To calculate
The virtual viewpoint image generation unit
An image processing device that performs image shift processing according to the optimal shift amount calculated by the optimal shift amount calculation unit on a virtual viewpoint image generated based on an input LR image to generate a final output virtual viewpoint image .

(2) The optimum shift amount calculation unit calculates a pixel correspondence evaluation value s corresponding to each pixel by using correspondence data between the parallax amount and the stereoscopic effect and correspondence data between the parallax amount and the crosstalk amount. Further, the image unit evaluation value S of the image unit obtained by adding the pixel correspondence evaluation values s of the constituent pixels of the entire image is calculated, and the shift amount that maximizes the image unit evaluation value S is calculated as the optimal shift amount. The image processing apparatus according to 1).
(3) The optimum shift amount calculation unit calculates a pixel correspondence evaluation value s corresponding to each pixel by using correspondence data between the parallax amount and the stereoscopic effect and correspondence data between the parallax amount and the crosstalk amount. Further, an image unit evaluation value S corresponding to each shift amount obtained by multiplying the pixel corresponding evaluation value s of the constituent pixels of the entire image by a weighting coefficient for performing weighting in units of the image area and adding it is calculated. The image processing apparatus according to (1) or (2), wherein a shift amount that maximizes the unit evaluation value S is calculated as an optimal shift amount.

(4) The optimum shift amount calculation unit sets a range in which the crosstalk amount is equal to or less than a predetermined value as a comfortable parallax range, changes the shift amount within the comfortable parallax range, and supports pixels corresponding to each shift amount An evaluation value s is calculated, and an image-unit evaluation value S corresponding to the shift amount is calculated by adding the pixel-corresponding evaluation values s of the constituent pixels of the entire image. The image processing apparatus according to any one of (1) to (3), which is calculated as an optimal shift amount.
(5) The optimum shift amount calculation unit sets a range in which the crosstalk amount is equal to or less than a predetermined value as a comfortable parallax range, changes the shift amount within the comfortable parallax range, and supports pixels corresponding to each shift amount An evaluation value s is calculated, and an image unit evaluation value S corresponding to each shift amount obtained by multiplying the pixel corresponding evaluation value s of the constituent pixels of the entire image by a weighting coefficient for performing weighting in units of image areas is added. The image processing apparatus according to any one of (1) to (4), wherein the shift amount that calculates and maximizes the image unit evaluation value S is calculated as the optimum shift amount.

  (6) The image processing apparatus further includes a parallax estimation unit that generates parallax information from the left viewpoint image (L image) and the right viewpoint image (R image), and the virtual viewpoint image generation unit includes the left viewpoint image (L image) The viewpoint image (L image), the right viewpoint image (R image), the parallax information, and the optimum shift amount are input to generate a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image. The image processing apparatus according to any one of (1) to (5).

  Furthermore, the configuration of the present disclosure includes a method of processing executed in the above-described apparatus and the like, and a program for executing the processing.

  The series of processing described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet, and installed on a recording medium such as a built-in hard disk.

  Note that the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, when generating a virtual viewpoint image based on an LR image as a three-dimensional image, a parallax image with enhanced stereoscopic effect is generated by performing image shift processing. It becomes possible to do.
Specifically, an L image and an R image, which are image signals for the left eye applied to 3D image display, are input, and parallax information of the L image, the R image, and the LR image is input, and a comfortable parallax range is input. An optimal shift amount calculation unit that calculates an optimal value of the image shift amount, and a virtual viewpoint image generation unit that generates a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image. The optimum shift amount calculation unit uses the correspondence data between the parallax amount and the stereoscopic effect and the correspondence data between the parallax amount and the crosstalk amount to calculate an optimum shift amount within a predetermined range of the crosstalk amount or less. Then, the virtual viewpoint image generation unit performs an image shift process according to the optimum shift amount calculated by the optimum shift amount calculation unit, and generates a final output virtual viewpoint image.
With this process, when generating a virtual viewpoint image based on an LR image as a three-dimensional image, it is possible to generate a parallax image with enhanced stereoscopic effect by performing an image shift process.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Left viewpoint image (L image) input part 102 Right viewpoint image (R image) input part 103 Parallax information input part 104 Optimal shift amount calculation part 105 Virtual viewpoint image generation part 106 Display control part 107 Display part 115 Parallax Estimation unit 121 Block 131 Input L image 132 Input R image 133 Virtual viewpoint image 301 Display surface 311 to 313 Observation position

Claims (8)

A left viewpoint image input unit for inputting a left viewpoint image (L image) which is an image signal for the left eye applied to three-dimensional image display;
A right viewpoint image input unit for inputting a right viewpoint image (R image) that is an image signal for the right eye applied to three-dimensional image display;
The parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image) is input, and the image within the comfortable parallax range An optimum shift amount calculation unit for calculating an optimum value of the shift amount;
A virtual viewpoint image generation unit that generates a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image;
The optimal shift amount calculation unit uses the correspondence data between the parallax amount and the stereoscopic effect, and the correspondence data between the parallax amount and the crosstalk amount, so that the optimum shift amount within a predetermined range of the crosstalk amount or less is used. To calculate
The virtual viewpoint image generation unit
An image processing device that performs image shift processing according to the optimal shift amount calculated by the optimal shift amount calculation unit on a virtual viewpoint image generated based on an input LR image to generate a final output virtual viewpoint image . The optimum shift amount calculation unit
Using the correspondence data between the parallax amount and the stereoscopic effect and the correspondence data between the parallax amount and the crosstalk amount, a pixel correspondence evaluation value s corresponding to each pixel is calculated,
Furthermore, the image unit evaluation value S of the image unit obtained by adding the pixel correspondence evaluation values s of the constituent pixels of the entire image is calculated, and the shift amount that maximizes the image unit evaluation value S is calculated as the optimum shift amount. The image processing apparatus described. The optimum shift amount calculation unit
Using the correspondence data between the parallax amount and the stereoscopic effect and the correspondence data between the parallax amount and the crosstalk amount, a pixel correspondence evaluation value s corresponding to each pixel is calculated,
Further, the image unit evaluation value S corresponding to each shift amount obtained by multiplying the pixel corresponding evaluation value s of the constituent pixels of the entire image by a weighting coefficient for performing weighting in units of the image area and adding it is calculated. The image processing apparatus according to claim 1, wherein the shift amount that maximizes the value S is calculated as the optimal shift amount. The optimum shift amount calculation unit
A range where the crosstalk amount is equal to or less than a predetermined value is defined as a comfortable parallax range, the shift amount is changed within the comfortable parallax range, and a pixel corresponding evaluation value s corresponding to each shift amount is calculated,
Further, the image unit evaluation value S corresponding to the shift amount is calculated by adding the pixel correspondence evaluation values s of the constituent pixels of the entire image, and the shift amount that maximizes the image unit evaluation value S is calculated as the optimum shift amount. Item 8. The image processing apparatus according to Item 1. The optimum shift amount calculation unit
A range where the crosstalk amount is equal to or less than a predetermined value is defined as a comfortable parallax range, the shift amount is changed within the comfortable parallax range, and a pixel corresponding evaluation value s corresponding to each shift amount is calculated,
Further, the image unit evaluation value S corresponding to each shift amount obtained by multiplying the pixel corresponding evaluation value s of the constituent pixels of the entire image by a weighting coefficient for performing weighting in units of the image area and adding it is calculated. The image processing apparatus according to claim 1, wherein the shift amount that maximizes the value S is calculated as the optimal shift amount. The image processing apparatus further includes:
A parallax estimation unit that generates parallax information from the left viewpoint image (L image) and the right viewpoint image (R image);
The virtual viewpoint image generation unit
The left viewpoint image (L image), the right viewpoint image (R image), the parallax information, and the optimal shift amount are input to generate a virtual viewpoint image including viewpoint images other than the viewpoint of the input LR image. The image processing apparatus according to claim 1. An image processing method executed in an image processing apparatus,
Via the image input unit, a left viewpoint image (L image) that is an image signal for the left eye applied to a three-dimensional image display and a right viewpoint image (R image) that is an image signal for the right eye are input,
An optimal shift amount calculation unit inputs parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image). An optimal shift amount calculation process for calculating an optimal value of the image shift amount within the comfortable parallax range;
The virtual viewpoint image generation unit executes a virtual viewpoint image generation process for generating a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image,
The optimum shift amount calculation process includes:
A process of calculating an optimum shift amount within a range equal to or less than a predetermined crosstalk amount by using correspondence data between a parallax amount and a stereoscopic effect and correspondence data between a parallax amount and a crosstalk amount.
The virtual viewpoint image generation process includes:
Executed as a process for generating a final output virtual viewpoint image by performing an image shift process according to the optimal shift amount calculated by the optimal shift amount calculation unit on the virtual viewpoint image generated based on the input LR image Image processing method. A program for executing image processing in an image processing apparatus;
Processing for inputting a left viewpoint image (L image) that is an image signal for the left eye to be applied to the three-dimensional image display and a right viewpoint image (R image) that is an image signal for the right eye via the image input unit When,
An optimal shift amount calculation unit inputs parallax information of the left viewpoint image (L image), the right viewpoint image (R image), the left viewpoint image (L image), and the right viewpoint image (R image). An optimal shift amount calculation process for calculating an optimal value of the image shift amount within the comfortable parallax range;
The virtual viewpoint image generation unit executes a virtual viewpoint image generation process for generating a virtual viewpoint image including a viewpoint image other than the viewpoint of the input LR image;
In the optimum shift amount calculation process,
Using the correspondence data between the parallax amount and the stereoscopic effect and the correspondence data between the parallax amount and the crosstalk amount, a process for calculating an optimum shift amount within a predetermined range of the crosstalk amount or less is executed.
In the virtual viewpoint image generation process,
The virtual viewpoint image generated based on the input LR image is subjected to image shift processing according to the optimum shift amount calculated by the optimum shift amount calculation unit, and the final output virtual viewpoint image is generated. Program to make.