JP2013118515A - Image processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To control a parallax amount while considering a visual property in human stereoscopy.SOLUTION: A parallax amount detection section 101 detects a parallax amount of a parallax image from image data for stereoscopic image display. An observation distance acquisition section 105 acquires an observation distance between a display screen on which an image represented by the image data are reproduced, and an observer of the image. A depth estimation section 102 estimates a depth amount on display of an object represented by the image data based on the parallax amount and the observation distance. A visual property holding section 112 holds a visual property indicating a relation between the depth amount on display and the depth amount on perception. A correction calculation section 103 calculates a correction amount of the parallax amount for matching the depth amount on display of the object and the depth amount on perception based on the depth amount on display of the object, the observation distance and the visual property. A parallax amount correction section 104 corrects the parallax amount of the parallax image based on the correction amount.

Description

  The present invention relates to image processing for correcting the depth of a stereoscopic image perceived by an observer.

  In recent years, research on stereoscopic display in a display device such as a television has been actively conducted.

  In the stereoscopic display technology, a method of making a viewer perceive a stereoscopic using binocular parallax is generally used. For example, a stereoscopic image display apparatus that allows an observer to wear liquid crystal shutter glasses employs a method of displaying different images for the right eye and the left eye in order to perceive a stereoscopic image using binocular parallax.

  A display method using binocular parallax sets a parallax amount between a right-eye image (hereinafter, right image) and a left-eye image (hereinafter, left image) targeting the same object. The parallax amount is a pixel shift amount between the right image and the left image in the same object. It is known that the stereoscopic effect perceived by the observer changes depending on the shift amount and the screen size of the display screen.

  Patent Document 1 discloses a technique for controlling the stereoscopic effect by correcting the parallax according to the screen size so as not to feel uncomfortable or uncomfortable.

  In Non-Patent Document 1, in stereoscopic vision using binocular parallax, the viewer perceives the depth distance on the display of the object (hereinafter referred to as the depth on display) predicted from the parallax amount and the observation distance. It is presented that the relationship between the depth distances of objects (hereinafter referred to as perceptual depth) is non-linear.

  The conventional stereoscopic display technology does not correct the non-linearity between the display depth amount and the perceptual depth amount presented by Non-Patent Document 1. For this reason, the observer may feel a sense of miniature garden space and a sense of book splitting in the perceived stereoscopic image.

JP 2010-045584 A

Takumi Tatsumi, Yukio Nakamizo “Relationship between Convergence, Retinal Image Difference and Perceived Depth”, Journal of the Visual Society of Japan VISION, Vol. 8, Japan, Visual Society of Japan, April 1996, pages 87-95

  An object of the present invention is to control the amount of parallax in consideration of visual characteristics in human stereoscopic vision.

  The present invention has the following configuration as one means for achieving the above object.

  The image processing according to the present invention detects a parallax amount of a parallax image from image data for stereoscopic image display, and obtains an observation distance between a display surface that reproduces an image represented by the image data and an observer of the image. , Based on the parallax amount and the observation distance, estimating a display depth amount of the object represented by the image data, holding a visual characteristic indicating a relationship between a display depth amount and a perceptual depth amount, and the object Based on the display depth amount, the observation distance, and the visual characteristics, a parallax amount correction amount that combines the depth amount on display of the object and the perceptual depth amount is calculated, and the parallax image is calculated based on the correction amount. It is characterized by correcting the amount of parallax.

  According to the present invention, the depth of a stereoscopic image can be linearly reproduced perceptually by controlling the amount of parallax in consideration of the visual characteristics in human stereoscopic vision. As a result, it is possible to suppress the feeling of miniature garden and the sense of book splitting that the observer remembers.

1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment. The figure explaining the relationship between binocular parallax amount D1 and the distance of a target object. The figure explaining perceptual depth distance Lv. The figure explaining the relationship between depth amount Ld on display and depth amount Lv on perception. The flowchart explaining the process of a correction | amendment calculating part. The figure which shows the table which shows correspondence of binocular parallax amount D1, the depth amount Ld on display, and the perceived depth amount Lv for every observation distance Ls. The figure explaining the correction method of binocular parallax amount D. The figure explaining the calibration of visual characteristic information Dv. FIG. 3 is a block diagram illustrating a configuration example of an image processing apparatus according to a second embodiment. The figure explaining the depth amount Ld on a display, the perceived depth amount Lv, and the fusion limit. The flowchart explaining the process of a correction | amendment calculating part. The flowchart explaining the calculation process of correction | amendment parallax amount. Diagram illustrating depth amount Ld _A on the display, the correction depth amount Lv _'A perceptual, the relationship between the fusion limit.

  Hereinafter, image processing according to an embodiment of the present invention will be described in detail with reference to the drawings. In the following, an image processing apparatus that can be applied to a system that provides perception of a stereoscopic image to an observer wearing liquid crystal shutter glasses will be described. The image processing apparatus has a function of correcting the depth of a stereoscopic image perceived by an observer.

[Device configuration]
The block diagram of FIG. 1 shows a configuration example of the image processing apparatus 100 of the embodiment.

  The image processing apparatus 100 receives the video signal Din and outputs the video signal Dout to the image display unit 111. Note that when the depth amount correction described later is not performed, the video signal passes through the image processing apparatus 100. The image display unit 111 is a liquid crystal display (LCD), a plasma display, an organic EL display, a projector, or the like, and provides a viewer with a stereoscopic image perception based on a predetermined stereoscopic image display method.

  The CPU 106 executes a control program stored in the ROM 107 or the like using the RAM 108 as a work memory, and controls a configuration to be described later via the system bus 113. In addition, the CPU 106 acquires a user instruction and various information input by the user via the operation unit 109 such as a button, a keyboard, or a touch panel.

  The observation distance acquisition unit 105 acquires the distance (observation distance) Ls between the observer (usually a user) and the display surface of the image display unit 111 stored in the ROM 107 and stores it in the RAM 108. This observation distance Ls is a standard observation distance defined by the ITU-R (International Radio Communications Advisory Committee), that is, a distance that is three to six times the screen height of the image display unit 111. The observation distance Ls may be acquired by the user operating the operation unit 109 to input the observation distance Ls, or by the CPU 106 instructing the distance sensor 110 to perform measurement. The CPU 106 stores the input or acquired observation distance Ls in a predetermined area of the RAM 108, and supplies the observation distance Ls to the observation distance acquisition unit 105 via the RAM 108.

  The distance sensor 110 irradiates an infrared laser in a direction facing the display surface of the image display unit 111, and measures the observation distance Ls from the time until the reception sensor receives the reflected light. The distance sensor 110 transmits the observation distance Ls as a measurement result to the observation distance acquisition unit 105 or the CPU 106, and the observation distance acquisition unit 105 or CPU 106 stores the received observation distance Ls in a predetermined area of the RAM 108. Note that the method of measuring the observation distance is not limited to this. For example, the observation distance may be measured by triangulation using a multi-view camera.

  The parallax amount detection unit 101 detects the binocular parallax amount D of the binocular parallax image from the image data Din when image data Din for stereoscopic image display composed of a left image and a right image is input. The depth estimation unit 102 inputs the binocular parallax amount D, the image data Din, the display device information Dd, and the observation distance Ls, and estimates the display depth amount Ld expressed by popping out or pulling in a stereoscopic image.

  The depth Ld on the display is the depth distance of the target object calculated from the binocular parallax amount, the observation distance, and the observer's interpupillary distance in a certain object. The target object is the same object that exists in both the left image and the right image.

The display device information Dd is information related to the specifications of the image display unit 111 stored in the ROM 107 or the like, and the size (width W [mm] × height H [mm]) of the displayable area of the image display unit 111 and the display _Contains information on the number of possible pixels (width N _DW × length N _DH ). Further, as the display device information Dd, for example, extended display identification data (EDID) may be acquired from the image display unit 111. The EDID is defined by VESA (Video Electronics Standards Association). In addition to the above information, the display device information Dd desirably includes information for identifying a display method such as a liquid crystal display or a projector.

  The visual characteristic holding unit 112 is a memory that holds visual characteristic information Dv. The visual characteristic information Dv represents the relationship between the displayed depth amount Ld and the perceptual depth amount Lv. For example, a table indicating the relationship between the displayed depth amount Ld and the perceptual depth amount Lv may be used, or an expression for calculating the perceptual depth amount Lv from the displayed depth amount Ld may be used.

  The correction calculation unit 103 uses the visual characteristic information Dv, the display device information Dd, and the observation distance Ls to correct the perceptual depth Lv and the corrected parallax amount C that combines the display depth Ld estimated by the depth estimation unit 102. Is calculated. The parallax amount correcting unit 104 corrects the binocular parallax amount D of the image data Din based on the corrected parallax amount C, and outputs the corrected image data Dout to the image display unit 111.

[Device operation]
The image processing apparatus 100 receives image data Din from an external stereoscopic image reproduction apparatus, a digital broadcast receiver, or the like.

Parallax amount detection unit The parallax amount detection unit 101 extracts an object from each of the left image and the right image of the image data Din by a known method, and determines the amount of pixel shift on the display surface of the target object as the binocular disparity amount D. Detect as. Then, the binocular parallax amount D is transmitted to the depth estimation unit 102 for each target object.

Depth estimation unit Depth estimation unit 102 inputs binocular parallax amount D, display device information Dd, and observation distance Ls of each target object, and calculates depth Ld on the display of each target object using equation (1) To do. For example, using the display width and the number of horizontal pixels of the image display unit 111 indicated by the display device information Dd and the number of horizontal pixels of the image data Din, the unit of the binocular parallax amount D is converted from the number of pixels to millimeters. The amount of parallax D1 is calculated.
D1 = D × W / N _DW × N _DW / N _inW (1)
Here, W is the display width [mm] of the image display unit 111,
N _DW is the number of horizontal pixels of the image display unit 111,
N _inW is the number of horizontal pixels of the image data Din.

Expression (1) is an expression for calculating the binocular parallax amount D1 when the left image and the right image having the number of pixels (horizontal) N _inW × (vertical) N _inH are displayed in full screen. When displaying the image at the same magnification, N _DW / N _inW = 1 in Equation (1).

Next, the depth estimation unit 102 calculates a display depth amount Ld with respect to the binocular parallax amount D1. The relationship between the binocular parallax amount D1 and the distance between the target objects will be described with reference to FIG. If the distance between the pupils of the observer is de [mm], the depth amount Ld on the display is calculated by the following equation. The depth estimation unit 102 transmits the display depth amount Ld obtained for each target object P to the correction calculation unit 103.
In Fig. 2 (a), de: D1 = Ld: Ld-Ls
∴ Ld = de × Ls / (de-D1)… (2)

  However, D1 is negative when the target object P is in front of the image display unit 111 as shown in FIG. 2 (a), and the target object P is from the image display unit 111 as shown in FIG. 2 (b). If it is behind, it is positive. This can be said to be positive when the object extracted from the left image is to the left of the same object extracted from the right image, and negative when it is to the right. Further, as can be seen from Expression (2), when the depth Ld on the display of the target object P is equal to the observation distance Ls (Ld = Ls), the binocular parallax amount D1 is zero. Note that the inter-pupil distance de is 65 mm in the case of a general adult, but is not limited to this, and the user (observer) may input the operation unit 109.

Observation distance acquisition unit, visual characteristic holding unit When the observation distance acquisition unit 105 requests the observation distance Ls from the depth estimation unit 102 or the correction calculation unit 103, the observation distance Ls stored in the RAM 108 is converted into the depth estimation unit 102 or It transmits to the correction | amendment calculating part 103. Further, when the visual characteristic information Dv is requested from the correction calculation unit 103, the visual characteristic holding unit 112 transmits the held visual characteristic information Dv to the correction calculation unit 103.

Correction Calculation Unit The correction calculation unit 103 uses the visual characteristic information Dv input from the visual characteristic holding unit 112 to obtain a perceptual depth Lv from the displayed depth Ld input from the depth estimation unit 102. Then, a correction amount C of the parallax amount with respect to the target object P in the image data Din is calculated.

The perceptual depth distance Lv will be described with reference to FIG. As shown in FIGS. 3 (a) and 3 (b), the perceived depth distance Lv is affected by visual characteristics, and a difference from the displayed depth distance Ld is perceived. The relationship between the displayed depth amount Ld and the perceptual depth amount Lv will be described with reference to FIG. The relationship between the perceived depth Lv and the displayed depth Ld is as follows.
Lv = f (Ld) (3)
Here, f is a function representing visual characteristics.

For example, when the depth amount on the display of a certain target object P is Ld1, a depth amount Lv1 different from the depth amount on the display is perceived. In other words, the depth amount Ld1 ′ on the display necessary for setting the perceptual depth amount to Ld1 is expressed by the following equation. That is, Ld1 ′ is a depth amount obtained by correcting the difference between the displayed depth amount and the perceptual depth amount.
Ld1 '= f ^-1 (Ld1)… (4)

  The processing of the correction calculation unit 103 will be described with reference to the flowchart of FIG. The correction calculation unit 103 acquires the observation distance Ls (S101), and acquires visual characteristic information Dv corresponding to the observation distance Ls from the visual characteristic holding unit 112 (S102).

When the visual characteristic information Dv is the function f of the expression (3) representing the relationship between the depth Ld on the display and the perceptual depth Lv, the function f is prepared in advance for each observation distance Ls, and the visual characteristic holding unit 112. That is, the correction calculation unit 103 acquires the function f corresponding to the observation distance closest to the observation distance Ls from the visual characteristic holding unit 112, and acquires, for example, the function f represented by the following equation.
Lv = f (Ld) = (Ld-Ls) ^k + Ls… (5)
Where k is a coefficient based on the observation distance Ls,
For example, observation distance Ls = 900 mm, coefficient k = 0.95.

  The visual characteristic information Dv is not limited to the function f. FIG. 6 shows a table indicating correspondence between the binocular parallax amount D1, the display depth amount Ld, and the perceptual depth amount Lv for each observation distance Ls. That is, the correction calculation unit 103 may acquire a table corresponding to the observation distance Ls. If a table corresponding to the observation distance Ls is not prepared, the table closest to the observation distance Ls may be selected, or the table corresponding to the observation distance Ls may be interpolated between the tables sandwiching the observation distance Ls. A table may be generated.

Next, the correction calculation unit 103 receives the depth Ld on the display of the target object P of interest (S103). Then, based on the visual characteristic information Dv (function f or table), a corrected depth Ld ′ that faithfully reproduces the target object P whose depth on display is Ld as perceptual depth Lv = Ld is calculated (S104). ). For example, when the depth Ld1 on the display of the target object P is 2000 mm, the corrected depth Ld1 ′ is calculated so that the perceptual depth Lv is 2000 mm.
From equations (4) and (5),
Ld1 '= f ^-1 (Ld1)
= (Ld1-Ls) ^{1 / k} + Ls
If Ld1 = 2000, Ls = 900, k = 0.95,
Ld1 '= (2000-900) ^{1 / 0.95} + 900 = 2490… (6)

Next, the correction calculation unit 103 acquires the display device information Dd from the ROM 107 or the like (S105), and calculates the corrected parallax amount C based on the display device information Dd and the corrected depth Ld1 ′ (S106). The unit of the corrected depth amount C is a pixel. Therefore, the corrected binocular parallax amount D2 [mm] necessary for the corrected depth amount Ld1 ′ is calculated by the following equation.
From equation (2)
D2 = de (Ld1 '-Ls) / Ld1'… (7)

Then, the corrected binocular parallax amount D2 is unit-converted using the display width W of the image display unit 111 indicated by the display device information Dd, the horizontal pixel number N _DW , and the horizontal pixel number N _inW of the image data Din, and the corrected parallax amount C To.
C = D2 × N _DW / W × N _inW / N _DW (8)

Expression (8) is an expression for calculating the corrected parallax amount C when the left image and the right image having the number of pixels (horizontal) N _inW × (vertical) N _inH are displayed in full screen. When displaying the image at the same magnification, N _inW / N _DW = 1 in Expression (8).

  Next, the correction calculation unit 103 determines whether or not the corrected parallax amount C has been calculated for all target objects included in the image data Din (S107), and there is a target object P for which the corrected parallax amount C has not been calculated. Then, the process returns to step S103. When the corrected parallax amount C for all target objects is calculated, the corrected parallax amount C is transmitted to the parallax amount correcting unit 104 (S108).

Parallax amount correction unit The parallax amount correction unit 104 outputs image data Dout obtained by correcting the binocular parallax amount D of the target object P based on the corrected parallax amount C corresponding to the target object P included in the image data Din. .

  A method for correcting the binocular parallax amount D will be described with reference to FIG. In the case of CD <0 shown in FIG. 7A, the object corresponding to the target object P in the left image is moved to the right by (C−D) / 2 pixels. Further, the object corresponding to the target object P in the right image is moved to the left by (C−D) / 2 pixels. On the other hand, in the case of CD ≧ 0 shown in FIG. 7B, the object corresponding to the target object P in the left image is moved to the left by (C−D) / 2 pixels. Further, the object corresponding to the target object P in the right image is moved to the right by (C−D) / 2 pixels.

  The parallax amount correction unit 104 performs the process shown in FIG. 7 on the objects corresponding to all target objects included in the image data Din, and outputs the corrected image data Dout.

  The corrected image data Dout is input to the image display unit 111 and displayed in accordance with a stereoscopic image display method (frame sequential method, line-by-line, etc.). Therefore, the non-linear relationship between the display depth amount and the perceptual depth amount in the stereoscopic image is corrected, and the depth of the stereoscopic image perceived by the observer is appropriately reproduced, so that the stereoscopic image box perceived by the observer can be reproduced. A feeling of feeling and book splitting can be prevented.

  The visual characteristic information Dv indicating the relationship between the depth on display and the depth on perception varies depending on the screen size of the display device. Therefore, it is preferable to hold the visual characteristic information Dv for each screen size. In the above description, the example in which the visual characteristic holding unit 112 holds the visual characteristic information Dv has been described. However, calibration is performed by allowing the user to observe a stereoscopic image, and the calibration result data is used as the visual characteristic information Dv. You can also.

  The calibration of the visual characteristic information Dv will be described with reference to FIG. When the user operates the operation unit 109 to instruct activation of the user setting mode for stereoscopic effect, the CPU 106 causes the image display unit 111 to display a bar image as shown in FIG. 8 held by the visual characteristic holding unit 112, for example. . The bar image has binocular parallax, and the user recognizes it as a stereoscopic image. The user inputs the perceived depth of the stereoscopic image by operating the operation unit 109. Thereby, the perceptual depth amount corresponding to the binocular parallax amount is obtained. Therefore, visual characteristic information Dv can be obtained by calculating the display depth with respect to the binocular parallax using equations (1) and (2). Note that the stereoscopic image for obtaining the visual characteristic information Dv is not limited to the bar image shown in FIG. 8, and a stereoscopic video standard chart defined by the Video Information Media Society may be used.

  The image processing according to the second embodiment of the present invention will be described below. Note that the same reference numerals in the second embodiment denote the same parts as in the first embodiment, and a detailed description thereof will be omitted.

  If the depth Ld on the display is to be faithfully reproduced perceptually as in the first embodiment, stereoscopic vision may be impossible beyond the fusion limit of vision. Thus, in the second embodiment, the depth of the display is linearly reproduced by compressing the depth on the display perceptually using the fusion limit information.

[Device configuration]
A block diagram of FIG. 9 shows a configuration example of the image processing apparatus 100 of the second embodiment.

  The image processing apparatus 100 according to the second embodiment includes a fusion limit acquisition unit 801 and an image signal output unit 802 in addition to the configuration illustrated in FIG. In addition, the image display unit 111 includes corresponding liquid crystal shutter glasses 803.

As will be described in detail later, the fusion limit acquisition unit 801 acquires the fusion limit information L _Df from the ROM 107, the RAM 108, and the like. The fusion limit is a limit of binocular parallax that allows human vision to fuse parallax in a binocular parallax image and perceive it as a stereoscopic image. Hereinafter, the unit of fusion limit will be described as millimeters.

The correction calculation unit 103 uses the fusion limit information L _Df in addition to the visual characteristic information Dv, the display device information Dd, and the observation distance Ls to display the perceptual depth Lv and the display estimated by the depth estimation unit 102. A corrected parallax amount C for equalizing the depth amount Ld is calculated.

  The image signal output unit 802 outputs the image data Dout with the binocular parallax amount corrected and the image display control signal Sf. The liquid crystal shutter glasses 803 operate based on the vertical synchronization signal Sv included in the image display control signal Sf, and perform a shutter operation for causing the observer to perceive a stereoscopic image.

[Device operation]
The operations of the parallax amount detection unit 101, the depth estimation unit 102, the observation distance acquisition unit 105, and the visual characteristic holding unit 112 are the same as those in the first embodiment, and detailed description thereof is omitted.

● fusional limit acquisition unit fusional limit acquisition unit 801 is requested the fusion limit information L _Df from the correction calculation unit 103, for example, acquires the fusion limit information L _Df from ROM 107, the fusion limit information L _Df Is transmitted to the correction calculation unit 103.

  As shown in FIG. 2 (b), when the target object P is perceived at the back of the image display unit 111, the limit of the binocular parallax that can be perceived as a stereoscopic image is set as the maximum fusion limit Df1. . In addition, as shown in FIG. 2 (a), when the target object P is perceived in front of the image display unit 111, the limit of the binocular parallax amount that can be perceived as a stereoscopic image is the minimum value Df2 of the fusion limit. And Note that the parallax amount at the fusion limit is positive when a stereoscopic image is perceived in the back of the image display unit 111 and negative when perceived in the front, like the binocular parallax amount D1.

  The fusion limit information Df may be a comfortable parallax range recommended in the safety guidelines of the 3D consortium as long as the information regarding the maximum value Df1 and the minimum value Df2 is included. Alternatively, in the user setting mode, a stereoscopic image as shown in FIG. 8 may be presented, and the user may increase or decrease the binocular parallax amount using the operation unit 109 to acquire the user's fusion limit.

The user observes a stereoscopic image as shown in FIG. 8 and decreases the binocular parallax amount by operating the operation unit 109. When the stereoscopic image cannot be perceived, the reduction of the binocular parallax amount is stopped. That is, the binocular parallax amount at the time when the operation is stopped is set to the minimum value L _Df 2 of the fusion limit. Similarly, the user observes a stereoscopic image as shown in FIG. 8, and increases the binocular parallax amount by operating the operation unit 109. When the stereoscopic image cannot be perceived, the increase in the binocular parallax amount is stopped. That is, the binocular parallax amount at the time when the operation is stopped is set as the maximum fusion limit value L _Df 1. The CPU 106 stores the acquired maximum value L _Df 1 and minimum value L _Df 2 as fusion limit information L _Df in, for example, a predetermined area of the RAM 108, and the fusion limit information L _Df via the RAM 108. To supply.

Correction Operation Unit The relationship between the displayed depth Ld, perceptual depth Lv, and fusion limit will be described with reference to FIG. In FIG. 10, the hatched area is an area exceeding the fusion limit outside the range of the maximum value L _Df 1 and the minimum value L _Df 2, and in this area, a fusion image cannot be observed and a stereoscopic image can be observed. Can not.

For example, as shown in FIG. 10 (a), when the depth on the display for a certain target object A, B, C is Ld _A , Ld _B , Ld _C , the perceived depth Lv _A of each target object, Lv _B and Lv _C are as shown below according to equation (3).
Lv _A = f (Ld _A )
Lv _B = f (Ld _B )… (9)
Lv _C = f (Ld _C )

There is a difference between the displayed depth and the perceived depth, and as shown in Fig. 10 (a), the displayed depth of the target objects A, B, and C corresponds to the perceived depth. do not do. Further, the relative distance relationship between the target objects A, B, and C with the position of the image display unit 111 as the origin also shifts as shown in the following equation.
Ld _A -Ls: Ld _B -Ls: Ld _C -Ls ≠ Lv _A -Ls: Lv _B -Ls: Lv _C -Ls (10)

Further, the depth _C Ld _C on the display of the target object C exceeds the fusion limit and is not perceived as a stereoscopic image. Therefore, as in the first embodiment, even when the displayed depth amount Ld _C and the perceived depth amount Lv _C are matched, the target object C is not perceived as a stereoscopic image.

Therefore, in the second embodiment, scaling and shifting are performed so that all target objects are within the fusion limit and the relative distance relationship of the target objects is perceptually reproduced. For this purpose, a correction amount for correcting the difference between the display depth amount and the perceptual depth amount is calculated. First, a perceptual depth Lv ′ after correction of a certain target object is calculated from the following equation.
Lv '= f _S (Ld) (11)

When Expression (11) is used, perceptual depths Lv ′ _A , Lv ′ _B , and Lv ′ _C after correction of the target objects A, B, and C are calculated as follows.
Lv ' _A = f _S (Ld _A )
Lv ' _B = f _S (Ld _B )… (12)
Lv ' _C = f _S (Ld _C )

Then, the corrected corrected depth amount Ld ′ necessary to obtain the corrected perceptual depth amounts Lv ′ _A , Lv ′ _B , and Lv ′ _C is expressed by the following equation using Equation (5).
Lv ' _A = f (Ld' _A )
Lv ' _B = f (Ld' _B )
Lv ' _C = f (Ld' _C )
Therefore,
Ld ' _A = f ^-1 (Lv' _A )
Ld ' _B = f ^-1 (Lv' _B )… (13)
Lc ' _C = f ^-1 (Lv' _C )

When Ld ' _A , Ld' _B , Ld ' _C obtained by Equation (13) reproduces the relative distance on the display with the perceptual depth of the target objects A, B, C This is the depth on the display.

The processing of the correction calculation unit 103 will be described with reference to the flowchart of FIG. The correction calculation unit 103 acquires the observation distance Ls and the visual characteristic information Dv (S301, S302), and acquires the fusion limit information L _Df from the fusion limit acquisition unit 801 (S901).

Next, the correction calculation unit 103 calculates the maximum value L _Df 1 and the minimum value L _Df 2 of the depth amount on the display using Expression (2) based on the fusion limit information L _Df and the visual characteristic information Dv. To do. Moreover, to calculate the depth amount Lv1 perceptual corresponding to the maximum value L _Df 1, the minimum depth amount Lv2 perceptual corresponding to L _Df 2 (S902). That is, Lv1 = f (L _Df 1) and Lv2 = f (L _Df 2) are calculated.

Next, the correction calculation unit 103 receives the depth amount Ld on the display of all target objects from the depth estimation unit 102 (S903), the maximum value of the received depth amount Ld on the display is Ldmax, and the minimum value is Ldmin Set (S904). In the example of FIG. 10 (a), Ld _C is set to the maximum value Ldmax, and Ld _A is set to the minimum value Ldmin.

Next, the correction operation unit 103 acquires the display information Dd (S105), performs the maximum value Ldmax depth amount on display, comparison of the maximum value L _Df 1 of fusion limit on Show (S905) . Furthermore, performing the minimum value Ldmin depth amount on the display, a comparison of the minimum value L _Df 2 of fusion limit on Show (S906, S907). Based on the comparison result, the correction is made so that the depth on the display of each target object is within the fusion limit, and the depth on the display is perceptually compressed to reproduce the depth of the stereoscopic image linearly. The parallax amount calculation process is performed as follows (S908-S911).
if (Ldmax> L _Df 1 && Ldmin <L _Df 2)
Perform calculation process 1 (S908);
if (Ldmax> L _Df 1 && Ldmin ≧ L _Df 2)
Perform calculation process 2 (S909);
if (Ldmax ≦ L _Df 1 && Ldmin <L _Df 2)
Perform calculation process 3 (S910);
if (Ldmax ≦ L _Df 1 && Ldmin ≧ L _Df 2)
Perform calculation process 4 (S911);

  Then, the correction calculation unit 103 transmits the corrected parallax amount C of each target object obtained by the calculation process of the corrected parallax amount to the parallax amount correcting unit 104 (S108).

[Correction amount calculation process 1]
The correction parallax amount calculation process will be described with reference to the flowchart of FIG.

The perceptual depth amount maximum value Lv1 should be a perceptual depth amount corresponding to the maximum depth value Ldmax on the display. However, when Ldmax> L _Df 1 and Ldmin <L _Df 2, since Ldmax exceeds the fusion limit L _Df 1, Lv1 does not fall within the fusion limit. Therefore, Lv1 is shifted to the perceptual depth corresponding to the maximum fusion limit L _Df1 . Similarly, Lv2 also shifts from the perceptual depth corresponding to Ldmin to the minimum fusion limit L _Df 2. That is, the correction calculation unit 103 calculates Lv1 = f (L _Df 1) and Lv2 = f (L _Df 2) according to the equation (3) (S1201).

Next, the correction calculation unit 103 compares the depth amount Ld on the display with the observation distance Ls (S1202). For example, if the depth amount on the display of the target object A is Ld _A , Ld _A and the observation distance Ls are compared.

  In the case of Ld> Ls, the correction calculation unit 103 uses the equation (11) to perceptually reproduce the depth of the stereoscopic image linearly by compressing the relative relationship between the depth amounts on the display of Ld, Ls, and Ldmax. The corrected depth amount Lv ′ is calculated (S1203).

For example, as shown in FIG. 10 (a), the depth on display of the target object A is Ld _A, and Lv ′ corresponding to Ld _A is Lv ′ _A. The relationship between the depth Ld _A on the display, the corrected depth Lv ′ _A on the perception, and the fusion limit will be described with reference to FIG. In FIG. 13, there is a lower relationship between Lv ′ _A , observation distance Ls, maximum depth Ldmax on display, maximum perceptual depth Lv1, and depth Ld _A on target object A. It holds.
Ld _A -Ls: Ldmax-Ls = Lv ' _A -Ls: Lv1-Ls (14)

From Equation (14), Equation (11) for scaling the perceptual depth is as follows.
Lv ' _A = (Ld _A -Ls) (Lv1-Ls) / (Ldmax-Ls) + Ls… (15)

As shown in equation (15), the correction depth amount Lv _'A on perception of the object A is calculated from the depth amount Ld _A on the display. In other words, according to Equation (15), the depth on the display of each target object is kept within the fusion limit, and the depth on the display is perceptually compressed to reproduce the depth of the stereoscopic image linearly. The upper correction depth amount Lv ′ is calculated.

  On the other hand, when Ld ≦ Ls, the depth Ld on the display of the target object is in front of the observation distance Ls. Therefore, the correction calculation unit 103 calculates a perceptual correction depth amount Lv ′ corrected so as to perceptually reproduce the relative relationship of the depth amounts on the display of Ld, Ls, and Ldmin according to the equation (11) ( S1204).

For example, the depth amount on the display of the target object B is Ld _B, and Lv ′ corresponding to Ld _B is Lv ′ _B. Lv _'B, the viewing distance Ls, the minimum value Ldmin depth amount on the display, the minimum value of the depth of perceptual Lv2, the relationship below between depth amounts Ld _B on the display of the target object B holds.
Ls-Ld _B : Ls-Ldmin = Ls-Lv ' _B : Ls-Lv2 (16)

From equation (16), equation (11) for scaling the perceptual depth is as follows.
Lv ' _B = Ls-(Ls-Ld _B ) (Ls-Lv2) / (Ls-Ldmin)… (17)

As shown in Expression (17), the perceptual corrected depth Lv ′ _B of the target object _B is calculated from the displayed depth Ld _B. In other words, according to Equation (17), the depth of display on each target object is kept within the fusion limit, and the depth of display is perceptually compressed to reproduce the depth of the stereoscopic image linearly. The upper correction depth amount Lv ′ is calculated.

Next, the correction calculation unit 103 calculates a corrected depth amount Ld ′ from the corrected depth amount Lv ′ (S1205). For example, the display depth amount corresponding to the perceptual correction depth amount Lv ′ _A is Ld ′ _A as shown in FIG. The corrected depth Ld ′ _A on the display of the target object A is calculated from Ld ′ _A = f ⁻¹ (Lv ′ _A ) according to the equation (4).

  Next, the correction calculation unit 103 calculates the corrected parallax amount C (S106), determines whether or not the corrected parallax amount C has been calculated for all target objects (S107), and the target for which the corrected parallax amount C has not yet been calculated. If there is an object P, the process returns to step S1202.

[Correction amount calculation process 2]
When Ldmax> L _Df 1 and Ldmin ≧ L _Df 2, the maximum perceived depth Lv1 is calculated from the perceptual depth corresponding to the maximum depth Ldmax on the display. Shift to perceptual depth corresponding to L _Df 1. That is, Lv1 = f ( _Df1 ) is calculated from the equation (3) (S1201).

In addition, since the minimum depth value Ldmin on the display is greater than or equal to the minimum fusion limit L _Df 2, the minimum perceptual depth value Lv2 is the perceptual value corresponding to the minimum Ldmin value on the display. To the depth amount. That is, Lv2 = f (Ldmin) is calculated from the equation (3) (S1201).

  The subsequent processing is the same as the calculation processing 1.

[Correction amount calculation process 3]
When Ldmax ≤ L _Df 1 and Ldmin <L _Df 2, the maximum depth value Ldmax on the display is less than the fusion limit maximum value L _Df 1, so the maximum perceptual depth Lv1 is displayed A perceptual depth amount corresponding to the maximum depth value Ldmax is set. That is, Lv1 = f (Ldmax) is calculated by the equation (3) (S1201).

The perceptual depth Lv2 is changed from the perceptual depth corresponding to the minimum depth Ldmin on the display to the perceptual depth L _Df 2 corresponding to the minimum fusion limit L _Df 2. shift. That is, Lv2 = f ( _Df2 ) is calculated by the equation (3) (S1201).

  The subsequent processing is the same as the calculation processing 1.

[Correction amount calculation process 4]
When Ldmax ≦ L _Df 1 and Ldmin ≧ L _Df 2, the maximum depth value Ldmax on the display and the minimum depth value Ldmin on the display are within the fusion limit. Therefore, the maximum perceived depth Lv1 is the perceptual depth corresponding to the maximum displayed depth Ldmax, and the minimum perceptual depth Lv2 is the minimum displayed depth Ldmin. The perceptual depth corresponding to. That is, Lv1 = f (Ldmax) and Lv2 = f (Ldmin) are calculated from the equation (3) (S1201).

  The subsequent processing is the same as the calculation processing 1.

Parallax amount correction unit, image signal output unitThe parallax amount correction unit 104 corrects the binocular parallax amount D of the target object P based on the corrected parallax amount C corresponding to the target object P included in the image data Din, Outputs image data Dout. In addition, an image display control signal Sf including a horizontal synchronization signal, a vertical synchronization signal, and the like is output to the image signal output unit 802.

  For example, the image display unit 111 displays left and right parallax images alternately with a left image, a right image, a left image, a right image,... In time series based on image data Dout in a video format called field sequential. The image signal output unit 802 controls the liquid crystal shutter of the liquid crystal shutter glasses 803 based on the image display control signal Sf. That is, the state of the liquid crystal shutter is alternately switched between left eye transmission / right eye shielding, left eye shielding / right eye transmission, left eye transmission / right eye shielding, left eye shielding / right eye transmission,. The state of the liquid crystal shutter is switched in synchronization with the vertical synchronization signal included in the image display control signal Sf. Only the left image is observed for the left eye of the observer, only the right image is observed for the right eye, and the stereoscopic image is observed. Is done. Note that a stereoscopic image may be observed with a line-by-line method and circularly polarized glasses instead of the frame sequential method.

  In this way, the depth of the displayed image is perceptually compressed and the depth of the stereoscopic image is linearly reproduced, thereby preventing the collapse of the stereoscopic image due to the depth exceeding the fusion limit and providing appropriate stereoscopic image observation. can do.

  In the above description, the perceptual depth is relatively reproduced with the image display unit 111 as the origin, but correction may be performed to reproduce the relative relationship with the observer's observation position as the origin. In this case, if the observation distance Ls in the equations (8), (9), (10), and (11) used by the correction calculation unit 103 is zero, the correction is performed with the observation position as the origin.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

Detecting means for detecting a parallax amount of the parallax image from the image data for stereoscopic image display;
Obtaining means for obtaining an observation distance between a display surface that reproduces an image represented by the image data and an observer of the image;
Estimating means for estimating a depth amount on display of an object represented by the image data based on the parallax amount and the observation distance;
Holding means for holding a visual characteristic indicating a relationship between a display depth amount and a perceptual depth amount;
An arithmetic means for calculating a correction amount of a parallax amount that combines a depth amount on the display of the object and a perceptual depth amount based on the depth amount on the display of the object, the observation distance, and the visual characteristics;
An image processing apparatus comprising: correction means for correcting a parallax amount of the parallax image based on the correction amount.   2. The image processing apparatus according to claim 1, wherein the visual characteristic indicates the perceptual depth amount corresponding to the depth amount on the display for each observation distance. Furthermore, it has a means for acquiring information indicating the fusion limit,
3. The calculation unit according to claim 1, wherein the calculation unit calculates the correction amount based on a depth amount on display of the object, the observation distance, the visual characteristic, and the fusion limit. Image processing device.   4. The image according to claim 3, wherein the calculation means calculates the correction amount for compressing the perceptual depth amount and linearly reproducing the depth of the stereoscopic image based on the fusion limit. Processing equipment.   5. The image processing apparatus according to claim 3, further comprising means for acquiring the fusion limit of the observer.   6. The image processing apparatus according to claim 1, further comprising means for acquiring the visual characteristic of the observer. Detecting the amount of parallax of the parallax image from the image data for stereoscopic image display;
Obtaining an observation distance between a display surface that reproduces an image represented by the image data and an observer of the image;
Based on the parallax amount and the observation distance, the depth amount on the display of the object represented by the image data is estimated,
It retains visual characteristics that show the relationship between the depth on the display and the depth on the perception,
Based on the depth on the display of the object, the observation distance, and the visual characteristics, a correction amount of the parallax amount that combines the depth on the display of the object and the perceptual depth is calculated,
An image processing method comprising correcting a parallax amount of the parallax image based on the correction amount.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6.

Images