JP5528162B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing device, a stereoscopic image display device, and an image processing method for processing a left-eye image and a right-eye image that form a pair of stereoscopic images.

  In recent years, many stereoscopic image display techniques using binocular parallax have been proposed as stereoscopic image display techniques for an observer to obtain a pseudo-depth feeling. This technology is a technology for inducing a sense of depth in a pseudo manner by displaying an image viewed in the three-dimensional space with the left eye and an image viewed with the right eye to the left eye and the right eye of the observer, respectively.

  Various techniques have been proposed as techniques for showing images corresponding to the left and right eyes of the observer. For example, the left-eye image and the right-eye image are displayed alternately on the left eye image and the right eye image at the same time, and at the same time, the left and right fields of view are temporally switched using glasses that close the left and right fields of view in synchronization with the image switching timing. A method of separating them is proposed. In addition, a method has been proposed in which a left eye image and a right eye image are displayed separately for each of the left and right eyes by using a barrier or a lens that limits the display angle of the image displayed on the display in front of the display.

  In such a stereoscopic image display apparatus, the observer focuses on the display surface while adjusting the convergence angle of the eye to the point at which the object being watched has popped out, and thus unnaturally sees the object. For this reason, there is a problem that eye fatigue is induced when the pop-out amount is too large. Moreover, the depth feeling suitable for the observer varies depending on the distance between the observer and the display surface of the display and the individual differences of the observer.

  In order to solve such a problem, Patent Document 1 below discloses a technique for protecting an observer's eyes by changing the parallax of a stereoscopic image when the stereoscopic image display time exceeds a predetermined time.

  Further, in order to deal with the problem that the depth varies depending on the distance between the observer and the display surface of the display, in Patent Document 2 below, the position of the observer's head is detected, and the light shielding portion of the light shielding unit is detected according to the detected head position. There is disclosed a technique for performing optimal stereoscopic viewing for an observer by performing movement control.

JP 2008-306739 A JP 2002-305759 A

  However, in the technique described in Patent Document 1, parallax is reduced by changing the horizontal relative position of the entire image data to protect the eyes of the observer. For this reason, there has been a problem that it is impossible to change to increase or decrease the stereoscopic effect of the entire screen. Also, if the input image contains both pixels that are displayed in front of the display surface and pixels that are displayed in the back of the display surface, the absolute value of the parallax of the entire image data cannot be reduced, There was a problem that it could not be properly protected.

  In the technique described in Patent Document 2, the position of the observer's head is detected, and movement control of the light shielding unit is performed according to the detected head position. Therefore, there is a problem that the control of the optical system with high accuracy according to the position of the observer's head is required, and the cost of the apparatus increases.

  The present invention has been made in view of the above, and protects the eyes of an observer, suppresses an increase in cost, and is suitable for a depth that does not depend on the distance between the observer and the display surface or on individual differences between observers. It is an object of the present invention to obtain an image processing apparatus, a stereoscopic image display apparatus, and an image processing method that can generate a stereoscopic image that gives a feeling of feeling.

In order to solve the above-described problems and achieve the object, the present invention divides a left-eye image and a right-eye image that are paired for visually recognizing a stereoscopic image into a plurality of blocks for each frame. Similarity detecting means for detecting the degree of similarity between the left eye image and the right eye image for each horizontal shift amount between the left eye image and the right eye image for each block; Based on the degree of similarity, a parallax calculation means for obtaining the parallax for each block as an intra-block parallax, and an intra-frame parallax that is a parallax within the frame based on the intra-block parallax corresponding to the block within the same frame Intra-frame parallax calculation means for calculating, parallax adjustment amount calculation means for calculating a parallax adjustment amount that is a parallax adjustment amount based on the intra-frame parallax, the parallax adjustment amount, the left-eye image, and the right eye Image generating means for generating an adjusted left-eye image and an adjusted right-eye image obtained by adjusting the parallax between the left-eye image and the right-eye image by the parallax adjustment amount based on the image for use. wherein the parallax adjustment amount calculation means, characterized that you calculate the parallax adjustment amount as a value obtained by multiplying a predetermined coefficient to the frame parallax.

  According to the present invention, a stereoscopic image that protects the eyes of an observer, suppresses an increase in cost, and generates a suitable depth feeling that does not depend on the distance between the observer and the display surface or individual differences between observers is generated. Thus, it is possible to reduce the parallax calculation and the parallax adjustment amount calculation load, which can be reduced as compared with the case where the calculation is performed in units of pixels.

FIG. 1 is a diagram illustrating a functional configuration example of a stereoscopic image display apparatus according to the present invention. FIG. 2 is a diagram illustrating a relationship between parallax between a left-eye image and a right-eye image that are a pair of stereoscopic images and a sense of depth felt by an observer. FIG. 3 is a diagram illustrating an example of the relationship between the gray level and the horizontal position of the input left-eye image data and the input right-eye image data in the block. FIG. 4 is a diagram illustrating an example of a relationship between cross-correlation information and disparity information. FIG. 5 is a diagram illustrating the relationship between the intra-frame parallax information and the parallax adjustment amount. FIG. 6 is a diagram illustrating a relationship between the intra-frame parallax information and the parallax adjustment amount according to another operation example of the parallax adjustment amount calculation unit. FIG. 7 is a diagram illustrating an example of the relationship between the parallax adjustment amount and the change in the feeling of depth.

  Embodiments of an image processing apparatus, a stereoscopic image display apparatus, and an image processing method according to the present invention will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

Embodiment.
FIG. 1 is a diagram illustrating a functional configuration example of a stereoscopic image display apparatus according to the present invention. As shown in FIG. 1, the stereoscopic image display apparatus according to the present embodiment includes a cross-correlation detection unit 1, a parallax calculation unit 2, an intra-frame parallax calculation unit 3, a parallax adjustment amount calculation unit 4, and an image generation unit. 5 and a display unit 6. In the present embodiment, a stereoscopic image display device including the display unit 6 will be described as an example. However, the cross-correlation detection unit 1, the parallax calculation unit 2, the intra-frame parallax calculation unit 3, and the parallax adjustment amount An image processing apparatus including the calculation unit 4 and the image generation unit 5 may be configured, and the operation of the present embodiment may be applied to the image processing apparatus. In this case, the image processing apparatus corrects the input stereoscopic image and outputs a corrected image (output of the image generation unit 5).

  The stereoscopic image display apparatus according to the present embodiment receives input left-eye image data Da1 and input right-eye image data Db1 generated for stereoscopic image display. The input left-eye image data Da1 and the input right-eye image data Db1 are a pair of stereoscopic images generated to obtain a pseudo depth. The input left-eye image data Da1 is generated as an image for the observer to see with the left eye, and the input right-eye image data Db1 is generated as an image for the observer to see with the right eye. The observer recognizes the stereoscopic image with a sense of depth by observing the input left-eye image data Da1 with the left eye and the input right-eye image data Db1 with the right eye. The generation method of the input left-eye image data Da1 and the input right-eye image data Db1 may be generated by any generation method as long as the generation method is based on the parallax of the left and right eyes.

  The cross-correlation detection unit 1 detects cross-correlation in units of blocks obtained by dividing one frame into a finite number based on the input left-eye image data Da1 and the input right-eye image data Db1, and cross-correlation information in the block Output T1. The parallax calculation unit 2 calculates the parallax based on the cross-correlation information T1, and outputs the calculation result as the parallax information T2. The intra-frame parallax calculation unit 3 calculates one parallax for the frame based on the parallax information T2, and outputs the calculation result as intra-frame parallax information T3.

  The parallax adjustment amount calculation unit 4 calculates and outputs the parallax adjustment amount T4 based on the parallax adjustment information S and the intra-frame parallax information T3 that are information for adjusting the pop-out amount felt by the observer. The image generation unit 5 generates and outputs adjusted left-eye image data Da2 and adjusted right-eye image data Db2 based on the input left-eye image data Da1, the input right-eye image data Db1, and the parallax adjustment amount T4. . The display unit 6 performs a display operation based on the adjusted left-eye image data Da2 and the adjusted right-eye image data Db2.

  Next, the detailed operation of the cross correlation detection unit 1 will be described. FIG. 2 is a diagram illustrating a relationship between parallax between a left-eye image and a right-eye image that are a pair of stereoscopic images and a sense of depth felt by an observer. It is assumed that the image for the left eye is controlled to be visible only to the left eye of the observer, and the image for the right eye is controlled to be visible only to the right eye of the observer.

  FIG. 2 illustrates pixels P1l and P2l as pixels of the image for the left eye displayed on the display surface 7, and pixels P1r and P2r as pixels of the image for the right eye. The pixel P1l and the pixel P1r displayed on the display surface 7 are pixels that display the same object, and the pixel P2l and the pixel P2r are pixels that display the same object. In the case of FIG. 2A, the observer 8 seems to have an object corresponding to the pixel P1l and the pixel P1r at the point F1 in the space. In the case of FIG. 2B, it appears that an object corresponding to the pixel P2l and the pixel P2r exists at the point F2.

  The difference between the horizontal coordinate of the pixel of the image for the left eye and the horizontal coordinate of the pixel of the image for the right eye when the same object is displayed is parallax. 2A, it can be expressed as parallax d1 = X (P1l) −X (P1r), and in FIG. 2B, it can be expressed as parallax d2 = X (P2l) −X (P2r). it can. X (z) represents the horizontal coordinate of the pixel z.

  In FIG. 2A, the parallax d1 is a positive value. In this case, the observer 8 feels that the pixel being watched (pixel P1l and pixel P1r) exists at the position of the point F1 in front of the display surface. Conversely, in FIG. 2B, the parallax d2 is a negative value. In this case, it is felt that the pixels (pixel P2l and pixel P2r) that the viewer is gazing at exist at the position of the point F2 deeper than the display surface. Thus, here, the parallax is a signed value, which means that pixels with a positive parallax are displayed in front of the display surface, and pixels with a negative parallax are displayed in the back of the display surface.

  The cross-correlation detection unit 1 detects the cross-correlation by dividing the input left-eye image data Da1 and the input right-eye image data Db1 into block units obtained by dividing one frame into a finite number. When dividing into blocks, it is necessary to make the horizontal width of the block sufficiently larger (for example, about 5 times or more) than the assumed parallax so that the parallax can be detected in the block. It is desirable to divide the blocks so that the end portions of the blocks having a horizontal width equal to or greater than the assumed parallax overlap (included in both adjacent blocks) so that no unmatched portion occurs.

  FIG. 3 is a diagram illustrating an example of the relationship between the gray level and the horizontal position of the input left-eye image data Da1 and the input right-eye image data Db1 in the block. FIG. 3 shows an example in which one line in one frame is divided into a plurality of blocks. In FIG. 3, the horizontal axis represents the horizontal position within the block. The vertical axis represents the gradation of the input left-eye image data Da1 in FIG. 3A and the gradation of the input right-eye image data Db1 in FIG.

  Comparing the waveforms of FIG. 3A and FIG. 3B, there is a position where when the waveform of FIG. 3B is shifted to the right, a waveform similar to that of FIG. Since the gradation of the left eye image and right eye image data of the same object image is similar, the waveform of FIG. 3B is shifted to a state most similar to the waveform of FIG. It can be considered that the amount of shift at that time represents the parallax in the block.

  The cross-correlation detection unit 1 according to the present embodiment shows how similar the waveform is when shifted, that is, the shift amount (pixel unit), the input left-eye image data Da1 in the block, and the input right-eye image. It is a similarity detection means for detecting the relationship with the similarity of the data Db1. Specifically, as the similarity, for example, the cross-correlation function Rxy [m] of the input left-eye image data Da1 and the input right-eye image data Db1 divided into blocks is used, and fast Fourier transform and inverse transform are used. And ask. Here, a case where one line is divided into a plurality of blocks will be described as an example. Here, a case where one line is divided into a plurality of blocks will be described as an example.

  Here, m indicates the number of shifts (shift amount in pixel units), M indicates the number of pixels in the horizontal direction (maximum shift number) corresponding to the assumed parallax in the block, and L is divided in block units. Indicates the number of input left-eye image data Da1 (the number of input right-eye image data Db1 divided in units of blocks is the same), N indicates a power of 2 that is greater than or equal to M + L, and 0 ≦ n ≦ (N−1), x [n] is a data string of input left-eye image data Da1 divided into blocks, and y [n] is an input right-eye image divided into blocks. A data string of the data Db1 is assumed. Further, X [n] = FFT [x [n]] (FFT [z] represents fast Fourier transform of z), Y [n] = FFT [y [n]]. X [n] * represents a complex conjugate of X [n], and IFFT [z] represents an inverse fast Fourier transform of z.

The cross correlation function Rxy [m] is obtained by the following equation (1). Rxy [m] means the m-th data from the front of the data string obtained by the inverse Fourier transform.
Rxy [m] =
(IFFT [(X [n] *) · Y [n]]) / N, 0 ≦ m ≦ M, M + L ≦ N (1)

  An example of a procedure for calculating the cross-correlation function Rxy [m] based on Expression (1) will be described. First, the cross-correlation detection unit 1 divides the data sequence x [n] of the input left-eye image data Da1 and the data sequence y [n] of the input right-eye image data Db1 divided into blocks into N points ( Fast Fourier transform with order N) to obtain X [n] and Y [n]. In order to obtain an N-point data string, x [n] and y [n] are the points where the data of the input left-eye image data Da1 and the input right-eye image data Db1 are insufficient (each NL points). The value of (minute) is set to 0, for example.

  Then, the complex conjugate of X [n] is obtained, and the complex conjugate of X [n] and Y [n] are multiplied for each data point, and then the inverse fast Fourier transform is performed on the multiplication result. Finally, the cross correlation function Rxy [m] is obtained by dividing the inverse fast Fourier transform result by the number N of data. In this way, Rxy [m] corresponding to m of 0 ≦ m ≦ M (the number of obtained data points is N but is significant up to M) is obtained. Further, a product obtained by multiplying the complex conjugate of X [n] and Y [n] for each data point is processed in the same manner to obtain a cross-correlation function Ryx [m] for the shift in the opposite direction. The obtained Rxy [m] and Ryx [m] are output as cross-correlation information T1. Here, since the relative magnitude of the similarity for each shift amount m is used, it is not divided by N in Equation (1), or a monotonically increasing function or a monotonically decreasing function is applied to the result of Equation (1). The obtained value may be used as the similarity of the shift amount m.

  As shown in FIG. 2, the input right-eye image data Db1 is used as a reference in cases where the object is viewed in front of the display surface 7 and in the case where the object is visually recognized as being behind the display surface 7. The input left-eye image data Da1 has a different shift direction. Therefore, either direction is set as a positive direction of the shift number, and the direction of the shift number can be distinguished by a sign.

  Here, the cross-correlation information T1 is obtained as the similarity using the fast Fourier transform or the inverse fast Fourier transform, but the method for obtaining the similarity is not limited to this. For example, one of the input right-eye image data Db1 and the input right-eye image data Db1 is shifted by m, and a difference for each data between the non-shifted and the shifted one is obtained, and the total difference is calculated for each shift number. The similarity for each number of shifts may be obtained by another method such as obtaining.

  In the above example, one line is divided into a plurality of blocks to form a horizontal one-dimensional data string. However, the horizontal width of the block is assumed to be larger than the maximum parallax that can be detected so that the parallax can be detected within the block. As long as it is sufficiently large, it may be blocked in two dimensions, horizontal and vertical. In this case, for example, a cross-correlation function for each shift number is obtained for each line in the block in the same manner as described above, and a result obtained by adding the cross-correlation functions for all lines in the block for each shift number is used as cross-correlation information T1. Alternatively, for each pixel having the same horizontal coordinate, an average value of gradations may be obtained in the vertical direction, and a cross-correlation function may be obtained for this average value. The method of obtaining the cross-correlation information T1 in the case of horizontally and vertically two-dimensionally blocking is not limited to these, and any method may be used as long as the image similarity can be evaluated according to the number of shifts. .

  Further, when obtaining the cross-correlation coefficient in the vertical direction within the block, it is possible to deal with the parallax in the vertical direction. An image photographed by a camera or the like may unintentionally shift slightly in the vertical direction, and the detection accuracy may be improved by obtaining vertical parallax.

  As described above, the cross-correlation detection unit 1 detects the cross-correlation information T1 of the block in units of blocks, and thus can perform processing for one block at a time. Therefore, the motion vector is obtained by moving image compression or the like. Compared with the case where cross-correlation information is detected by performing block matching or the like in units of pixels that are also used at the time, the amount of calculation can be significantly reduced to about 1/10 or less.

  Next, the operation of the parallax calculation unit 2 will be described. The parallax calculation unit 2 calculates the parallax based on the cross-correlation information T1, and outputs the parallax information T2. FIG. 4 is a diagram illustrating an example of the relationship between the cross-correlation information T1 and the parallax information T2. In FIG. 4, the horizontal axis indicates the number of shifts, and the vertical axis indicates cross-correlation information. As shown in FIG. 4, there is a shift number that maximizes the value of the cross-correlation information T1, that is, a shift number that has the strongest cross-correlation between the input left-eye image data Da1 and the input right-eye image data Db1 of the block. . This shift number corresponds to the parallax. Accordingly, the parallax calculation unit 2 sets the shift number that maximizes the value of the cross-correlation information T1 as the parallax between the input left-eye image data Da1 and the input right-eye image data Db1 of the block, and outputs the parallax information T2.

  As described above, the shift number is a positive or negative value depending on the horizontal position of the corresponding pixel of the input right-eye image data Db1 when the pixel of the input left-eye image data Da1 is used as a reference. Similarly, the parallax is obtained as a value having a sign.

  The parallax calculation unit 2 selects the number of shifts when the cross-correlation between the input left-eye image data Da1 and the input right-eye image data Db1 of the block is the strongest in order to determine the parallax information T2 of the block. To do. Therefore, since the calculation for obtaining the parallax only needs to be performed once per block, the amount of calculation can be greatly reduced as compared with the case of calculating the parallax information for each pixel. Furthermore, by setting the number of shifts when the cross-correlation is the strongest as the disparity information T2 of the block, the difference from the case where the disparity information is obtained for each pixel can be reduced.

  Next, the operation of the intra-frame parallax calculation unit 3 will be described. Based on the disparity information T2 output for each block, the intra-parallax calculation unit 3 uses the disparity information T2 that is the maximum among the disparity information T2 of the blocks included in the frame as the disparity corresponding to the frame. Obtained and output as intra-frame disparity information T3. In the present embodiment, the maximum disparity information among the disparity information T2 of the block included in the frame is the intra-frame disparity information T3. However, the present invention is not limited to this. One intra-parallax information T3 of the frame is determined based on the parallax information T2 in units of blocks, such as calculating the average of the parallax information of the blocks included in the frame to obtain the intra-frame parallax information T3 of the frame. Any method can be used.

  In the intra-frame parallax calculation unit 3, the maximum parallax information among the parallax information T2 of the blocks included in the frame is set as the intra-frame parallax information T3. Therefore, since only one piece of parallax information needs to be output per frame, the amount of parallax information for one frame can be greatly reduced as compared with the case where the parallax information is calculated and output for each pixel. Furthermore, it is possible to output the maximum parallax information in the frame, which is an important factor in suppressing the pop-up amount of the stereoscopic video that has a great influence on the observer, as the parallax information of the frame.

  Next, an example of the operation of the parallax adjustment amount calculation unit 4 will be described. The parallax adjustment amount calculation unit 4 calculates and outputs the parallax adjustment amount T4 based on the parallax adjustment information S and the intra-frame parallax information T3. The parallax adjustment information S includes a parallax adjustment coefficient Sa. The parallax adjustment amount calculation unit 4 calculates the parallax adjustment amount T4 by multiplying the intra-frame parallax information T3 by the parallax adjustment coefficient Sa.

  FIG. 5 is a diagram illustrating a relationship between the intra-frame parallax information T3 and the parallax adjustment amount T4. In FIG. 5, the horizontal axis indicates the intra-frame parallax information T3, and the vertical axis indicates the parallax adjustment amount T4. The parallax adjustment coefficient Sa is a value that determines the slope of the straight line shown in FIG. The parallax adjustment coefficient Sa is a positive value. If the value of the parallax adjustment coefficient Sa is large, the parallax adjustment amount T4 changes greatly according to the intra-frame parallax information T3, and if the value of the parallax adjustment coefficient Sa is small, the parallax adjustment amount T4 is Change modestly. The parallax adjustment amount T4 is an adjustment amount used to limit the amount of pop-up observed by the observer by the parallax adjustment processing of the image generation unit 5 described later. For example, if the observer feels that the pop-out amount is too large by providing a means for adjusting the pop-out amount that is easy for the observer to see while viewing the actual picture, the parallax should be reduced to such an extent that it does not feel uncomfortable. By increasing the parallax adjustment coefficient Sa so as to be a large value, the parallax adjustment amount T4 increases, and the pop-out amount can be further suppressed. For example, if the parallax is up to 25 pixels, it is not uncomfortable and the assumed maximum parallax is 50 pixels. In this case, the parallax adjustment coefficient Sa = T4 / T3 so that the intra-frame parallax information T3 of 50 pixels becomes 25 parallaxes, that is, the parallax adjustment amount T4 becomes 25 (= 50−25) pixels. = 25/50 = 0.5.

  As described above, the intra-frame parallax information T3 and the parallax adjustment amount T4 are proportional to each other, and the inclination thereof can be adjusted by the parallax adjustment coefficient Sa, so that the effect of suppressing the pop-out amount becomes larger when the intra-frame parallax is large. If not, the effect of suppressing the pop-out amount can be reduced. Moreover, since the effectiveness of the deterrent effect can be controlled by the adjustment coefficient, it can be adjusted to a pop-out amount suitable for the observer.

  Here, in order to suppress the pop-out amount, the parallax adjustment amount T4 is calculated as a value for reducing the parallax, but it can also be applied to increase the pop-out amount. In this case, for example, when the parallax adjustment amount T4 is a negative value, the image generating unit 5 performs parallax adjustment so as to increase the parallax, and the parallax adjustment coefficient Sa is set to a negative value. Can be considered.

  Next, another operation example of the parallax adjustment amount calculation unit 4 will be described. FIG. 6 is a diagram illustrating a relationship between the intra-frame parallax information T3 and the parallax adjustment amount T4 according to another operation example of the parallax adjustment amount calculation unit 4. In this operation example, the parallax adjustment information S includes a parallax adjustment coefficient Sa and a parallax adjustment threshold Sb. The parallax adjustment coefficient Sa represents the slope of a straight line indicating the relationship between the intra-frame parallax information T3 and the parallax adjustment amount T4, as in the above example. The parallax adjustment threshold Sb is a value for constraining the parallax adjustment amount T4. When the intra-frame parallax information T3 is equal to or less than the parallax adjustment threshold Sb, the parallax adjustment amount T4 is set to 0 as illustrated in FIG. Thereby, when the intra-frame parallax information T3 is equal to or smaller than the parallax adjustment threshold Sb, the parallax adjustment in the image generation unit 5 can be prevented from being performed. The parallax adjustment threshold Sb is also adjusted and set while the viewer looks at the image so that the viewer can obtain an image that is easy to see while viewing the actual picture. When emphasizing that the pop-out amount does not increase, the parallax adjustment threshold value Sb is set small. In the case where priority is given to the original image and only excessive pop-up is suppressed, the parallax adjustment threshold value Sb is set large. When the intra-parallax information T3 changes across the parallax adjustment threshold Sb, it is desirable to change the parallax adjustment amount T4 with an appropriate time delay operation so that the pop-out amount does not change discontinuously.

  Thus, by using the parallax adjustment coefficient Sa and the parallax adjustment threshold value Sb, it is possible for the observer to perform the parallax adjustment when the pop-out amount is larger than the predetermined size, and not to perform the parallax adjustment otherwise. Become. Therefore, it is possible to adjust the amount of projection to a suitable amount so as not to impair the parallax set in the input video signal as much as possible and to avoid a burden on the observer's eyes.

  Next, the operation of the image generation unit 5 will be described. Based on the parallax adjustment amount T4 output from the parallax adjustment amount calculation unit 4, the image generation unit 5 translates either or both of the left-eye image and the right-eye image, and parallaxes by the parallax adjustment amount T4. The parallax adjustment is performed by generating a new left-eye image and right-eye image in which is changed. In a normal TV or the like, there is an area that is not displayed on the screen due to overscan. Therefore, there is no case where there is no image to be displayed at the end even if the image is translated. When all the image data is displayed on the screen, the left-eye image and the right-eye image are slightly enlarged, and either or both of the left-eye image and the right-eye image change in parallax by the parallax adjustment amount T4. For example, it may be translated.

  FIG. 7 is a diagram illustrating an example of the relationship between the parallax adjustment amount and the change in the feeling of depth. FIG. 7A is the same as FIG. 2A. FIG. 7B shows the positions of the pixels felt on the screen after being adjusted by the image generator 5 and the object felt by the observer. Pixels corresponding to the pixels P1l and P1r of the image before the parallax adjustment by the image generation unit 5 are the pixel P1l ′ and the pixel P1r ′, respectively. In FIG. 7B, the parallax d1 is adjusted to parallax so that the parallax d1 is reduced by d1 · Sa by the image generation unit 5 to become parallax d1 ′.

  FIG. 7 is a diagram illustrating an example of the relationship between the parallax adjustment amount and the change in the feeling of depth. FIG. 7A is the same as FIG. 2A. FIG. 7B shows the positions of the pixels felt on the screen after being adjusted by the image generator 5 and the object felt by the observer. Pixels corresponding to the pixels P1l and P1r of the image before the parallax adjustment by the image generation unit 5 are the pixel P1l ′ and the pixel P1r ′, respectively. In FIG. 7B, the parallax d1 is adjusted to parallax by the image generation unit 5 to become parallax d1 ′.

  As illustrated in FIG. 7B, the image generation unit 5 changes the left-eye image and the right-eye image based on the parallax adjustment amount T <b> 4, so that the observer 8 can obtain a point F <b> 1 ′ in the space. It seems that there is an object corresponding to the pixel P1l ′ and the pixel P1r ′. That is, compared to FIG. 7A, it can be felt that the object has moved to the back by ΔF. Accordingly, the amount of object popping out for the observer 8 is reduced.

  As described above, the image generation unit 5 adjusts the parallax adjustment image T2 for the left eye and the parallax that is the image for the left eye after the parallax adjustment based on the parallax adjustment amount T4, the input left-eye image Da1, and the input right-eye image Db1. An adjusted right-eye image Db2 that is an adjusted right-eye image is generated and output.

  Next, the operation of the display unit 6 will be described. The display unit 6 displays the adjusted left-eye image data Da2 and the adjusted right-eye image data Db2 as a left-eye image and a right-eye image, respectively. As a display method of the display unit 6 for performing stereoscopic display, for example, a method of displaying a left-eye image and a right-eye image at spatially different coordinates, or a left-eye image and a right image in temporally different frames. There are methods for displaying an image for the eye, but the method is not limited and any method may be used.

  As described above, in the present embodiment, the input left-eye image and the input right-eye image that are paired are calculated in units of blocks obtained by dividing one frame into a finite number of frames, and then one intra-frame parallax for the frame is obtained. The new left-eye image and right-eye image in which the parallax is changed from the input left-eye image and the input right-eye image based on the parallax adjustment information and the intra-frame parallax are calculated for each frame. . Therefore, regardless of the distance between the observer and the display surface and the individual differences between the observers, it is possible to reduce the burden on the eyes due to the excessive amount of protrusion, and to display a stereoscopic image with a sense of depth optimal for the observer, The parallax calculation and parallax adjustment amount calculation load can be reduced as compared with the case of calculating in units of pixels. In addition, it is possible to increase or decrease the sense of depth of the entire screen without requiring control of an optical system with high accuracy and without causing an increase in cost.

  As described above, the image processing device, the stereoscopic image display device, and the image processing method according to the present invention are useful for an image processing device that processes a left-eye image and a right-eye image that form a pair of stereoscopic images. In particular, it is suitable for an image processing apparatus that generates an image with reduced burden on the eyes of an observer of a stereoscopic video.

DESCRIPTION OF SYMBOLS 1 Cross correlation detection part 2 Parallax calculation part 3 In-frame parallax calculation part 4 Parallax adjustment amount calculation part 5 Image generation part 6 Display part 7 Display surface 8 Observer Da1 Input left-eye image data Db1 Input right-eye image data Da2 Adjustment Left eye image Db2 Adjusted right eye image P1l, P1r, P1l ′, P1r ′, P2l, P2r Pixel T1 Cross correlation information T2 Disparity information T3 Intraframe disparity information T4 Disparity adjustment amount S Disparity adjustment information Sa Disparity adjustment coefficient Sb Disparity Adjustment threshold

Claims (7)

A left-eye image and a right-eye image that form a pair for visually recognizing a stereoscopic image are divided into a plurality of blocks for each frame, and the left-eye image and the right-eye image are divided for each block. Similarity detection means for detecting the similarity between the left-eye image and the right-eye image for each horizontal shift amount;
A parallax calculating means for obtaining a parallax for each block as an intra-block parallax based on the similarity;
An intra-frame parallax calculating means for calculating an intra-frame parallax, which is a parallax within the frame, based on the intra-block parallax corresponding to the block within the same frame;
Parallax adjustment amount calculating means for calculating a parallax adjustment amount that is an adjustment amount of parallax based on the intra-parallax;
Based on the parallax adjustment amount, the left-eye image, and the right-eye image, the adjusted left-eye image and adjustment after adjusting the parallax between the left-eye image and the right-eye image by the parallax adjustment amount Image generating means for generating an image for the rear right eye;
Equipped with a,
The parallax adjustment amount calculating means, the image processing apparatus characterized that you calculate the parallax adjustment amount as a value obtained by multiplying a predetermined coefficient to the frame parallax. The similarity detection means calculates the similarity using a cross-correlation function between a horizontal data sequence of the left-eye image and a horizontal data sequence of the right-eye image in the block;
The image processing apparatus according to claim 1. The degree of similarity of the shift amount m for each block is represented by a data sequence in the horizontal direction of the left eye image in the block and a horizontal data sequence of the right eye image in the block, respectively. Data obtained by fast Fourier transform by performing a fast Fourier transform using a power of 2 N (> = M + L) that is equal to or greater than the sum of the number L of L and the assumed maximum shift number M (> m). The sequence is multiplied by one complex conjugate for each data point and multiplied by the other at the same position to generate a new data sequence, and m of the data sequence obtained by performing inverse Fourier transform of the order N on the generated data sequence The image processing apparatus according to claim 1, wherein the image processing apparatus is obtained from individual data. The parallax calculation means sets the shift amount that maximizes the similarity as the intra-block parallax.
The image processing apparatus according to claim 1, 2, or 3. The intra-frame parallax calculation means sets the maximum value of the intra-block parallax corresponding to the block in the same frame as the intra-frame parallax,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The parallax adjustment amount calculation means sets the parallax adjustment amount to 0 when the parallax adjustment amount is equal to or less than a predetermined threshold, and sets the parallax adjustment amount when the parallax adjustment amount exceeds a predetermined threshold. Calculated by multiplying the intra-frame parallax by a predetermined coefficient,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A left-eye image and a right-eye image that form a pair for visually recognizing a stereoscopic image are divided into a plurality of blocks for each frame, and the left-eye image and the right-eye image are divided for each block. A similarity detection step of detecting a similarity between the left-eye image and the right-eye image for each horizontal shift amount;
A parallax calculation step of obtaining a parallax for each block as an intra-block parallax based on the similarity;
An intra-frame parallax calculation step of calculating an intra-frame parallax, which is a parallax within the frame, based on the intra-block parallax corresponding to the block within the same frame;
A parallax adjustment amount calculating step of calculating a parallax adjustment amount that is an adjustment amount of parallax based on the intra-frame parallax;
Based on the parallax adjustment amount, the left-eye image, and the right-eye image, the adjusted left-eye image and adjustment after adjusting the parallax between the left-eye image and the right-eye image by the parallax adjustment amount An image generation step for generating an image for the rear right eye;
Only including,
In the parallax adjustment amount calculating step, the parallax adjustment amount is calculated as a value obtained by multiplying the intra-frame parallax by a predetermined coefficient .

Images