JPWO2019017290A1 - Stereoscopic image display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a more natural and eye-friendly stereoscopic image without causing reverse vision regardless of the position where the stereoscopic image is observed, and it is possible to observe the stereoscopic image from a wide range. Provided is a naked-eye type stereoscopic image display device capable of improving the hygiene condition of the observer's eyes. SOLUTION: This stereoscopic image display device includes an image display panel composed of a plurality of stereoscopic pixels each including m (m is an integer of 3 or more) arranged in a horizontal direction and a vertical direction, and an image signal. Image processing that obtains depth information for each pixel from the image frame for the right eye and the image frame for the left eye, and generates m viewpoint image frames for horizontally expanding and displaying the pixel image based on the depth information. The apparatus includes a display control device for displaying m viewpoint images represented by the m viewpoint image frames on the m pixels in a plurality of three-dimensional pixels of the image display panel. [Selection diagram] Fig. 1

Description

The present invention relates to a stereoscopic image display device, and more particularly to a naked-eye type stereoscopic image display device using a light ray reproduction method.

Conventionally, stereoscopic image display technology has often been used as a tool for entertainment. However, in recent years, due to their usefulness, their use in the medical field has been drawing attention.

As a currently popular stereoscopic image display device, a stereoscopic image display device using glasses such as an anaglyph system, a polarization system, or a shutter system, and a stereoscopic image with a parallax barrier or a lenticular killer lens without glasses is used. There is a stereoscopic image display device for recognizing an image. In these stereoscopic image display devices, two images slightly shifted to the right eye and the left eye are respectively input so that the amount of depth can be recognized based on the amount of parallax between the left and right eyes (two-eye type). ).

In the human visual system, solids are recognized by depth perception and stereoscopic perception, and most of the stereoscopic image production techniques and 3D techniques that have been developed so far utilize depth perception. However, a person usually involuntarily converges the left and right eyes to the gazing point in the visual field to obtain a sense of depth and a stereoscopic effect. In this case, even if there is no parallax in the gazing point, a person can see the object with a stereoscopic effect by stereoscopic perception. As described above, it is stereoscopic perception that is frequently used for stereoscopic recognition in daily life, eye strain is much smaller, and long-term observation is possible.

The depth perception used in the conventional stereoscopic image display device artificially creates a situation that does not occur naturally and causes extraordinary stereoscopic recognition.The depth perception depends on the parallax between the left and right eyes. The vergence of the eyes creates a feeling of depth, and the frequency and amount of that causes physiological fatigue. For this reason, safety is a problem such as suppression of eye strain when used for viewing for a long time, and the range of use is limited in the related art.

In addition, as a stereoscopic image display device that does not use glasses, a two-dimensional integral system that uses a lenticular lens that utilizes the principle of integral photography, a three-dimensional integral system that uses a lens array, a ray reproduction system, and a wavefront reproduction system. Examples include a stereoscopic image display device such as a system. Here, the integral method found in the market is to provide two parallax images having parallax at least in the horizontal direction to the left and right eyes by controlling the direction of a light beam by a lenticular lens or a lens array. After all, it is practically the same as the one based on the twin-lens type.

FIG. 14 is a diagram for explaining the principle of a conventional three-dimensional integral method using a lens array. In a photographing device, a gradient index lens array is arranged between a subject and a high-definition camera. The depth control lens arranged in front of the gradient index lens array causes a light beam emitted from a subject to enter the surface of the gradient index lens array. A condenser lens arranged behind the gradient index lens array focuses the light rays passing through the gradient index lens array on a high-definition camera, and the high-definition camera is formed on the back surface of the gradient index lens array. Take an elemental image.

The image signal representing the elemental image acquired by the high-definition camera is sent to the display device. The elemental image displayed on the flat panel display of the display device based on the image signal is a part of the observer's rays which substitutes the rays emitted in each direction from the subject by the lens array arranged in front of the flat panel display. Reach the eye. Thereby, the observer can recognize the stereoscopic image.

The three-dimensional integral method is intended to realize a stereoscopic effect not only in the horizontal direction but also in the vertical direction. However, since both eyes of a person are arranged along the horizontal direction, the depth in the vertical direction is recognized. It is difficult to do so, and although the recognizability due to the natural posture remains, the depth is actually perceived based on the horizontal parallax. Depth perception by perspective is based on a priori knowledge. After all, this three-dimensional integral method also has a problem that does not occur in natural vision, like the two-eye type stereoscopic recognition.

Patent Document 1 describes a naked-eye type stereoscopic image display device using an integral photography method or a light ray reproduction method. In the stereoscopic image display device described in Patent Document 1, a plurality of pinholes corresponding to a small area are provided on the front surface of a liquid crystal display device that displays a plurality of patterns corresponding to a stereoscopic image in a planarly divided small area. Alternatively, an array plate having microlenses arranged in a plane is provided. According to the invention of Patent Document 1, a plurality of patterns corresponding to a stereoscopic image to be displayed are subjected to discrete cosine transform and compressed, and when compressed image data is expanded by inverse discrete cosine transform for display, discrete cosine transform or inverse cosine transform is performed. It is characterized in that block noise is suppressed and a high-definition stereoscopic image can be displayed by matching the block unit to which the discrete cosine transform is applied with the unit of the element image.

However, in the stereoscopic image display device according to Patent Document 1, since the density of pinholes or microlenses is limited, the number of pixels and the number of light rays as a stereoscopic display are limited, and sufficient definition can be achieved. Have difficulty. In addition, there are problems that a stereoscopic image does not appear when the observer's eyes deviate from the expected observation position, and that so-called myopia occurs in which the left and right images are incident on the left and right eyes of the observer in reverse and the sense of depth is reversed. , There are restrictions on the observation position.

JP, 2006-148885, A

Therefore, in view of the above points, the problem to be solved by the present invention is that it is possible to observe a stereoscopic image from a wide range, and even if the stereoscopic image is observed from any position, no stereoscopy occurs. It is an object of the present invention to provide a naked-eye type three-dimensional image display device capable of displaying a more natural and eye-friendly three-dimensional image and improving the hygiene of the eyes of an observer.

In order to solve at least a part of the above problems, a stereoscopic image display apparatus according to one aspect of the present invention includes a plurality of stereoscopic pixels each including m (m is an integer of 3 or more) pixels in a horizontal direction. An image display panel configured by arranging in the vertical direction, depth information for each pixel is obtained from the image frame for the right eye and the image frame for the left eye of the image signal, and the pixel image is expanded in the horizontal direction based on the depth information. An image processing apparatus that generates m viewpoint image frames for display, and m viewpoint images represented by the m viewpoint image frames in the plurality of stereoscopic pixels of the image display panel. And a display control device for displaying each pixel.

A stereoscopic image display device according to one aspect of the present invention can sample a light beam poured from a viewing angle in natural vision, create an array of light that resembles natural light, and create a situation similar to natural vision. Noting that the positional shift between the right-eye image and the left-eye image corresponds to the depth of the image, the stereoscopic image display device uses one of the right-eye image and the left-eye image as a reference image. The amount of pixel shift in the reference image may be calculated in the other image, converted into the position of the viewpoint, and m viewpoint image frames corresponding to the m viewpoints set based thereon may be generated. .. It should be noted that the positions of those viewpoints are preferably determined based on minute viewpoint movements due to the involuntary eye movement (usually about 0.05 degree) related to stereoscopic perception that occurs in natural vision.

The vergence/divergence movements, the saccadic eye movements, and the impulsive eye movements of both eyes follow Hering's law, and the fixational tremor that occurs when the gazing point is viewed is the tremor of the vergence movement at the gazing point. Appears. For example, when the number of pixels in the horizontal direction of the image display panel is 1920, when the observer is located at a distance of 2H to 3H (H is the vertical width of the image display panel) from the image display panel, the viewpoint generated by the involuntary eye movement. Is equivalent to 3 to 5 three-dimensional pixels of the image display panel. It is said that one cycle of normal involuntary eye movement is about 5 msec, and when there are m viewpoint images, in a stereoscopic pixel, in about 5 msec (about 0.33 m to about 0.56 msec per subpixel). In a short time), m viewpoint images are recognized to cause a motion parallax, and a relative stereoscopic effect and a sense of depth are perceived based on the motion parallax.

The m viewpoint images generated in this way are displayed at the corresponding positions of the m pixels in the plurality of stereoscopic pixels as images captured from the positions of the virtual m cameras. Therefore, the light rays emitted from the m viewpoint images simultaneously enter the eyes of the observer, and the ray reproduction method reproduces the state in which the observer perceives a stereoscopic effect and a sense of depth from the object observed in nature. be able to.

As described above, according to one aspect of the present invention, it is possible to observe a stereoscopic image from a wide range, and no matter what position the stereoscopic image is observed, pseudoscopy does not occur and the stereoscopic image is more natural. It is possible to provide a naked-eye type stereoscopic image display device capable of displaying a stereoscopic image that is easy on the eyes and improving the hygiene of the eyes of the observer.

It is a block diagram which shows the structural example of the stereo image display apparatus which concerns on one Embodiment of this invention. FIG. 6 is a process diagram showing a procedure for displaying a stereoscopic image in the stereoscopic image display device according to the embodiment of the present invention. 6 is a diagram for explaining a situation in which a person receives light emitted from an object with his/her eyes. 3 is a diagram for explaining a relationship among subpixels, pixels, and stereoscopic pixels in the image display panel shown in FIG. 1. 9 is a diagram showing another example of arrangement of sub-pixels in a stereoscopic pixel in the image display panel shown in FIG. 1. 9 is a view showing still another arrangement example of sub-pixels in a stereoscopic pixel in the image display panel shown in FIG. 1. 9 is a view showing still another arrangement example of sub-pixels in a stereoscopic pixel in the image display panel shown in FIG. 1. It is a figure for demonstrating the shift|offset|difference of the image when a target object is image|photographed by the parallel method. It is drawing for demonstrating the shift|offset|difference of the image when a target object is image|photographed by the intersection method. It is a conceptual diagram for demonstrating the relationship between a viewpoint image and a pixel in case there are six viewpoints. It is a conceptual diagram for explaining an operation of a stereoscopic image display device concerning one embodiment of the present invention. It is a conceptual diagram for demonstrating the generation|occurrence|production mechanism of the stereoscopic perception by a microscopic fixation. It is a conceptual diagram for demonstrating the generation|occurrence|production mechanism of the stereoscopic perception by a microscopic fixation. 6 is a diagram for explaining the principle of a conventional three-dimensional integral method using a lens array.

The stereoscopic image display device according to an embodiment of the present invention is based on the principle of stereoscopic perception and is more natural and gentle to the naked eye, compared to a conventional device that provides depth perception by providing images with parallax to the left and right eyes. By realizing the stereoscopic image display of a formula, the eye strain of the observer is eliminated, and the restriction on the observation position is relaxed. Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a configuration example of a stereoscopic image display device according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating a stereoscopic image display device according to an embodiment of the present invention after an image signal is input. It is a flowchart showing a process until a stereoscopic image is displayed.

The stereoscopic image display device shown in FIG. 1 displays an image with m light rays obtained by sampling (convoluting) light emitted from an object in all directions at m (m is an integer of 3 or more) viewpoints. It uses a light ray reproduction method for displaying, and includes an image display panel 100, an image processing device 200, and a display control device 300. It is preferable that the image processing device 200 and the display control device 300 are configured by respective signal processing processors or integrated signal processing processors to enable a series of processing for each frame of the image signal.

The image display panel 100 is configured by arranging a plurality of stereoscopic pixels (polygons) each including m pixels in the horizontal direction and the vertical direction. The stereoscopic pixels are used to display m viewpoint images (multi-viewpoint images) in a superimposed manner on one screen. Each of the m pixels may include a plurality of sub-pixels that display the three primary colors of light. A subpixel is the smallest pixel element that emits light of each color that is modulated by the image signal.

For example, as the image display panel 100, a 2D display panel such as a transmissive or reflective liquid crystal or an organic EL (OLED: Organic Light Emitting Diode), particularly a display panel for a high definition television can be used. Alternatively, a video projector that includes a lamp, an image display panel 100, and a projection lens unit, and that emits light of the lamp toward the screen via the image display panel 100 and the projection lens unit is provided in the stereoscopic image display device. May be.

The image displayed by the image display panel 100 is a composite of viewpoint images of the number of viewpoints (m), and since each stereoscopic pixel includes m pixels, it can be used in a 2K, 4K, or 8K television or the like. It is preferable to use the high definition display panel used. In the image display panel 100, sub-pixels of each color that display the three primary colors of light RGB are arranged over the entire display area.

The image processing apparatus 200 obtains a parallax matrix (RL matrix) as depth information for each pixel from the image frame for the right eye and the image frame for the left eye of the image signal, and the pixel image in the horizontal direction based on the depth information. To generate m viewpoint image frames to be expanded and displayed. The image processing device 200 can be configured by an analog circuit, a digital circuit, or a CPU and software.

As illustrated in FIG. 1, the image processing apparatus 200 includes, as functional blocks, a distribution unit 1, frame memories 2 and 3, column sum calculation units 4 and 5, a column sum comparison unit 6, and a parallax matrix generation unit 7. , A viewpoint image generation unit 8 and a weighting function generation unit 10, and may further include an image signal conversion unit 11 and a frame selection unit 12.

For example, the image processing apparatus 200 stores the right-eye image frame and the left-eye image frame, accumulates the pixel values of each frame in the vertical direction for each column to obtain a column sum, and calculates the column sum between both frames. By comparing, the overlapping area in the images of both frames is specified, and the positional relationship between the images of both frames is adjusted.

After that, the image processing apparatus 200 obtains depth information for each pixel based on the displacement of the positions of the corresponding pixel images in both frames, and horizontally moves the pixel image in one frame based on the weighted depth information. By doing so, m viewpoint image frames including one frame are generated.

The display control device 300 causes the m viewpoint images represented by the m viewpoint image frames to be displayed on the m pixels among the plurality of stereoscopic pixels of the image display panel 100. Therefore, the display control device 300 includes an image memory 9 that stores m viewpoint image frames, and is configured by a digital circuit or the like.

FIG. 3 is a diagram for explaining a situation in which a person receives light emitted from an object with his/her eyes. In nature, light comes from all directions at 360 degrees. However, when a person looks at an object, only an area (gazing point) within a viewing angle of about 1 to 2 degrees can be seen. The depth information seen in the visual field of the gazing point brings about stereoscopic perception due to the line of sight converging to the gazing point, and images outside this visual field do not contribute to stereoscopic perception. Therefore, by convoluting (sampling) a plurality of light rays within a viewing angle of about 1 to 2 degrees within the display accuracy of the image display panel 100 (FIG. 1), the light rays emitted from the image display panel 100 are naturally It can resemble a ray of light. It should be noted that the larger the number of sample lights, the more possible the image reproduction with less discomfort becomes.

The depth information is obtained from the contraction/extension information of the internal rectus muscle and the external rectus muscle that horizontally rotate the eyeball among the six muscles arranged in the eyeball. The contraction/extension of the internal rectus muscle and the external rectus muscle occurs in vergence/divergence movement of both eyes, compliant eye movement (pursuit), and impulsive eye movement (saccade). What is noted here is the involuntary eye movement. The eyeball is constantly swaying due to a pupil oscillating motion, a fine motion adjusting motion, and a fixation micromotion which have different cycles. Microsaccades, drifts, tremors, and the like are known as involuntary eye movements. When the eyeball sways, the depth of the line of sight changes, and a plurality of viewpoint images can be acquired.

Microsaccades, which are weak fluctuations that occur involuntarily, are particularly closely related to stereoscopic perception, and the eyeballs sway at a cycle of about 1 to 3 Hz. The rotation angle of the eyeball due to the involuntary eye movement is usually measured to be about 0.05 degree. The speed is measured at about 10 degrees/second. When the eyeball rotates due to the involuntary eye movement, the line of sight of the observer with respect to the object also changes. For example, the line-of-sight L(i) from the observation point A(i) of the observer to the gazing point B of the object is injected from another observation point A(i+1) that is a distance D away from the observation point A(i). The line of sight L(i+1) toward the viewpoint B changes. Here, the closer the object is to the observer, the larger the angle formed by the line of sight L(i) and the line of sight L(i+1).

Assuming that the position of the gazing point B recognized by the observer changes due to the change of the line of sight, the lines of sight L(i) and L(i+1) are moved in parallel to determine the position of the observation point A(i) and the observation point A(i). Consider a state in which the position of point A(i+1) matches. In such a state, the gazing point B appears as two gazing points B(i) and B(i+1) separated by the distance D. Since the involuntary eye movement is repeated in a predetermined cycle, m viewpoint images corresponding to the m watching points B(i), B(i+1), B(i+2),... Are displayed on one screen. This makes it possible to embody the movement of the gazing point generated by the involuntary eye movement in the natural vision on the display screen. Here, the closer the object is to the observer, the larger the distance between the viewpoint images.

Basically, the sensory sensor of the body is designed to ignite when the stimulus exceeds a threshold value. Stereoscopic perception is created by detecting a change in light from the gaze area by involuntary eye movement and transmitting it to the brain. The minimum delay time for discriminating the relative depth is about 160 μsec, which is about 2 sec in terms of binocular parallax.

Next, a method of displaying light emitted from an object on a plurality of three-dimensional pixels of the image display panel will be described with reference to FIGS. 4 to 7.
FIG. 4 is a diagram for explaining a relationship among subpixels, pixels, and stereoscopic pixels in the image display panel shown in FIG. In the image display panel shown in FIG. 4, a plurality of sub-pixels displaying the same color are arranged in the vertical direction (vertical direction).

FIG. 4 shows a striped arrangement in which a sub-pixel R that displays red, a sub-pixel G that displays green, and a sub-pixel B that displays blue are repeatedly arranged side by side in the horizontal direction. Has been done. Each pixel is composed of one sub-pixel R, one sub-pixel G, and one sub-pixel B. Further, each three-dimensional pixel is composed of pixels corresponding to the number of viewpoints (m). FIG. 4 exemplifies a case where the number of viewpoints is 6, and one stereoscopic pixel includes 6 pixels, that is, 18 sub-pixels.

Pixels in a display panel of a general television receiver are composed of RGB sub-pixels arranged in a horizontal direction, but the pixels used in the image display panel shown in FIG. The image display panel includes a plurality of sub-pixels arranged in a plurality of adjacent lines. For example, as shown by connecting line segments in the figure, subpixels in adjacent rows and adjacent columns are selected to be diagonally arranged.

The position observed as the center of light emission of the pixel when all the sub-pixels included in each pixel are lit is called the light center of gravity, and as shown by the white circles in the figure, In the three-dimensional pixel, it is preferable that the m pixels are arranged so that the optical centers of gravity of the pixels are sequentially adjacent and arranged in the horizontal direction. In the example shown in FIG. 4, the sub-pixels are selected so that the pixels incline upward to the right, but the sub-pixels may be selected so that the pixels incline in the opposite direction.

The three-dimensional pixels are repeatedly arranged in the horizontal direction and the vertical direction over the entire display area of the image display panel to form a matrix. For example, in the case of a display panel for a 2K television, 1920 pixels in the horizontal direction and 1080 pixels in the vertical direction are formed by horizontally arranging RGB sub-pixels. Therefore, the stereoscopic pixel is 960 in the horizontal direction. , A matrix of 360 arranged in the vertical direction is formed. When the size of the three-dimensional pixel is set so that the observer who views the image display panel with a visual acuity of 1.0 has an angle of sight of about 1 minute or less, the observer can distinguish the pixels in the three-dimensional pixel. Instead, the three-dimensional pixel is recognized as a point.

The viewing angle of the line of sight with respect to the three-dimensional pixel can be appropriately adjusted by selecting the observation distance according to the size of the three-dimensional pixel. For example, in the case where the number of pixels in the horizontal direction of the image display panel is 1920, when the observer is located at a distance of 2H to 3H (H is the vertical width of the image display panel) from the image display panel, it is caused by the involuntary eye movement. The movement of the viewpoint is equivalent to 3 to 5 three-dimensional pixels of the image display panel.

The higher the definition of the panel, the more viewpoint images can be displayed. The larger the number of viewpoints, the more natural the three-dimensional effect can be obtained. Further, the larger the number of three-dimensional pixels in the screen, the finer the image can be obtained. For example, in a display panel for a 4K television, 3840 pixels are arranged in the horizontal direction and 2160 pixels are arranged in the vertical direction, so that nine viewpoint images are displayed using the nine pixels arranged in the horizontal direction. Even with such a configuration, a sufficiently fine stereoscopic image display panel in which 1280 stereoscopic pixels are arranged in the horizontal direction and 720 stereoscopic pixels are arranged in the vertical direction is obtained.

Further, in the display panel for the 8K television, 7680 pixels are arranged in the horizontal direction and 4320 pixels are arranged in the vertical direction, so that 12 viewpoint images are displayed using the 12 pixels arranged in the horizontal direction. Even with this configuration, a high-definition stereoscopic image display panel having 1920 horizontal pixels and 1440 vertical pixels is provided. In addition, when it is desired to suppress the spread of the three-dimensional pixel when the number of viewpoints increases, a plurality of sub-pixels forming each three-dimensional pixel are arranged in adjacent six lines in the image display panel, thereby The width can be halved.

FIG. 5 is a drawing showing another example of arrangement of sub-pixels in the three-dimensional pixel in the image display panel shown in FIG. The pixels used in the image display panel shown in FIG. 5 include a plurality of sub-pixels arranged in every other line in the image display panel in order to halve the horizontal width of the stereoscopic pixel. That is, a pixel composed of sub-pixels in the first tier, the third tier and the fifth tier, and a pixel composed of the sub-pixels in the second tier, the fourth tier and the sixth tier , Three-dimensional pixels are arranged alternately in the horizontal direction.

In the example shown in FIG. 5, each stereoscopic pixel includes 24 pixels, and 24 viewpoint images can be displayed, but the spread of the stereoscopic pixel in the horizontal direction is suppressed to 12 sub-pixels. There is. The optical centers of gravity of the pixels are not arranged in a straight line in the horizontal direction, but have widths alternately arranged at the positions of the two sub-pixels in the center, but when viewed broadly, the optical centers of gravity of the pixels are arranged horizontally in a small width. It is acceptable because it is connected.

6 and 7 are views showing still another arrangement example of the sub-pixels in the stereoscopic pixels in the image display panel shown in FIG.
In the image display panel shown in FIG. 6, a plurality of subpixels displaying the same color are arranged side by side in the horizontal direction (horizontal direction). That is, the sub-pixels R, G, and B are arranged in the horizontal direction for each color to form a row, and the rows of the sub-pixels R, G, and B are repeatedly arranged side by side in the vertical direction. Good. In this case, as shown by vertical line segments in the figure, each pixel is composed of sub-pixels of three colors arranged in the vertical direction, and the light center of gravity is composed of m pixels arranged in the horizontal direction. Three-dimensional pixels are configured.

Alternatively, as shown in FIG. 7, the sub-pixels of the image display panel may have a mosaic arrangement. Also in this case, if a plurality of sub-pixels forming each pixel are selected from three lines that are vertically adjacent to each other, a stereoscopic pixel having a narrow width is formed by m pixels whose light centers of gravity are arranged in the horizontal direction. It

Next, the process of displaying a multi-viewpoint image on the image display panel will be described with reference to FIGS. 1 and 2.
(Step 1)
When the right-and-left image signals RL representing the right-eye image and the left-eye image are input to the stereoscopic image display device illustrated in FIG. 1, the distribution unit 1 outputs the input right-and-left image signals RL to the right. An image frame for the eye (R frame) and an image frame for the left eye (L frame) are extracted, and are extracted according to the frame rate in a frame matrix storage unit (frame memory 2 for the R frame and frame memory 3 for the L frame). Sequentially store in.

As a result, in the frame matrix storage unit, pixel values representing the color intensity with appropriate resolution for each of the three primary color (RGB) subpixels are stored as the R frame and the L frame. For example, the pixel value representing the color intensity of each color by 256 values of (0 to FF), ternary value of (-1, 0, +1), or binary value of (0, 1) is stored.

When a component image signal that uses a luminance signal (Y signal) and two color difference signals (Pr signal, Pb signal) is input, the distribution unit 1 determines the pixel values of the Y signal, Pr signal, and Pb signal. The pixel value of RGB may be used by converting into the pixel value of RGB. On the other hand, when a broadcast image signal such as a side-by-side format (SbyS), a top-and-bottom format (Top & Bottom), or a frame packing method (FP) is input, the image signal conversion unit 11 inputs the input image. By separating the image for the right eye and the image for the left eye in the signal, the input image signal is converted into the image signal RL for the left and right eyes and supplied to the distribution unit 1.

Further, the stereoscopic image display device shown in FIG. 1 can also be used for processing a two-dimensional image in a conventional television broadcast to form a three-dimensional image and then displaying the image. In this case, the frame selection unit 12 selects the standard frame and the reference frame having a predetermined number of frame differences from the standard frame from the frame sequence of the two-dimensional image signal (2D), and the standard frame and the reference frame are selected. An image signal RL for the left and right eyes having an image frame for the right eye and an image frame for the left eye is generated based on the frame and supplied to the distribution unit 1. For example, the frame selection unit 12 may set the image of the selected reference frame as the image for the right eye, and the image of the reference frame selected with a predetermined number of frame differences to the image for the left eye.

A television image carried by an ordinary television broadcast wave is a two-dimensional image and does not have depth information of the subject. However, when the subject moves with respect to the camera or the camera moves with respect to the subject, the position of the subject in different frames in the two-dimensional image changes, causing a time lag in the image of the subject. When two images in which the image of the subject is horizontally shifted are distributed to the left and right eyes and supplied, the convergence angle when viewing the subject in the two images with both eyes is the same as when viewing the subject in the image of the reference frame with both eyes. Is different from the convergence angle of the object, the image of the subject appears to be displaced in the depth direction with respect to the display screen, and a stereoscopic effect is given to the image. Based on such a principle, the left and right eye image signals RL can be generated based on the two-dimensional image signals.

(Step 2)
The image processing apparatus 200 calculates the horizontal shift amount for the R frame and L frame shooting regions, and specifies the overlapping region where the left and right images overlap. The stereoscopic image display apparatus according to the present embodiment detects, for example, an overlapping area in which the left and right images captured by the two left and right cameras are overlapped, and gives a stereoscopic effect to the image based on the parallax between the left and right images in the overlapping area. Give rise to. The fields of view of the cameras do not match in the left and right images. However, in the overlapping region in the field of view, the left-eye image and the right-eye image display the same target object, so that it is expected that the pixel values in the left and right images will be substantially the same.

Therefore, for the column sums obtained by vertically summing the pixel values of the frame matrix by the column sum total calculation units 4 and 5, the column sum total comparison unit 6 calculates the R frame columns and the L frame columns. When the comparison is performed while shifting in the horizontal direction, the difference between the column sums becomes close to zero in the region where the left and right images overlap (overlap region). Therefore, the column sum total comparison unit 6 identifies the pair of columns for which the difference between the column sums is the smallest, and/or identifies the region where the difference between the column sums is smaller than a predetermined value, so that the left and right images are displayed. It is possible to discriminate the area where the and the area do not overlap. The column sum comparison unit 6 calculates the difference between the horizontal coordinates of the overlapping areas in the R frame and the L frame as the amount of shift between the left and right images.

Here, the sub-pixels used to obtain the column sum may be sub-pixels of all three primary colors, or may be sub-pixels of any one of the three primary colors. Further, when a component image signal using a luminance signal (Y signal) and two color difference signals (Pr signal, Pb signal) as image signals is input to the stereoscopic image display device, column summation is calculated for the luminance signals and duplication is performed. The area may be specified. Further, the left and right eye image signals input to the stereoscopic image display device can be obtained by shooting the object by the parallel method or the crossing method.

FIG. 8 is a diagram for explaining an image shift when an object is photographed by the parallel method. In the parallel method, the two cameras L and R are arranged so that the optical axes thereof are parallel to each other, and the left and right images captured by the left and right cameras are provided to the left and right eyes, respectively, and the observer The depth is sensed from the angle of convergence when the left and right eyes are aligned with the point of interest. If the actual camera interval is wider than the eye interval, the observer feels a large amount of depth and exaggerates the stereoscopic effect.

In the parallel method, an overlapping area appears in the left part in the image of the right camera R and in the right part in the image of the left camera L. For example, if the right camera R is shifted to the position of the left camera L, the same image will be acquired, so the shift Δ between the left and right images corresponds to the camera interval. It is not known from the image signal how long the left and right images were taken with the camera interval, but according to the above procedure, the shift amount between the left and right images can be calculated at high speed regardless of the camera interval. can do.

FIG. 9 is a diagram for explaining an image shift when an object is photographed by the intersection method. In the crossing method, the optical axes of the two cameras L and R are arranged so as to converge, and the left and right images captured by the left and right cameras are provided to the left and right eyes, respectively. The observer gets a sense of depth based on the angle of convergence required to match the images of both eyes. In the image obtained by the intersection method, the optical axes of the left and right cameras intersect with each other, so that the image of the right camera R has an overlapping region in the right portion, and the image of the left camera L has an overlapping region in the left portion. Further, the farther the point (convergence point) where the optical axes of the left and right cameras intersect becomes, the larger the overlapping area in the image becomes. Even in this case, it is not known from the image signal how much the left and right images were taken with the convergence angle, but according to the above procedure, regardless of the convergence angles of the two cameras, The amount of shift between the left and right images can be calculated at high speed.

As described above, in the stereoscopic image display device according to the present embodiment, it is possible to perform high-speed processing by the same procedure by using the procedure, regardless of the method by which the input image signal is acquired. it can.

(Step 3)
The parallax matrix generation unit 7 illustrated in FIG. 1 generates a parallax matrix (RL matrix) that is depth information for each pixel based on the parallax of the object in the R frame and the L frame in which the displacement amount is compensated. For the R frame and the L frame, when the positions are corrected by the amount of displacement calculated previously and the images are overlapped, the area where the left and right images overlap is located in the center, and the areas where the left and right images do not overlap are at both ends. An image map of the position is formed. Then, the parallax matrix generation unit 7 is a portion in which the position of the object represented by each pixel is changed between the left and right images in the image signal acquired in the same region of the images captured by the two cameras. Calculate the displacement amount of. The displacement amount of the region where only the image captured by one camera exists may be 0 (zero).

The parallax matrix generation unit 7 calculates the element value (parallax value) of the parallax matrix based on the displacement amount of each pixel between the left and right images. Note that the objects in the left and right images may not only be displaced in the horizontal direction in the images, but may also have displacement components in the vertical direction. In that case, since the displacement is obtained as a vector, the length of the displacement vector may be used as the element value of the parallax matrix. The human eye recognizes the depth because of the parallax in the horizontal direction.

The parallax matrix indicates the parallax of each of the objects in the left and right images, and the parallax is larger for an object that is nearer and smaller for an object that is far away, and defines the depth of the object in the image. Will be things. Note that the element value of the parallax matrix can be either positive or negative, and for pixels that display a certain object, a positive value when the object is viewed forward with the convergence point as a reference, and when viewed backward. The element value should have a negative value.

In addition, the weighting function generation unit 10 adjusts the shooting conditions and the depth vision regarding the object by appropriately weighting the element values of the parallax matrix when the parallax matrix generation unit 7 generates the parallax matrix. Can be adjusted. For example, if an appropriate value is added to the element value, the vergence point moves forward and the image of the object approaches, and if an appropriate value is subtracted from the element value, the vergence point moves away and the image of the object recedes. Also, the stereoscopic effect is emphasized in the area where the element value is multiplied by an appropriate value larger than 1, and the stereoscopic effect is reduced when the element value is multiplied by a value smaller than 1. Further, upper and lower limits may be set for the element values of the parallax matrix, or an appropriate function such as a gamma function may be used to select an appropriate value according to the magnitude of the element values.

(Step 4)
For each frame (one screen) of an image displayed on the image display panel 100, the viewpoint image generation unit 8 generates m viewpoint image frames for horizontally expanding and displaying a pixel image based on the parallax matrix. To generate. For example, the viewpoint image generation unit 8 uses one of the R frame and the L frame (R frame in FIG. 1) stored in the frame matrix storage unit as the reference image frame, and the pixel image in the reference image frame. Is generated in the horizontal direction by the element value (parallax value) of the parallax matrix to generate a reference image frame.

Further, the viewpoint image generation unit 8 interpolates between the standard image frame and the reference image frame based on the standard image frame and the element value (parallax value) of the parallax matrix (m-2) interpolated image frames. To generate. For example, the viewpoint image generation unit 8 uses the value obtained by dividing the element value of the parallax matrix by (m-1) as the horizontal movement amount for each viewpoint image, and sequentially moves the pixel images in the reference image frame in the horizontal direction. , (M−2) interpolated image frames may be sequentially generated from the reference image frame.

Alternatively, the viewpoint image generation unit 8 may generate (m-1) interpolated image frames in order from the standard image frame without generating the reference image frame. Here, the m viewpoint images are generated with reference to one of the left and right images. The other image is used only to obtain depth information for each pixel in the reference image. Therefore, the viewpoint image farthest from the reference image is not the same as the image for the left eye even when the image for the right eye is used as the reference image, and is generated from the reference image based on the depth information.

(Step 5)
The viewpoint image generation unit 8 stores m viewpoint image frames representing m viewpoint images in the image memory 9 of the display control device 300. Accordingly, the display control device 300 causes the m viewpoint images represented by the m viewpoint image frames to be displayed on the m pixels in the plurality of stereoscopic pixels of the image display panel 100. The image memory 9 can store pixel values that specify the intensity of color development for all subpixels on the display screen. The display control device 300 causes the sub-pixels of the image display panel 100 to develop colors in accordance with the pixel values stored in the image memory 9.

In the image memory 9, m viewpoint image frames are stored in the storage area for one screen. Each of the three-dimensional pixels of the image display panel 100 displays the pixel images at the same position in the m viewpoint images almost at the same time. However, when the three-dimensional pixels are arranged over the 3 to 6 lines of the image display panel 100, there is a time difference for scanning the 3 to 6 lines. The display control device 300 assigns m viewpoint images to m pixels in a plurality of three-dimensional pixels of the image display panel 100 and displays them. A plurality of sub-pixels in each pixel are colored with primary colors having intensities designated in multiple stages, and the three primary colors appear to be mixed at the position of the optical center of gravity, so that the designated colors can be displayed.

As described above, in the stereoscopic image display device according to the present embodiment, one screen image displayed on the image display panel 100 is composed of m viewpoint images, and the m viewpoint images are displayed on the image display panel 100. Each is displayed by m types of pixels that form a three-dimensional pixel. Each pixel in the three-dimensional pixel is composed of, for example, three sub-pixels arranged obliquely in the vertical direction, and m pixels are arranged in the horizontal direction, for example. A plurality of stereoscopic pixels, each of which includes m pixels, develop and display m viewpoint images in the horizontal direction, thereby causing the observer to perceive a stereoscopic effect and a sense of depth due to the involuntary eye movement.

FIG. 10 is a conceptual diagram for explaining the relationship between viewpoint images and pixels when there are six viewpoints. FIG. 10 shows the positional relationship between the cross section of the object cut along a horizontal plane and the pixels of the viewpoint image. The images of the object captured by the two cameras are displaced according to the depth. The magnitude of this shift is taken in the horizontal direction as parallax P, and six viewpoint images A, B, C, D, E, and F arranged in the horizontal direction are shown in the meantime.

Since the six viewpoint images are displayed on one screen, the storage area for one screen in the image memory 9 stores the pixel information of the six viewpoint images to be projected at the corresponding positions. Six pixels lined up in the horizontal direction in each of the three-dimensional pixels correspond to the viewpoint images A, B, C, D, E, and F of the object, and the viewpoint images a, b, c, d on the screen, Display e and f. These viewpoint images correspond to a plurality of viewpoints generated by the involuntary eye movement.

As a result, the image display panel 100 displays an image of the target object reproduced by sampling appropriate m pieces of light rays emitted from the target object in all directions in the natural world. Here, “sampling” refers to compression of light rays emitted from all directions according to the definition of a display panel by using a convolution principle. Since a plurality of light rays relating to m viewpoint images are simultaneously incident on the observer's eyes, a natural stereoscopic effect equivalent to that sensed in the natural world is formed.

Note that the viewing angle with respect to the point where the human eye gazes is about 1 to 2 degrees, and even an image cannot be perceived in a wider range than this (Troxler effect), so the spread of the viewpoint image is about 1 to 2 degrees. It is preferable to set it within the range of. That is, it is preferable that the parallax P in FIG. 10 be within a range of a viewing angle of 1 to 2 degrees from the position of the observer. If the parallax P is too large in the actual image, the parallax value can be adjusted by weighting the element values of the parallax matrix using an appropriate weighting function.

Therefore, the weighting function generation unit 10 performs addition/subtraction, multiplication/division, or an operation of an appropriate function on the element values of the parallax matrix to adjust the vergence angle of the two cameras or to change the viewpoint image. It is possible to adjust the parallax and the depth feeling of the subject. For example, when a positive number is added to the element value, the parallax increases and the stereoscopic effect can be emphasized. This has the same effect as adjusting the shooting conditions, tilting the optical axis of the camera further, and bringing the convergence position closer. Further, by multiplying and dividing the element value by an appropriate value, the depth feeling of the entire image can be expanded or compressed.

Furthermore, to prevent the presence of extraordinarily large parallax or small parallax from excessively suppressing the stereoscopic effect in parallax with a high output frequency, use a filter that utilizes the gamma characteristic or S-shaped tone curve. It can also be used to adjust the vergence angle according to the depth of the image. Further, the weighting function generating unit 10 can be provided with a function of emphasizing the stereoscopic effect of the attention portion while suppressing the stereoscopic effect of the near portion and the distant portion. Further, when a maximum value or a minimum value that is incompatible with the surrounding situation is detected, it is possible to remove these as an abnormal value. It is also possible to non-linearly compress a displacement that deviates from the deviation value σ, 2σ, or 3σ of the displacement amount, or ignore it by setting the parallax amount to 0.

FIG. 11 is a conceptual diagram for explaining the operation of the stereoscopic image display device according to the embodiment of the present invention. FIG. 11 shows an image pickup optical system and an image pickup surface of an image pickup apparatus which picks up an object (subject) in the external world, and a ray reproduction display screen of a display apparatus which displays an image obtained by the image pickup apparatus.

On the ray reproduction display screen, viewpoint images obtained by sampling the set number m of viewpoints are displayed, and light rays are emitted forward from the three-dimensional pixels of the ray reproduction display screen. At this time, since a plurality of light rays relating to m viewpoint images radiated from one location are simultaneously incident on the observer's eye, the light ray reproduction method by sampling is combined with the image change recognition due to the involuntary eye movement. It is possible to perceive a more natural stereoscopic effect and a sense of depth based on. Those rays can also be caught by the line of sight of another angle.

12 and 13 are conceptual diagrams for explaining the mechanism of occurrence of stereoscopic perception due to involuntary eye movement. 12 and 13 show a situation in which an image is sampled by the light ray reproduction method and displayed on the image display panel. This situation simulates the perceptual mechanism in natural light.

FIG. 12 illustrates a case where six pixels are provided in the three-dimensional pixel. Six pixels (viewpoint pixels) corresponding to images from viewpoints A to F are arranged in one stereoscopic pixel, and six viewpoint images are displayed by a set of stereoscopic pixels. In FIG. 12, the pixels in the three-dimensional pixel are shifted in the vertical direction in order to show the connections of the images in an easy-to-see manner, but the pixels may be arranged substantially in the horizontal direction on the display surface.

In such a situation, the involuntary eye movement causes the image reflected on the retina to move laterally geometrically by the amount of the slight movement of the line of sight. As shown in the lower side of FIG. 12, the retinal image slightly moves ±0.05 degrees with respect to the reference image receiving pixels (a1, b1, c1, d1, e1, f1) of the stage 1, It changes like the left eye image receiving pixel (b1, c1, d1, e1, f1, a2) and the right eye image receiving pixel (f0, a1, b1, c1, d1, e1) of the stage 2. Note that the left eye and the right eye have opposite image shift directions.

FIG. 13 is a conceptual diagram for explaining a mechanism in which depth perception corresponding to a fine movement occurs due to a shift between images of the left eye and the right eye.
The human optic nerve is semi-crossed in the brain, the nasal nerve of the retina of the right eye crosses and connects to the primary visual cortex of the left brain, and the nerve of the left ear of the retina of the left eye does not cross Connected to the primary visual cortex of the right eye, the nerves on the ear side of the retina of the right eye do not cross and connect to the primary visual cortex of the right brain, and the nerves on the nasal side of the retina of the left eye cross and connect to the primary visual cortex of the right brain .. Therefore, the image in the right half of the visual field is processed by the left brain, and the image in the left half of the visual field is processed by the right brain.

As a result, the shift between the left-eye image and the right-eye image on the stage 2 is detected, and depth perception occurs based on the shift amount. In FIG. 13, for example, stereoscopic perception is obtained by the shift of the b1 image perceived by the right brain and the shift of the e1 image perceived by the left brain. This shift is a vergence that is caused by involuntary eye movement that occurs within the visual field of the gazing point.When the object is distant, it shifts slightly, and when the object is close, it becomes a large parallax of movement, and in a small visual field of gaze. A stereoscopic perception is obtained.

For example, when an object 10 m ahead is viewed naturally, a range of 28 cm in diameter can be seen at a viewing angle of 1.6 degrees, a viewpoint movement distance due to a microscopic fixation is 1.78 cm, and a depth movement distance is about It will be 3m. In addition, when an object 3 m ahead is viewed naturally, a range of 8.4 cm in diameter can be seen at the same viewing angle, the viewpoint movement distance due to the involuntary eye movement is 0.52 cm, and the depth movement distance is about It will be 26 cm. When this is converted into pixels and converted into moving speed, the motion parallax is obtained. The speed of the involuntary eye movement is about 10 degrees/second. The human brain ignores when there is no change in the input information and starts information processing using the change in information as a trigger, so changes in the constituent elements of each viewpoint image that come in and go out with the involuntary eye movement are fixed. Capture and perceive a sense of depth and depth.

The stereoscopic image display device according to the present embodiment embodies movement of a gaze target generated by fixational eye movements in natural vision on an image display panel, and includes a plurality of (at least three) viewpoint images in the horizontal direction. It is realized by arranging at equal intervals. It should be noted that allocating the left and right eye images to the viewpoint images at both ends does not result in a simulation of natural vision.

In a twin-lens system such as a lenticular system or a parallax barrier system (control of emission of light rays), stereoscopic vision is realized by a convergence angle obtained by distributing and displaying images for the left and right eyes. However, the conventionally known naked-eye type stereoscopic image display device has a drawback in that stereoscopic perception cannot be performed well by light ray reproduction because the focus is placed in the direction of light.

On the other hand, the stereoscopic image display apparatus according to the present embodiment realizes a stereoscopic effect and a sense of depth by using a method different from the twin-lens type, and although the stereoscopic effect is weaker than that of the twin-lens type, the stereoscopic image display is not natural. It has the characteristic of being close. The ray reproduction in the present embodiment can sufficiently simulate natural vision, and for example, the change in the viewpoint caused by the involuntary eye movement is 3 to 5 stereoscopic pixels, and the change is the viewpoint in the object. It corresponds to about 0.52 to 1.78 cm when converted to the moving distance, and the light is convoluted within this change.

The stereoscopic image display device according to the present embodiment determines a viewpoint in either the left or right image frame and generates a plurality of viewpoint images. At this time, the ray reproduction method works by keeping a large number of rays emitted from each point in the image within the range of the viewing angle of 1 to 2 degrees. Natural stereoscopic vision is possible even when the value changes, or when a plurality of observers simultaneously observe the stereoscopic image.

INDUSTRIAL APPLICABILITY The present invention can be used in a stereoscopic image display device, particularly in a naked-eye type stereoscopic image display device using a light ray reproduction method.

1... Distributor, 2, 3... Frame memory, 4, 5... Column summation calculation unit, 6... Column summation comparison unit, 7... Parallax matrix generation unit, 8... Viewpoint image generation unit, 9... Image memory, 10... Weighting Function generating unit, 11... Image signal converting unit, 12... Frame selecting unit, 100... Image display panel, 200... Image processing device, 300... Display control device

Claims (10)

An image display panel configured by arranging a plurality of stereoscopic pixels each including m (m is an integer of 3 or more) pixels in the horizontal direction and the vertical direction,
Depth information for each pixel is obtained from the image frame for the right eye and the image frame for the left eye of the image signal, and based on the depth information, m viewpoint image frames for horizontally expanding and displaying the pixel image are generated. An image processing device that
A display control device for displaying m viewpoint images represented by the m viewpoint image frames on the m pixels in a plurality of stereoscopic pixels of the image display panel;
A stereoscopic image display device. The stereoscopic image display device according to claim 1, wherein, in each stereoscopic pixel of the image display panel, the m pixels are arranged such that optical centers of gravity are arranged in a horizontal direction. The stereoscopic image display apparatus according to claim 1, wherein each of the m pixels includes a plurality of sub-pixels that display three primary colors of light. The stereoscopic image display device according to claim 3, wherein the plurality of sub-pixels are arranged in a plurality of adjacent lines in the image display panel. The stereoscopic image display device according to claim 3, wherein the plurality of sub-pixels are arranged in a plurality of lines in every other line in the image display panel. The three-dimensional image display device according to claim 3, wherein in the image display panel, a plurality of sub-pixels that display the same color are arranged so as to be aligned in the vertical direction. The three-dimensional image display device according to claim 3, wherein in the image display panel, a plurality of sub-pixels that display the same color are arranged side by side in the horizontal direction. The image processing device stores the right-eye image frame and the left-eye image frame, obtains a column sum by vertically accumulating pixel values of each frame for each column, and a column sum between both frames. After identifying the overlapping area in the images of both frames by comparing, adjusting the positional relationship of the images of both frames, determine the depth information for each pixel based on the displacement of the corresponding pixel image in both frames, 8. The m viewpoint image frames including the one frame are generated by moving a pixel image in the one frame in the horizontal direction based on the weighted depth information. The three-dimensional image display device according to the item. The image processing device selects a standard frame and a reference frame having a predetermined number of frame differences with respect to the standard frame from a frame sequence of a two-dimensional image signal, and based on the standard frame and the reference frame, The stereoscopic image display device according to any one of claims 1 to 7, which generates an image signal having an image frame for the right eye and an image frame for the left eye. The stereoscopic image according to any one of claims 1 to 7, wherein the display control device allocates the m viewpoint images to the m pixels in a plurality of stereoscopic pixels of the image display panel for display. Display device.

Images