JP2015029215A - Stereoscopic image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereoscopic image processing device capable of reducing a processing load for displaying a stereoscopic image having high quality, or a stereoscopic image display device comprising the stereoscopic image processing device.SOLUTION: A stereoscopic image processing device includes: means for acquiring parallax image data to which pixel deviation information is added; information acquisition means for reading the pixel deviation information added to the parallax image data; relative parallax amount calculation means for calculating a relative parallax amount from a display condition and the pixel deviation information acquired by the information acquisition means; fusion limit determination means for performing fusion limit determination on whether or not an observer can perform stereoscopic view of a parallax image, from the relative parallax amount; and stereoscopic image processing means for executing image processing on the parallax image according to a result of the determination by the fusion limit determination means.

Description

  The present invention relates to a stereoscopic image processing apparatus capable of displaying a stereoscopic image using a plurality of images captured with parallax with respect to the same subject or a stereoscopic image display apparatus having the same.

  In order to capture a stereoscopic image, it is necessary to capture images from a plurality of locations separated from each other, that is, images having parallax with respect to the same subject. Conventionally, as a generally-known stereoscopic image capturing method, there is a method of acquiring two subject images (parallax images) from two different locations by using two cameras that are spaced apart from each other. Also known is a method of acquiring the parallax image in a time-division manner using a liquid crystal shutter or the like by a single imaging device provided with a single imaging device.

  As a means for displaying the parallax image in a three-dimensional manner, for example, a method of displaying the right-eye image and the left-eye image on one screen in a time-sharing manner and using liquid crystal shutter glasses synchronized with the screen time-sharing is used. is there. In addition, the left and right eye images are simultaneously displayed on the screen with polarized light beams in directions orthogonal to each other, and the polarizing filters configured to be orthogonal to each other are viewed using polarized glasses corresponding to the left and right eyes. Etc. are known.

  By the way, in such a stereoscopic image display, there may be a situation (a fusion limit) in which the left and right eye parallax images are perceived as a double image due to the display principle and human visual characteristics, and the observer cannot stereoscopically view. Are known. Although this fusion limit will be described in detail later, it has been found that when a display that is perceived by the observer as a double image is given, the observer feels a strong sense of discomfort, fatigue, and discomfort.

  In order to improve the above-mentioned fusion limit, in the conventional stereoscopic image pickup device, shooting is performed assuming the size of the display screen and the viewing distance for displaying the stereoscopic image at the shooting site so that the photographer should not become the fusion limit. The person controls the baseline length and the convergence angle according to the subject distance. Further, in Patent Document 1, the display device and the imaging device are integrated, the amount of parallax is calculated from the parallax image obtained on the imaging device side, and the display conditions of the display device on which the stereoscopic image is actually displayed are used. The reproduction depth position of the stereoscopic image is calculated from the parallax amount and the display condition. It is determined whether or not the stereoscopic image displayed from the reproduction depth position information is within the fusion limit range of the observer. In addition, when the stereoscopic image is outside the fusion limit range, the improvement has been made by adjusting the baseline length and the convergence angle on the imaging device side.

  However, the fusion limit range varies greatly depending on the display screen size of the display device and the viewing distance of the observer. Therefore, even if the base line length and the convergence angle on the imaging device side are controlled assuming a certain display condition as in the prior art, the fusion limit range is set in the case of a display screen size or viewing distance that is significantly different from the assumed display condition. Beyond that, it gives fatigue and discomfort to the observer. The control method proposed in Patent Document 1 is also effective in the case of a display device integrated with an imaging device as a system. However, a parallax image captured by the system is displayed on another display device. When observing with the above, the same problem as in the prior art occurs.

  Further, the amount of parallax is calculated for each input image as proposed in Patent Document 1, and the fusion limit range is displayed in the image using the display conditions of the display device on which the stereoscopic image is actually displayed. Determining whether or not it exists is considered to increase the processing load. In particular, when displaying a 3D moving image that displays 60 frames or more of image data per second, performing the above-described determination on the frame-by-frame display device side increases the processing load. For this reason, it is difficult to further increase the frame rate of the moving image, and it is difficult to express the motion of the subject smoothly.

JP 07-167633 A

  Accordingly, an object of the present invention is to provide a stereoscopic image processing apparatus that can reduce a processing load for displaying a high-quality stereoscopic image, or a stereoscopic image display apparatus including the same.

Means for acquiring parallax image data to which pixel shift information is added;
Comprising information acquisition means for reading the pixel shift information added to the parallax image data;
A relative parallax amount calculating means for calculating a relative parallax amount from the pixel shift information acquired by the information acquiring means and the display condition;
Fusion limit determination means for performing a fusion limit determination as to whether or not the observer can stereoscopically view the parallax image from the relative parallax amount;
The image processing apparatus includes a stereoscopic image processing unit that performs image processing on the parallax image according to a determination result of the fusion limit determination unit.

  ADVANTAGE OF THE INVENTION According to this invention, the stereo image processing apparatus which can reduce the processing load with respect to the display of a high quality stereo image, or a stereo image display apparatus provided with the same can be provided.

Configuration diagram of stereoscopic image processing apparatus according to Embodiment 1 Flow chart of processing according to embodiment 1 Configuration diagram of stereoscopic image processing apparatus according to embodiment 2 Flow chart of processing according to embodiment 2 Configuration diagram of stereoscopic image processing apparatus according to Embodiment 3 Flow chart of processing according to embodiment 3 3D image shooting model illustration Illustration of a 3D image display model Explanatory drawing about offset control of stereoscopic image display model Illustration of corresponding point extraction method A diagram showing an example of the structure of a stereoscopic image data file

  The main viewpoint of the present invention is to improve the problem of reducing the processing load for displaying a high-quality stereoscopic image in the stereoscopic image processing apparatus.

  First, the principle of the stereoscopic image capturing system, the display system, and the fusion limit will be described. FIG. 7 is a diagram for explaining a model of the above-described conventional stereoscopic image photographing method. The coordinates are the center of the left and right cameras, the x axis in the horizontal direction, and the y axis in the depth direction. The height direction is omitted for simplification. Conventionally, a method of controlling the convergence angle (tilting the optical axis of the camera) has also been proposed. Here, for the sake of simplicity of explanation, the principle of the parallel photographing method in which the optical axes of the left and right cameras are parallel will be described. The same geometric theory holds for the method for controlling the convergence angle by considering the distance to the convergence point.

  Assume that the principal points of the imaging optical systems of the left and right cameras are arranged at (−Wc, 0) and (Wc, 0), respectively. Here, the focal length of the imaging optical system of the left and right cameras is assumed to be f. In this state, when the subject A on the y axis (0, y1) is photographed by the respective cameras, the displacement amount of the subject image A from the sensor center of the left and right cameras is taken as the parallax, and Plc and Prc are expressed by the following equations, respectively. I can do it.

  By capturing the same subject from different viewpoints based on the above principle, it is possible to acquire right and left parallax images each having a shift amount represented by the above formulas (1) and (2) in the horizontal direction.

  Next, depth perception and fusion limit will be described in detail using a display system model. FIG. 8 is a diagram for explaining a model of the above-described conventional stereoscopic image display method. The coordinates are the center of both eyes of the observer as the origin, the x axis in the horizontal direction, and the y axis in the depth direction. The height direction is omitted for simplification. Here, the viewing distance from the observer to the display screen 100 is ds, the left and right eyes are positioned at (−We, 0) and (We, 0), respectively, and the right-eye position of the point-parallax image of the subject A to be reproduced Is (Pr, ds) and the left eye is (Pl, ds).

  In this case, the observer has a line segment connecting the right eye (We, 0) and the right eye image (Pr, ds) and a line segment connecting the left eye (-We, 0) and the right eye image (Pl, ds). It is perceived that the subject image A is at the intersection (0, y2). In this case, the difference (y2−ds) between the display screen 100 in the depth direction and the intersection is the distance at which the observer perceives the subject image A in the depth direction from the display screen. Here, y2 is expressed by the above coordinates.

It can be expressed as

  In other words, the distance in the depth direction is expressed in terms of the relative parallax (α−β) between the subject image A (0, y2) and the display screen (0, ds), and this size is represented by the display screen and the subject image. This corresponds to the relative depth direction distance of A. From various previous studies, it is known that humans perceive the position in the depth direction by calculating the difference of this angle in the brain. Here, the absolute value of relative parallax | α−β |

It can be seen that the relative parallax is an amount depending on the viewing distance ds and the stereoscopic image position y2. Furthermore, from equation (3), y2 is an amount that depends on the amount of parallax, which is the difference between the display positions Pl and Pr of the left and right parallax images, and the viewing distance ds. It can be seen that the amount depends on the difference between Pl and Pr) and the viewing distance ds.

  Next, the fusion limit will be described. In the conventional stereoscopic image display method described above, an actual image is displayed on the display screen 100, but the observer perceives that the subject A is at the position y2. That is, in the stereoscopic image that looks like an actual image, the focus position of the eye is different. In other words, the eye convergence of the observer (a state in which both eyes are directed to the intersection A in a cross-like shape) and adjustment (a state in which the eyes are focused on the display screen) are different. When this difference increases, the observer cannot perceive the left and right parallax images as one, and perceives it as a double image. Moreover, even if it does not become a double image, it will be in the state where discomfort and a feeling of fatigue will increase.

  For example, when an observer is gazing at the center of the display screen 100 in FIG. 8, the absolute value of relative parallax that can perceive a stereoscopic image comfortably is about 1 degree, and outside the fusion limit range where a double image is seen is about 2 degrees. That's it.

(Concept of stereoscopic image processing apparatus in the present invention)
In the above, the shooting, display, and fusion limit of a stereoscopic image based on stereo shooting have been described. Here, the concept of the stereoscopic image processing apparatus in the present invention will be described.

  Considering a normal two-dimensional image, there are very many situations where the number of pixels on the imaging device side and the number of pixels on the display device side are not equal. In particular, in the case of still images, an ordinary camera has an image sensor with 10 million pixels or more, while the display side has various specifications such as VGA 300,000 pixels to Full HD 2 million pixels, and recently 8 million pixels called 4K2K. Exists. In general, when there is a difference in the number of pixels, the pixel number conversion process such as down-sampling or up-sampling is performed to meet the specifications of various display devices. In addition, there are various display sizes ranging from several inch sizes such as mobile phone screens to large screen sizes exceeding 100 inches by projectors. The display size can be easily dealt with by changing the display magnification M when the number of pixels of the image data and the display device is the same.

  In the stereoscopic image display, it is necessary to be able to display a high-quality stereoscopic image under various display conditions as in the conventional two-dimensional image. Here, the high quality in the stereoscopic image display is a stereoscopic image display without discomfort and fatigue. However, from the expressions (3) and (4) indicating the depth position of the stereoscopic image described above, the depth distance perceived by the observer is an amount depending on the parallax amount (difference between Pl and Pr) and the viewing distance ds. I understand that. That is, for example, even when the same stereoscopic image data is displayed, the amount of parallax varies greatly depending on the display screen size.

  Furthermore, what viewing distance the observer observes is completely unknown. In other words, the fusion limit range cannot be calculated until the display size and viewing distance, which are the viewing conditions on the display side, are known, and it is impossible to obtain image data corresponding to various display conditions by only controlling the imaging device. It can be seen that it is.

  Therefore, it is assumed that offset control is performed on the display device side so that images are translated and displayed in the horizontal direction with respect to the left and right parallax images based on arbitrary stereoscopic image data. The offset control will be described with reference to FIG. The coordinates are the center of both eyes of the observer as the origin, the x axis in the horizontal direction, and the y axis in the depth direction. The height direction is omitted for simplification.

  Here, the viewing distance from the observer to the display screen 100 is ds, the left and right eyes are positioned at (−We, 0) and (We, 0), respectively, and the right side of the point parallax images Irs and Ils of the reproduced subject As. The eye position is (Pr + S, ds), and the left eye position is (Pl + S, ds). Here, S is an offset amount (a parallel movement amount of the left and right parallax images in the horizontal direction). In this case, the observer connects the line segment connecting the right eye (We, 0) and the right eye image (Pr-S, ds) to the left eye (-We, 0) and the right eye image (Pl + S, ds). It is perceived that the subject image As exists at the intersection (0, y3) of the line segment. Here, if y3 is expressed by the above coordinates,

It can be expressed as

  Here, FIG. 9 shows the stereoscopic image position when the offset amount S is given in FIG. 8, and shows that the stereoscopic image position is shifted in the direction of the display screen 100 by (y2−y3) by the offset amount S. ing. Here, FIG. 9 shows only the subject As. However, when there are other subjects in the image, the offset control affects the entire stereoscopic image display surface, and all the stereoscopic image positions are approximately (y2). Shift by -y3).

  From the above, if subject A in FIG. 8 is outside the fusion limit range of the observer, it can be controlled as subject image As within the fusion limit range by giving offset amount S. That is, if the display device is provided with a sensor for detecting the position of the observer and the control mechanism, the fusion limit can be dealt with even under various display conditions with different display sizes and viewing distances. However, since it is not always necessary to perform the above-described offset control on all stereoscopic image data, it has a determination means for determining whether or not to perform fusion limit countermeasure processing on stereoscopic image data in order to reduce processing load. is important. Here, when a sensor for detecting the position of the observer is provided on the display device side, information necessary for calculating relative parallax for determining whether or not there is a subject outside the fusion limit range in the stereoscopic image data Is only the maximum parallax amount | Pl−Pr | max.

  Although the difference between the parallax amounts Pl and Pr is described as a distance in the above description of the principle, it is clear that it is expressed as a pixel shift amount on display on an actual display device. On the imaging device side, the shooting parallaxes Plc and Prc shown in the expressions (1) and (2) are reproduced as the pixel shift amount of the stereoscopic image data. Here, it is assumed that the maximum pixel shift amount of the stereoscopic image data output from the imaging apparatus is | Plc-Prc | max. Further, when the horizontal recording pixel number on the imaging device side is Hc, the display pixel number on the display device side is Hm, and the pixel pitch of the display device is T, the maximum parallax amount | Pl−Pr | max is given by 6).

  Here, a method of calculating the maximum pixel shift amount on the imaging apparatus side will be briefly described. One of the acquired left and right parallax images is selected as a reference image for calculating a pixel shift amount.

  Here, the image for the left eye is selected as a reference image. Next, with respect to the selected standard image for the left eye, the image for the right eye is used as a reference image, and corresponding pixels are detected. Here, the corresponding pixels are, for example, the respective pixels corresponding to the same subject A in the left and right parallax image data obtained for the point image subject A in the shooting model. The corresponding point extraction method will be described in detail.

  Here, the image coordinates (X, Y) are used. The image coordinates (X, Y) are defined with the upper left corner of each pixel group in FIG. 10 as the origin, the horizontal direction being the X axis, and the vertical direction being the Y axis. In addition, the luminance of the image coordinates (X, Y) of the standard image data 301 will be described as F1 (X, Y), and the luminance of the reference image data 302 will be described as F2 (X, Y).

  Pixels of reference image data (referred to as vertical line pixels within 302 in FIG. 10) corresponding to arbitrary coordinates (X, Y) (referred to as vertical line pixels within 301 in FIG. 10) in the reference image data are coordinates (X, Y). It can be obtained by searching for the luminance of the reference image data most similar to the luminance F1 (X, Y) of the standard image data in Y). However, in general, it is difficult to search for a pixel that is most similar to an arbitrary pixel. Therefore, a pixel near the image coordinates (X, Y) is also used to search for a similar pixel by a technique called block matching.

For example, a block matching process when the block size is 3 will be described. The luminance values of a total of three pixels, that is, a pixel at an arbitrary coordinate (X, Y) of the reference image data and two pixels before and after (X-1, Y) and (X + 1, Y) are respectively
F1 (X, Y), F1 (X-1, Y), F1 (X + 1, Y)
It becomes.

On the other hand, the luminance values of the pixels of the reference image data shifted from the coordinates (X, Y) by k in the X direction are respectively
F2 (X + k, Y), F2 (X + k-1, Y), F2 (X + k + 1, Y)
It becomes.

  In this case, the similarity E with the pixel at the coordinates (X, Y) of the reference image data is defined by the following equation (7).

  In this equation (7), the value of the similarity E is calculated by successively changing the value of k, and (X + k, Y) giving the smallest similarity E is a corresponding point with respect to the coordinates (X, Y) of the reference image data It is.

  Next, a pixel shift amount is calculated for each corresponding point extracted above. As a calculation method, the pixel position difference is calculated from each pixel of the reference image data corresponding to each pixel of the standard image data obtained by the block matching method. The maximum pixel shift amount kmax is detected from the pixel shift amount for each pixel of the reference image data calculated above, and the maximum pixel shift amount kmax in the left-right parallax image obtained above is detected in the file header of the stereoscopic image data file. Can be recorded.

  Here, FIG. 11 is a diagram illustrating a configuration example of a stereoscopic image data file. The stereoscopic image data file 1000 includes a file header portion 1001 and an image data portion 1003. As shown in FIG. 11, the above-mentioned maximum pixel shift amount kmax information 1002 is embedded in the file header portion 1001, and the left and right parallax images are embedded in the image data portion 1003 as stereoscopic image data. In addition, various data such as the number of photographic pixels, the F value, and the focal length, which are normal photographic information, can be simultaneously recorded in the file header portion.

  If simple fusion limit determination can be performed on the processing device or display device side using information on the number of recorded pixels and the maximum pixel shift amount on the imaging device side as described above, the processing load during stereoscopic image display is reduced. Therefore, it is possible to realize a high frame rate of a stereoscopic moving image.

  Therefore, in the present invention, a stereoscopic image data file to which the maximum pixel shift amount information or the maximum pixel shift ratio information of the left and right parallax images is added is acquired, and whether or not a subject outside the fusion limit range exists in the stereoscopic image data file. A three-dimensional image processing apparatus capable of efficiently determining is proposed.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Example 1]
Hereinafter, a stereoscopic image processing apparatus according to a first embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of a stereoscopic image processing apparatus according to the first embodiment. The image pickup means 101 includes an imaging optical system (not shown) and an image pickup device that take right and left parallax images. Furthermore, in this embodiment, the maximum pixel shift amount kmax in the left-right parallax image is recorded in the file header portion 1001 of the stereoscopic image data file 1000 output from the imaging means 101 as shown in FIG. The image data acquisition unit 10 acquires a stereoscopic image data file from the imaging unit 101. The data information acquisition unit 20 acquires data included in the file header unit 1001. The display condition acquisition unit 30 acquires display condition information of the display unit 201.

  The maximum relative parallax amount calculation unit 40 calculates the maximum relative parallax amount in the image. The fusion limit determination unit 50 determines whether or not a fusion limit range is included in the image. The stereoscopic image processing unit 60 calculates the relative parallax amount in the image for calculating the relative parallax amount of the entire image data, the fusion limit region determination unit 62 for determining the fusion limit region in the image data, and adjusts the relative parallax amount of the image data And a relative parallax amount adjustment processing unit 63 for performing the processing. The display unit 201 displays the image data output from the image processing apparatus 1.

  Next, the fusion limit determination processing operation of the stereoscopic image processing apparatus of the present embodiment will be described in detail using the flowchart of FIG. First, the image data acquisition unit 10 acquires the stereoscopic image data file 1000 from the imaging unit 101. The data acquisition method may be direct connection using a USB cable (not shown) or wireless connection using radio waves or infrared rays.

  Next, the data information acquisition unit 20 acquires the maximum pixel shift amount and the shooting information included in the file header unit 1001 in the stereoscopic image data file 1000 acquired by the image data acquisition unit 10. Next, the display condition acquisition unit 30 acquires display condition information from the display unit 201. Here, the display condition is information regarding the number of display pixels, the display size, and the viewing distance. The display condition acquisition method may be directly connected by a cable or the like (not shown), or may be wirelessly connected using radio waves or infrared rays. Next, the maximum relative parallax amount calculation unit 40 calculates the maximum relative parallax amount on the display device side from the above equation (6) using the maximum pixel shift amount, the shooting information, and the display conditions.

  Next, the fusion limit determination unit 50 determines whether or not the stereoscopic image data input from the calculated maximum relative parallax amount includes an image region outside the fusion limit range. Here, for example, it is determined that an area outside the fusion limit range is included when the double image looks like the above-described double image, and conversely, an area outside the fusion limit range is included when it is 2 degrees or less. Judge that it is not.

  Next, the operation in the case where the fusion limit determination unit 50 includes an image area outside the fusion limit range in the stereoscopic image data, that is, in the case of YES determination in the flowchart of FIG. 2 will be described. In the case of YES determination, the stereoscopic image data file 1000 is sent to the stereoscopic image processing unit 60. Next, the relative parallax amount calculation unit 61 calculates the relative parallax amount for the entire image data in the stereoscopic image data file. As a calculation method, for example, a method using the block matching method described above can be used.

  Next, the fusion limit area determination unit 62 detects an area outside the fusion limit range in the image data from the relative parallax amount with respect to the entire image data. Next, the relative parallax adjustment processing unit 63 performs desired image processing on the stereoscopic image data so that the region outside the fusion limit range falls within the fusion limit range. In this embodiment, it is assumed that offset control is performed so that images are translated and displayed in the horizontal direction with respect to the above-described left and right parallax images. As described above, the stereoscopic image data processing in the stereoscopic image processing unit 60 is completed, and the processed stereoscopic image data is transferred to the display unit 201 and displayed as a stereoscopic image.

  Next, the operation in the case where the fusion limit determination unit 50 does not include an image area outside the fusion limit range in the stereoscopic image data, that is, in the case of NO determination in the flowchart of FIG. 2 will be described. In the case of NO determination, the stereoscopic image data file 1000 does not include an image area outside the fusion limit range, that is, the image data can be displayed as a stereoscopic image without discomfort and fatigue. Therefore, since it is not necessary to perform harmful processing as stereoscopic image data, the stereoscopic image data can be transferred and displayed on the display unit 201 as it is.

  As described above, by using the information of the maximum pixel shift amount kmax = | Plc-Prc | max added to the file header part, the maximum parallax amount can be easily obtained on the processing device side from the equation (6) as described above. Can be calculated. Therefore, it is possible to easily execute the fusion limit determination, and the processing load on the processing apparatus side can be greatly reduced. In particular, when displaying stereoscopic video images that display 60 frames or more of image data per second, it is extremely processing-intensive to perform relative parallax calculation for the entire image data on the frame-by-frame display device side. growing.

  Therefore, as in the configuration of the present embodiment, if the stereoscopic image processing apparatus includes the fusion limit determination unit, it is not necessary to perform unnecessary fusion limit subject processing on the processing apparatus side. Further, for example, by using the maximum pixel shift amount information added to the file header portion of the entire stereoscopic video data file, it becomes possible to avoid processing for all data when the above fusion limit condition is satisfied. A significant processing load reduction effect can be obtained. In addition, in the case of a stereoscopic video data file, by using the maximum pixel shift amount information for each chapter with different scenes, processing determination on the processing device side can be performed for each scene change, and a higher quality stereoscopic video can be obtained. Display is possible.

  Here, in the present embodiment, it is assumed that the file header portion has the maximum pixel shift amount kmax information. However, the maximum pixel shift amount is not necessarily required. For example, in the range of about 0.8 × kmax to 1.2 × kmax. It can be changed to information k regarding any corresponding pixel. If the pixel misalignment information in the file header is close to 0.8 × kmax, the fusion limit criterion becomes stricter and can be displayed as a stereoscopic image with no discomfort and fatigue. However, when the pixel shift information is smaller than 0.8 × kmax, the number of stereoscopic image data frames determined to be processed increases, and the processing load reduction effect in the present invention cannot be obtained.

  If the pixel shift information in the file header portion is close to 1.2 × kmax, the fusion limit criterion is relaxed, and the number of image frames to be processed is reduced, so that it is possible to reduce processing addition. However, when the pixel shift information is larger than 1.2 × kmax, the number of stereoscopic image data frames that feel uncomfortable and fatigued increases, and high-quality stereoscopic image display becomes difficult.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

[Example 2]
Hereinafter, a stereoscopic image processing apparatus according to a second embodiment of the present invention will be described with reference to the drawings.

  FIG. 3 is a configuration diagram of the stereoscopic image processing apparatus according to the second embodiment. The image pickup means 101 includes an imaging optical system (not shown) and an image pickup device that take right and left parallax images. Furthermore, in the present embodiment, the maximum pixel shift ratio rmax in the left-right parallax image is recorded in the file header portion of the stereoscopic image data file output from the imaging means 101. Here, the maximum pixel shift ratio rmax is a value corresponding to | Plc−Prc | max / Hc in Expression (6). In addition, the stereoscopic image processing unit 60 includes an intra-image relative parallax amount calculating unit 61 that calculates the relative parallax amount of the entire image data, a fusion limit region determining unit 62 that determines a fusion limit region in the image data, and image processing on the fusion limit region. A fusion limit region image processing unit 64 for performing Since other configurations are the same as those of the first embodiment, the description thereof is omitted.

  Next, the fusion limit determination processing operation of the stereoscopic image processing apparatus according to the present embodiment will be described in detail with reference to the flowchart of FIG. First, the image data acquisition unit 10 acquires a stereoscopic image data file from the imaging unit 101. The data acquisition method may be direct connection using a USB cable (not shown) or wireless connection using radio waves or infrared rays.

  Next, the data information acquisition unit 20 acquires the maximum pixel shift ratio and the shooting information included in the file header portion in the stereoscopic image data file acquired by the image data acquisition unit 10. Next, the display condition acquisition unit 30 acquires display condition information from the display unit 201. Here, the display condition is information regarding the number of display pixels, the display size, and the viewing distance. The display condition acquisition method may be directly connected by a cable or the like (not shown), or may be wirelessly connected using radio waves or infrared rays. Next, the maximum relative parallax amount calculation unit 40 calculates the maximum relative parallax amount on the display device side from the above-described equation (6) using the maximum pixel shift ratio, shooting information, and display conditions.

  Next, the fusion limit determination unit 50 determines whether or not the stereoscopic image data input from the calculated maximum relative parallax amount includes an image region outside the fusion limit range. Here, for example, it is determined that an area outside the fusion limit range is included when the double image looks like the above-described double image, and conversely, an area outside the fusion limit range is included when it is 2 degrees or less. Judge that it is not.

  Next, the operation in the case where the fusion limit determination unit 50 includes an image region outside the fusion limit range in the stereoscopic image data, that is, in the case of YES determination in the flowchart of FIG. 4 will be described. In the case of YES determination, the stereoscopic image data file is sent to the stereoscopic image processing unit 60. Next, the relative parallax amount calculation unit 61 calculates the relative parallax amount for the entire image data in the stereoscopic image data file.

  As a calculation method, for example, a method using the block matching method described above can be used. Next, the fusion limit area determination unit 62 detects an area outside the fusion limit range in the image data from the relative parallax amount with respect to the entire image data. Next, the fusion limit area image processing unit 64 performs desired image processing on the stereoscopic image data so that the observer does not recognize the area outside the fusion limit range. In this embodiment, the blur addition control is performed on the region outside the fusion limit range of the left and right parallax images. By adding the blur, the observer does not pay attention to the region outside the fusion limit range, so that the discomfort and fatigue of the observer can be alleviated.

  As described above, the stereoscopic image data processing in the stereoscopic image processing unit 60 is completed, and the processed stereoscopic image data is transferred to the display unit 201 and displayed as a stereoscopic image.

  Next, the operation when the fusion limit determination unit 50 does not include an image area outside the fusion limit range in the stereoscopic image data, that is, when NO is determined in the flowchart of FIG. 4 will be described. In the case of NO determination, the stereoscopic image data file does not include an image region outside the fusion limit range, that is, the image data can be displayed as a stereoscopic image without discomfort and fatigue. Therefore, since it is not necessary to perform harmful processing as stereoscopic image data, the stereoscopic image data can be transferred and displayed on the display unit 201 as it is.

  As described above, by using the information of the maximum pixel shift ratio | Plc-Prc | max / Hc added to the file header part, the maximum amount of parallax can be easily obtained on the processing apparatus side from the equation (6) as described above. Can be calculated. Therefore, it is possible to easily execute the fusion limit determination, and the processing load on the processing apparatus side can be greatly reduced. In particular, when displaying stereoscopic video images that display 60 frames or more of image data per second, it is extremely processing-intensive to perform relative parallax calculation for the entire image data on the frame-by-frame display device side. growing.

  Therefore, as in the configuration of the present embodiment, if the stereoscopic image processing apparatus includes the fusion limit determination unit, it is not necessary to perform unnecessary fusion limit subject processing on the processing apparatus side. Further, for example, by using the maximum pixel shift ratio information added to the file header portion of the entire stereoscopic video data file, it becomes possible to avoid processing for all data when the above fusion limit condition is satisfied. A significant processing load reduction effect can be obtained. In addition, in the case of a stereoscopic video data file, processing determination on the processing device side can be performed for each scene change by using the maximum pixel shift ratio information for each chapter that has different scenes, so that a higher quality stereoscopic video can be obtained. Display is possible.

  Here, in this embodiment, it is assumed that the file header portion has the maximum pixel shift ratio rmax information. However, it is not always necessary to have the maximum pixel shift ratio, for example, in the range of about 0.8 × rmax to 1.2 × rmax. It is possible to change to information on any corresponding pixel. If the pixel misalignment information in the file header is close to 0.8 × rmax, the fusion limit criterion becomes stricter and can be displayed as a stereoscopic image with less discomfort and fatigue. However, if the pixel shift information is smaller than 0.8 × rmax, the number of stereoscopic image data frames determined to be processed increases, and the processing load reduction effect in the present invention cannot be obtained.

  If the pixel shift information in the file header portion is close to 1.2 × rmax, the fusion limit criterion is relaxed, and the number of image frames to be processed is reduced, so that it is possible to reduce processing addition. However, when the pixel shift information is larger than 1.2 × rmax, the number of stereoscopic image data frames that feel uncomfortable and fatigued increases, and it becomes difficult to display a high-quality stereoscopic image.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

[Example 3]
A stereoscopic image display device according to a third embodiment of the present invention will be described below with reference to the drawings.

  FIG. 5 is a configuration diagram of the stereoscopic image display apparatus 2 according to the third embodiment. For example, the display means 201 has a method of displaying a right-eye image and a left-eye image on a single screen in a time division manner and using a liquid crystal shutter glasses synchronized with the screen time division. In addition, the left and right eye images are simultaneously displayed on the screen with polarized light beams in directions orthogonal to each other, and the polarizing filters configured to be orthogonal to each other are viewed using polarized glasses corresponding to the left and right eyes. Can be used.

  Further, the display unit 201 includes an image processing unit 70, a fusion limit determination level adjustment unit 80, and a viewing distance calculation unit 90. The image processing unit 70 performs image processing such as edge enhancement and color correction performed on a general two-dimensional image or moving image by a conventional TV apparatus or the like. Further, the fusion limit determination level adjustment unit 80 sets a desired fusion limit determination level by the operation of the observer. Further, the viewing distance calculation unit 90 calculates a viewing distance that is a distance from the display screen of the observer. Since other configurations are the same as those of the first embodiment, the description thereof is omitted.

  Next, the fusion limit determination processing operation of the stereoscopic image processing apparatus according to the present embodiment will be described in detail with reference to the flowchart of FIG. First, the image data acquisition unit 10 acquires a stereoscopic image data file from the imaging unit 101. The data acquisition method may be direct connection using a USB cable (not shown) or wireless connection using radio waves or infrared rays. Next, the data information acquisition unit 20 acquires the maximum pixel shift amount and the shooting information included in the file header portion in the stereoscopic image data file acquired by the image data acquisition unit 10.

  Next, the display condition acquisition unit 30 acquires display condition information from the display unit 201. Here, the display conditions are information regarding the number of display pixels, the display size, the viewing distance sent from the viewing distance calculation unit 90, and the determination level sent from the fusion limit determination level adjustment unit 80. The display condition acquisition method may be directly connected by a cable or the like (not shown), or may be wirelessly connected using radio waves or infrared rays. In addition, for obtaining the viewing distance information of the observer by the viewing distance calculation unit 90, a method using infrared reflection conventionally used for distance measurement can be used.

  Next, the maximum relative parallax amount calculation unit 40 calculates the maximum relative parallax amount on the display device side from the above equation (6) using the maximum pixel shift amount, the shooting information, and the display conditions. Next, the fusion limit determination unit 50 determines whether or not the stereoscopic image data input from the calculated maximum relative parallax amount includes an image region outside the fusion limit range. Here, for example, the above-described relative parallax amount of 2 degrees is set as a reference determination amount, and information on the determination level sent from the fusion limit determination level adjustment unit 80 is a ratio to the reference determination amount. Specifically, the observer selects one level from a plurality of determination levels, and controls the determination level by multiplying the reference determination amount by 2 times.

  For example, if the level is 80%, 1.6 degrees, which is 80% of the reference determination amount of 2 degrees, is set as a new determination standard. The fusion limit determination unit 50 determines whether or not the stereoscopic image data includes an image area outside the fusion limit range according to the newly set determination criterion.

  Next, the operation in the case where the fusion limit determination unit 50 includes an image area outside the fusion limit range in the stereoscopic image data, that is, the determination in the case of YES in the flowchart of FIG. 6 will be described. In the case of YES determination, the stereoscopic image data file is sent to the stereoscopic image processing unit 60. Next, the relative parallax amount calculation unit 61 calculates the relative parallax amount for the entire image data in the stereoscopic image data file. As a calculation method, for example, a method using the block matching method described above can be used. Next, the fusion limit area determination unit 62 detects an area outside the fusion limit range in the image data from the relative parallax amount with respect to the entire image data.

  Next, the fusion limit area image processing unit 64 performs desired image processing on the stereoscopic image data so that the observer does not recognize the area outside the fusion limit range. In the present embodiment, the luminance adjustment control is performed on a region outside the fusion limit range of the left and right parallax images described above. By reducing the brightness of the region outside the fusion limit range, the viewer does not gaze at the region outside the fusion limit range, so that the viewer's discomfort and fatigue can be alleviated. As described above, the stereoscopic image data processing in the stereoscopic image processing unit 60 is completed, and the processed stereoscopic image data is transferred to the display unit 201.

  Next, the operation in the case where the fusion limit determination unit 50 does not include an image area outside the fusion limit range in the stereoscopic image data, that is, in the case of NO determination in the flowchart of FIG. 6 will be described. In the case of NO determination, the stereoscopic image data file does not include an image region outside the fusion limit range, that is, the image data can be displayed as a stereoscopic image without discomfort and fatigue. For this reason, since it is not necessary to perform harmful processing as stereoscopic image data, the stereoscopic image data is transferred to the display unit 201 as it is. Finally, the image processing unit 70 performs image processing such as edge enhancement and color correction performed by a conventional TV apparatus or the like on a general two-dimensional image or moving image, and then displays a stereoscopic image.

  As described above, by using the information of the maximum pixel shift amount | Plc-Prc | max added to the file header portion, the maximum parallax amount can be easily calculated on the display device side from the equation (6) as described above. It becomes possible to do. Therefore, it is possible to easily execute the fusion limit determination, and the processing load on the display device side can be greatly reduced. In particular, when displaying stereoscopic video images that display 60 frames or more of image data per second, it is extremely processing-intensive to perform relative parallax calculation for the entire image data on the frame-by-frame display device side. growing.

  Therefore, if the stereoscopic image display apparatus has the fusion limit determination unit as in the configuration of the present embodiment, it is not necessary to perform unnecessary fusion limit subject processing on the processing device side. Further, for example, by using the maximum pixel shift amount information added to the file header portion of the entire stereoscopic video data file, it becomes possible to avoid processing for all data when the above fusion limit condition is satisfied. A significant processing load reduction effect can be obtained. Also, in the case of a stereoscopic video data file, processing determination on the display device side can be performed for each scene change by using so-called maximum pixel shift amount information for each chapter with different scenes, so that a higher quality stereoscopic video can be obtained. Display is possible.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The present invention relates to a stereoscopic image processing apparatus capable of displaying a stereoscopic image using a plurality of images captured with parallax with respect to the same subject or a stereoscopic image display apparatus having the same.

1 Stereoscopic image processing device
2 Stereoscopic image display device
40 Maximum relative parallax calculator
50 Fusion limit judgment part
60 Stereoscopic image processing unit
101 Imaging means
201 Display means

Claims (9)

Means for acquiring parallax image data to which pixel shift information is added;
Comprising information acquisition means for reading the pixel shift information added to the parallax image data;
A relative parallax amount calculating means for calculating a relative parallax amount from the pixel shift information acquired by the information acquiring means and the display condition;
Fusion limit determination means for performing a fusion limit determination as to whether or not the observer can stereoscopically view the parallax image from the relative parallax amount;
A stereoscopic image processing apparatus, comprising: a stereoscopic image processing unit that performs image processing on the parallax image according to a determination result of the fusion limit determination unit. The pixel shift information is a pixel shift amount of an arbitrary pixel corresponding point of the parallax image added to the parallax image data,
When the pixel shift information is k and the maximum pixel shift amount in the parallax image data is kmax,
1.2kmax>k> 0.8kmax
The stereoscopic image processing apparatus according to claim 1, wherein: The pixel shift information is a pixel shift ratio of an arbitrary pixel corresponding point of the parallax image added to the parallax image data,
When the pixel shift information is r and the maximum pixel shift ratio in the parallax image data is rmax,
1.2rmax>r> 0.8rmax
The stereoscopic image processing apparatus according to claim 1, wherein: The stereoscopic image processing apparatus according to any one of claims 1 to 3, wherein a fusion limit determination criterion used by the fusion limit determination means can be controlled to increase or decrease by an observer. 5. The parallax amount control process for controlling a parallax amount of the parallax image data when the image processing unit determines that stereoscopic viewing is impossible by the fusion limit determination unit. The three-dimensional image processing apparatus of any one of Claims. 5. The blur amount control process for controlling a blur amount of the parallax image data when the image processing unit determines that stereoscopic viewing is impossible by the fusion limit determination unit. The three-dimensional image processing apparatus of any one of Claims. 5. The luminance control process for controlling the luminance of the parallax image data when the image processing unit determines that stereoscopic viewing is impossible by the fusion limit determination unit. The three-dimensional image processing apparatus according to item 1. Means for acquiring parallax image data to which pixel shift information is added;
Comprising information acquisition means for reading the pixel shift information added to the parallax image data;
A relative parallax amount calculating means for calculating a relative parallax amount from the pixel shift information acquired by the information acquiring means and the display condition;
Fusion limit determination means for performing a fusion limit determination as to whether or not the observer can stereoscopically view the parallax image from the relative parallax amount;
Stereoscopic image processing means for performing image processing on the parallax image according to the determination result of the fusion limit determination means;
A stereoscopic image display device comprising display means for displaying a stereoscopic image. The stereoscopic image display apparatus according to claim 7, further comprising a viewing distance calculation unit that calculates a viewing distance that is a distance from the display unit to an observer.