JP5323165B2 - Stereoscopic image processing apparatus and stereoscopic image processing method - Google Patents

Abstract

A three-dimensional image processing device pertaining to an embodiment of the present invention accepts an input three-dimensional image, calculates one or more parallax features in the three-dimensional image, and divides the three-dimensional image into multiple scenes according to the parallax features. In addition, the three-dimensional image processing device performs three-dimensional image correction processing on the basis of the scenes.

Description

  The present invention relates to a stereoscopic image processing apparatus and a stereoscopic image processing method.

  Humans have the ability to grasp space from the difference in visual information obtained by two eyes having a fixed interval. The shift of images obtained from different viewpoints by the left and right eyes is called parallax. Humans grasp the distance to an object using parallax as a clue. In fact, the distance to the object can be calculated from the parallax. By using this, a stereoscopic image display device that displays a right-eye image and a left-eye image on the right eye is prepared, and an image with parallax is provided as a right-eye image and a left-eye image. It is known that stereoscopic viewing is possible. Here, a plurality of images provided with parallax for the purpose of stereoscopic viewing are referred to as stereoscopic images. The distance to the object is expressed as “depth”. The parallax and the depth can be converted to each other if the focal length of the camera that captured the stereoscopic image and the interval between the cameras are known.

  It is said that humans associate the angle formed by the optical axes of both eyes according to parallax, that is, the size of the convergence angle, with the distance to the object in stereoscopic vision. Therefore, when an image with a parallax is displayed such that the position of the subject on the right-eye image is relatively shifted to the right when viewed from the left-eye image than the position of a certain subject on the left-eye image, the subject is It can be perceived as being on the far side of the actual display surface. On the contrary, when an image with parallax is displayed such that the position of the subject on the right-eye image is relatively shifted to the left when viewed from the left-eye image than the position of a certain subject on the left-eye image, Can be perceived as being closer to the foreground than the actual display surface.

  In the following, the shift in the right-direction parallax of the subject on the right-eye image with reference to the left-eye image will be described as positive parallax. If the amount of parallax is a positive value, the subject appears far away from the display screen. When the amount of parallax is 0, it appears to be at the same distance as the display screen. When the amount of parallax becomes a negative value, that is, when the subject on the right-eye image is shifted leftward with respect to the left-eye image, it appears in the foreground direction from the display screen. As a unit of the amount of parallax, for example, the number of pixels in the image can be used.

  However, if the parallax amount of the stereoscopic image is too large in the positive direction and the parallax amount exceeds the distance between the eyes of the observer, precisely the distance between the pupils when looking at infinity, it can occur in the natural world. In this state, stereoscopic viewing becomes impossible or even if stereoscopic viewing is possible, a heavy burden is imposed on the human body. On the other hand, if the amount of parallax is too large in the negative direction, extreme crossing will be forced on the viewer, and comfortable stereoscopic viewing will not be possible. In any case, as the amount of parallax increases in the positive or negative direction, the difference between the convergence and the focus adjustment of the eyes increases, resulting in an unnatural state, which causes a sense of discomfort. As described above, stereoscopic viewing can be comfortably performed within a certain range of the parallax amount of the stereoscopic image. However, when the parallax amount increases, the images of both eyes are not fused, and stereoscopic viewing becomes difficult or impossible.

  Patent Document 1 discloses a solution for this. FIG. 25 illustrates a main part of the invention described in Patent Document 1, and will be described using this. In FIG. 25, the parallax calculation unit 500 calculates a parallax map obtained by calculating the parallax at each coordinate for the entire screen from the left and right eye images. As a calculation method, a correlation matching method for calculating a correlation between luminance patterns of the left and right images is used. The gazing point calculation unit 501 calculates the maximum parallax value of the stereoscopic image, that is, the farthest view parallax amount, and the parallax control unit 502 calculates the left and right shift amount on the display screen of the screen display unit 503, and the farthest view parallax amount. Set so that the distance between the eyes of the observer is not exceeded. The distance between eyes is about 65 mm for adults. Thereby, the line of sight of the observer does not spread more than parallel, and control can be performed so that the parallax of the images of both eyes is within the range where they are fused. Alternatively, the gaze point calculation unit 501 calculates the minimum parallax value of the stereoscopic image, that is, the closest scene parallax amount, and the parallax control unit 502 calculates the right and left shift amount on the display screen of the screen display unit 503 as the nearest scene parallax. A method is disclosed in which the amount is set so as not to be equal to or less than a certain predetermined size β. As a result, the observer's viewpoint is located very close, and it is possible to eliminate a large mismatch state between the focus information of the eyes from the 3D image display surface and the convergence angle of the line of sight. The left and right images can be controlled so as to be easier. That is, by performing such correction processing on a stereoscopic image, a stereoscopic image that is always easy for the viewer to view can be presented.

  In addition, for example, when a subject is placed on the peripheral portion of the image, such as a stereoscopic image in which the upper body of a person is taken up, and the subject has a parallax amount that is perceived in front of the screen, the parallax amount Nevertheless, there is a phenomenon that it is difficult to see by jumping out to the front. For viewers, the situation that the subject displayed on the screen of the display device is cut off at the frame part of the screen and appears floating in front is caused by a sense of discomfort, preventing the subject from jumping out and perceiving Conceivable.

  On the other hand, Patent Document 2 discloses a technique for appropriately adjusting video display characteristics according to the type of video to be displayed.

  FIG. 26 illustrates a main part of the invention described in Patent Document 2, and will be described using this. The luminance information detection unit 609 detects the luminance information value Yi from the luminance at each pixel of the luminance signal Y included in the video signal Db output from the receiving unit 602. The content feature detection unit 610 determines the feature of the video content of each frame based on the luminance information value Yi, and outputs the content feature determination value Ji to the multiple content feature detector 611. The multiple content feature detection unit 611 performs video content determination with less variation (content determination that is not easily affected by temporary variation) from the content feature determination value Ji for a plurality of frames, and the multiple content feature as a more likely determination result. The determination value Fi is output to the correction control unit 606. The correction control unit 606 outputs the video correction value Pi corresponding to the feature of each content to the video correction unit 605 based on the multiple content feature determination value Fi. The video correction unit 605 performs video correction on the video signal Dc using the input video correction parameter Pi, and outputs it to the display unit 604 as the video signal Dd. With such a configuration, appropriate image quality correction is performed according to the type of input video.

JP-A-7-167633 JP 2008-134277 A

  As described above, if the amount of parallax is too large in the positive direction or too large in the negative direction, it is difficult or impossible to make a stereoscopic image. For this reason, many stereoscopic images are manufactured so as not to add too much parallax. In addition, as described above, when a subject covers the peripheral part of the image and has a parallax amount that is perceived in front of the screen as described above, the stereoscopic image appears to pop out to the front regardless of the amount of parallax. There is a phenomenon that is difficult. For this reason, many stereoscopic images are produced so that the subject does not protrude too much in front of the screen. However, if a 3D image is produced according to the above-mentioned policy uniformly, there will be more 3D images in which the depth is not felt and 3D images in which the subject does not appear to pop out, that is, 3D images with a poor 3D effect. The characteristics will not be utilized.

  On the other hand, it is conceivable to perform a stereoscopic effect enhancement process on a stereoscopic image with a poor stereoscopic effect when the stereoscopic image is displayed. At this time, since the stereoscopic effect of all the stereoscopic images is not poor, it is desirable to selectively perform the stereoscopic effect enhancement process on the stereoscopic image with poor stereoscopic effect. In addition, since there are various factors that cause a lack of stereoscopic effect, it is desirable to change the type and amount of enhancement processing for the stereoscopic effect according to the stereoscopic effect of the stereoscopic image.

  However, the stereoscopic image adjustment function described in Patent Document 1 is intended to make the stereoscopic image easy to see and less fatigue, that is, considering the safety of stereoscopic vision, and does not consider enhancing the stereoscopic effect. It does not have a function of discriminating a stereoscopic image with a poor stereoscopic effect or a function of detecting a feature of the stereoscopic effect of the stereoscopic image, and the above-described stereoscopic effect enhancement process cannot be appropriately performed.

  The technique described in Patent Document 2 is a technique for determining the characteristics of a video using luminance information of the displayed video and appropriately adjusting the video display characteristics according to the type of the video. This is a technique for a normal image that is not an image. It does not have a function to discriminate a stereoscopic image with poor stereoscopic effect, a feature to detect the stereoscopic effect of a stereoscopic image, or a function to enhance the stereoscopic effect, and performs the above-described stereoscopic effect enhancement processing. I can't.

  The present invention has been made in view of the above problems, and an object of the present invention is to determine a stereoscopic image with a poor stereoscopic effect and to perform a correction process that enhances the stereoscopic effect according to the stereoscopic effect of the stereoscopic image. .

  According to an aspect of the present invention, the stereoscopic image processing apparatus of the present invention is a stereoscopic image processing apparatus that classifies a scene of a stereoscopic image, the input unit for inputting the stereoscopic image, and the stereoscopic image from the stereoscopic image. A parallax feature amount calculation unit that calculates one or more parallax feature amounts, and a stereoscopic image classification unit that classifies the stereoscopic image into a plurality of scenes based on the parallax feature amount.

  According to another aspect of the present invention, the stereoscopic image processing apparatus of the present invention may include an image correction unit that performs a stereoscopic image correction process based on the scene.

  According to another aspect of the present invention, the stereoscopic image correction process may be a process of enhancing a stereoscopic effect by a parallax adjustment process of the stereoscopic image.

  According to another aspect of the present invention, the parallax feature amount may be a parallax range width that is a width between the parallax amount of the closest view and the parallax amount of the farthest view of the stereoscopic image.

  According to another aspect of the present invention, the parallax feature amount may be a foreground-side frequency sum determined by a near-field parallax amount from a first constant related to the parallax amount.

  According to another aspect of the present invention, the parallax feature amount is determined based on a second foreground-side frequency sum determined by a near-field parallax amount from a second constant relating to a parallax amount smaller than a first constant relating to the parallax amount. It may be.

  According to another aspect of the present invention, the parallax feature amount is an image peripheral portion foreground side frequency sum determined by a parallax amount in the peripheral portion of the stereoscopic image closer to the foreground than the third constant related to the parallax amount. There may be.

  According to another aspect of the present invention, the stereoscopic image classification unit performs a process of classifying the stereoscopic image by comparing the parallax range width with a first parallax range width constant, and the image correction unit includes: You may perform the parallax adjustment process which enlarges the parallax of the said stereo image to the scene where the said parallax range width is smaller than the said 1st parallax range width constant.

  According to another aspect of the present invention, the stereoscopic image classification unit compares the parallax range width with the first parallax range width constant and a second parallax range width constant larger than the stereoscopic image. A process of classifying, and a process of classifying the stereoscopic image by comparing the foreground-side power sum total with the first foreground-side power sum constant and a larger second foreground-side power sum constant. The scene is a scene in which the parallax range width is larger than the first parallax range width constant and smaller than the second parallax range width constant, and the foreground side power sum is the first foreground side power sum constant A parallax adjustment process for reducing the parallax of the stereoscopic image may be performed on a scene that is larger and smaller than the second foreground-side power sum constant.

  According to another aspect of the present invention, the stereoscopic image classification unit compares the parallax range width with the first parallax range width constant and the second parallax range width constant larger than the stereoscopic image. , A process for comparing and classifying the foreground side power sum and the third foreground side power sum constant, and comparing the second foreground side power sum and the fourth foreground side power sum constant The image correction processing unit is a scene in which the parallax range width is larger than the first parallax range width constant and smaller than the second parallax range width constant; and The parallax of the stereoscopic image is applied to a scene in which a foreground side power sum is larger than the third foreground side power sum constant and the second foreground side power sum is smaller than the fourth foreground side power sum constant. Perform parallax adjustment processing to reduce Good.

  According to another aspect of the present invention, the stereoscopic image classifying unit compares the parallax range width with the first parallax range width constant to classify the stereoscopic image, and the image peripheral portion near view side A process of classifying the stereoscopic image by comparing the frequency sum and the first image peripheral portion near-field-side power sum constant, and the image correction processing unit is configured such that the parallax range width is greater than the first parallax range width constant. A parallax adjustment process for increasing the parallax of the stereoscopic image may be performed on a scene that is a large scene and whose image peripheral portion near view side power sum is larger than the first image peripheral portion near view side power sum constant.

  According to another aspect of the present invention, a stereoscopic image processing apparatus of the present invention is a stereoscopic image processing apparatus that classifies a scene of a stereoscopic image, and the stereoscopic image and at least one parallax feature amount of the stereoscopic image are obtained. An input unit for input and a stereoscopic image classification unit for classifying the stereoscopic image into a plurality of scenes based on the parallax feature amount are provided.

  According to another aspect of the present invention, the stereoscopic image processing apparatus of the present invention may include a scene change detection unit that detects a scene change of the stereoscopic image based on a change in the scene.

  According to another aspect of the present invention, a stereoscopic image processing method of the present invention is a stereoscopic image processing method for classifying a scene of a stereoscopic image, the step of inputting the stereoscopic image, and the stereoscopic image from the stereoscopic image. The method includes calculating one or more parallax feature amounts of the image and classifying the stereoscopic image into a plurality of scenes based on the parallax feature amount.

  According to another aspect of the present invention, a stereoscopic image processing method of the present invention is a stereoscopic image processing method for classifying a scene of a stereoscopic image, wherein the stereoscopic image and one or more parallax feature quantities of the stereoscopic image are obtained. And a step of classifying the stereoscopic image into a plurality of scenes based on the parallax feature amount.

  According to another aspect of the present invention, the program of the present invention is a program for causing a computer to execute a stereoscopic image process for classifying a stereoscopic image scene, the step of inputting the stereoscopic image, and the stereoscopic image The method includes calculating one or more parallax feature amounts of the stereoscopic image from the image and classifying the stereoscopic image into a plurality of scenes based on the parallax feature amount.

  According to another aspect of the present invention, a program of the present invention is a program for causing a computer to execute a stereoscopic image process for classifying a stereoscopic image scene, the stereoscopic image and at least one of the stereoscopic image. The method includes inputting a parallax feature value and classifying the stereoscopic image into a plurality of scenes based on the parallax feature value.

  By the above means, the stereoscopic image processing apparatus according to the present invention can calculate the parallax feature amount of the stereoscopic image and use this to classify the stereoscopic image into a plurality of scenes. Further, it is possible to perform a correction process for enhancing the stereoscopic effect of the stereoscopic image based on the scene.

It is a figure explaining the concept of the frame of a stereo image. It is a block diagram which shows the structure of the stereo image display apparatus which concerns on a 1st Example. It is a block diagram which shows the structure of the parallax feature-value calculation part which concerns on a 1st Example. It is an example of a parallax histogram. It is a figure which shows the kind of classification | category of the stereo image in 1st Example, and the image correction process corresponding to it. It is a figure explaining a stereo image and the depth perception by it. It is a figure explaining a stereo image and the depth perception by it. It is a figure explaining a stereo image and the depth perception by it. It is a block diagram which shows the structure of the stereo image display apparatus which concerns on a 2nd Example. It is a block diagram which shows the structure of the parallax feature-value calculation part which concerns on a 2nd Example. It is the figure which showed the parallax histogram for demonstrating near view side frequency sum total. It is a figure which shows the kind of classification | category of the stereo image in the 2nd Example, and the image correction process corresponding to it. It is a figure explaining the determination method of the parallax adjustment amount in a 2nd Example. It is a block diagram which shows the structure of the stereo image display apparatus which concerns on a 3rd Example. It is a block diagram which shows the structure of the parallax feature-value calculation part which concerns on a 3rd Example. It is a figure explaining the peripheral part of an image. It is a figure which shows the kind of classification | category of the stereo image in a 3rd Example, and the image correction process corresponding to it. It is a block diagram which shows the structure of the stereo image display apparatus which concerns on a 4th Example. It is a block diagram which shows the structure of the parallax feature-value calculation part which concerns on a 4th Example. It is the figure which showed the parallax histogram for demonstrating a foreground side frequency sum total and a 2nd foreground side frequency sum total. It is a figure which shows the kind of classification | category of the stereo image in 4th Example, and the image correction process corresponding to it. It is a block diagram which shows the structure of the stereo image display apparatus which concerns on a 5th Example. It is a block diagram which shows the structure of the scene change detection part which concerns on a 5th Example. It is a figure explaining the relationship between the frame used for scene change detection, and the position of the scene change detected by it. It is a figure explaining the parallax control method of patent document 1. FIG. It is a figure explaining the video display apparatus of patent document 2. FIG.

  Hereinafter, preferred embodiments of a stereoscopic image processing apparatus of the present invention will be described in detail with reference to the accompanying drawings. The present invention can be applied to both a field signal and a frame signal, but since a field and a frame have a similar relationship to each other, the frame signal will be described as a representative example.

  First, referring to FIG. 1, the concept of a frame of a stereoscopic image composed of right-eye image data and left-eye image data in the present invention will be described. FIG. 1A shows a left-eye image (L) in which the horizontal resolution is halved to the normal half on the left half of one image, and the horizontal resolution to the normal half in the right half. 3 is a stereoscopic image in a format storing a right-eye image (R) 2. It is called side-by-side format. In this case, one image composed of the left-eye image (L) and the right-eye image (R) is one frame. In the case of a moving image, if the frame rate is X frames / second, X images of the above format are required per second, that is, the same number as the frame rate.

  FIG. 1B shows a left-eye image (L) in which the vertical resolution is halved in the upper half of one image, and the vertical resolution in the lower half of the normal 1 /. 3 is a stereoscopic image in a format storing a right-eye image (R) 2. It is called the top and bottom format. Also in this case, one image composed of the left-eye image (L) and the right-eye image (R) is one frame. Similarly to the side-by-side format, the number of images in this format is the same as the frame rate per second. In addition to this, there is a method called frame packing in which the left-eye image and the right-eye image are connected to each other in the vertical direction so as to form a single large image. In the case of a moving image, the same number of images of this format as the frame rate is required per second.

  On the other hand, FIG. 1C shows a stereoscopic image in a form in which left-eye images (L) and right-eye images (R) are alternately arranged on the time axis. It is called the frame sequential format. In this case, in the present application, as shown in FIG. 1C, one left-eye image (L) and one right-eye image (R) are counted as one frame. Therefore, in the case of a moving image, assuming that the frame rate is X frames / second, the left-eye image (L) is alternately arranged X times per second, and the right-eye image (R) X times per second. The total is 2X images.

  Thus, in the present application, a set of the left-eye image and the right-eye image is considered as one frame regardless of the actual image storage format. Furthermore, in the case of an n-eye image composed of images of n viewpoints (where n> 2), the image for n eyes is set as one frame.

  Next, the definition of scenes and scene changes in the present invention will be described.

  The scene in the present invention refers to an image group that can be subjected to the same correction processing. In other words, the same correction process can be applied to images classified into the same scene.

  In the present invention, scene similarity is determined and classified using disparity information of a stereoscopic image. If the parallax information of two stereoscopic images is similar, the images are classified into the same scene.

  When the stereoscopic image is a moving image, a series of consecutive images captured by a single camera is often the same scene. This is because the disparity information is not largely changed in these series of images.

  In addition, a scene change in a stereoscopic video is called a scene change. The parallax of a stereoscopic image usually changes discontinuously before and after a scene change. This is because the subject changes for each scene, or the positional relationship between the subject and the camera changes.

  1st Example of this invention is a stereo image processing apparatus, Comprising: The image correction process which emphasizes a stereo effect is performed with respect to the scene where a stereo effect is felt poor because the width | variety of the depth of a stereo image is narrow. It is aimed. For this purpose, scene classification is performed using the width of the parallax range of the stereoscopic image as the parallax feature amount of the stereoscopic image.

  FIG. 2 is a block diagram illustrating the configuration of the stereoscopic image processing apparatus 1A according to the first embodiment. The stereoscopic image processing apparatus 1A includes an input unit 2, a stereoscopic image processing unit 3, a parallax feature amount calculation unit 4A, a stereoscopic image classification unit 5A, an image delay unit 6, an image correction unit 7A, a display control unit 8, a display unit 9, and glasses. The synchronization unit 10 is configured. Further, the shutter glasses 11 worn by the user are used.

  The input unit 2 is a part that inputs stereoscopic image data to the stereoscopic image processing apparatus 1A. The input stereoscopic image data may be any data such as a broadcast wave, electronically read from a recording medium, or acquired through communication. That is, the input unit 2 may be a broadcast radio receiver device, or may have a semiconductor memory reading device, an optical disk or magnetic disk reading device, or a communication function with a network. Alternatively, the stereoscopic image processing apparatus 1A may be a camera that can capture a stereoscopic image, and the input unit 2 may be a multi-view camera. In short, any data can be used as long as it can input data that can be interpreted as a stereoscopic image. Further, the stereoscopic image data may be composed of right-eye image data and left-eye image data, or may be multi-view image data for multi-view display. Further, it may be composed of image data and depth data or parallax data. Furthermore, the stereoscopic image data may be a still image or a moving image. The input unit 2 sends the input stereoscopic image data to the stereoscopic image processing unit 3. When the stereoscopic image data is a still image, the same image is continuously output and handled in the same manner as a moving image.

  The stereoscopic image processing unit 3 receives image data input from the input unit 2 and develops it into left-eye image data and right-eye image data. If there is additional information in the input image data, the additional information is extracted. The additional information may be camera parameters at the time of shooting, parallax information and depth information, parallax histogram information, and the like. As described above, the stereoscopic image data input to the input unit 2 may have various formats in addition to the right-eye image data and the left-eye image data. If the stereoscopic image data consists of right-eye image data and left-eye image data, they are used as they are. In the case of multi-viewpoint image data, data for two viewpoints is selected from the data and used as right-eye image data and left-eye image data. The stereoscopic image processing unit 3 sends the left-eye image data and the right-eye image data to the parallax feature amount calculation unit 4A and the image delay unit 6. If there is additional information, it is also sent to the parallax feature amount calculation unit 4A. When the stereoscopic image data is composed of image data and parallax data, these data may be sent to the parallax feature amount calculation unit 4A as they are. Further, when the stereoscopic image data includes image data and depth data, the depth data may be converted into parallax data, and the image data and the parallax data may be sent to the parallax feature amount calculation unit 4A.

  The parallax feature amount calculation unit 4A calculates a parallax from the left-eye image data and the right-eye image data, and further extracts a parallax feature amount that is a feature amount related to the parallax. In some cases, a parallax feature amount is extracted using additional information. Details of these processes will be described later. The parallax feature amount calculation unit 4A sends the extracted parallax feature amount to the stereoscopic image classification unit 5A.

  The stereoscopic image classification unit 5A classifies the stereoscopic image into a predetermined type using the parallax feature amount. Then, the stereoscopic image classification result is output to the image correction unit 7A.

  The image delay unit 6 has a memory for holding the left-eye image data and the right-eye image data input from the stereoscopic image processing unit 3, delays the input image data by one frame, and sends it to the image correction unit 7A. Output. That is, when the (n + 1) th frame image is input, the nth frame image is output. The result of processing the parallax feature quantity calculation unit 4A and the stereoscopic image classification unit 5A on the nth frame image can be obtained only after all the nth frame images have been processed. Therefore, in order to perform image correction processing on the same n-th frame image using the result, it is necessary to hold and delay the n-th frame image data.

  The image correction unit 7A determines an image correction process based on the stereoscopic image classification result input from the stereoscopic image classification unit 5A, and performs image correction on the left-eye image data and the right-eye image data input from the image delay unit 6. Process. Then, the image data for left eye and the image data for right eye after image correction processing are output to the display control unit 8.

  The display control unit 8 outputs the left-eye image data and the right-eye image data after the image correction process input from the image correction unit 7A to the display unit 9 in a manner that matches the stereoscopic image presentation method. In addition, a synchronization signal for operating the shutter glasses 11 worn by the observer in synchronization with image display on the display unit 9 is sent to the glasses synchronization unit 10.

  The display unit 9 displays the image input from the display control unit 8 as needed. In this embodiment, a liquid crystal display panel is used as the display unit 9, and the left-eye image and the right-eye image are alternately displayed.

  The glasses synchronization unit 10 transmits the synchronization signal sent from the display control unit 8 to the shutter glasses 11 using infrared rays or radio waves.

  The shutter glasses 11 receive the synchronization signal transmitted from the glasses synchronization unit 10 and open and close the shutters for the right eye and the left eye accordingly. An observer sees the display unit 9 wearing the shutter glasses 11. In the present embodiment, as described above, a liquid crystal display panel is used for the display unit 9, a left-eye image and a right-eye image are alternately displayed, and a stereoscopic view is performed in synchronization with the shutter glasses 11 worn by the observer. ing. When the left-eye image is displayed on the display unit 9, the left-eye image is presented to the left eye by opening the left-eye shutter of the shutter glasses 11 and closing the right-eye shutter, and the right-eye image is displayed. Sometimes the left-eye shutter is closed and the right-eye shutter is opened, so that the right-eye image is presented to the right eye to realize stereoscopic viewing.

  Furthermore, more detailed processing contents of the parallax feature amount calculation unit 4A, the stereoscopic image classification unit 5A, and the image correction unit 7A will be described.

  FIG. 3 is a block diagram illustrating a configuration of the parallax feature amount calculation unit 4A. It includes a parallax calculation unit 41, a parallax histogram creation unit 42, and a parallax range width calculation unit 43.

  The parallax calculation unit 41 obtains a shift between corresponding points of the right-eye image and the left-eye image constituting the stereoscopic image, that is, parallax from the left-eye image data and the right-eye image data over the entire image using block matching or the like. At this time, the parallax may be obtained for each pixel in the image, or the parallax may be obtained for each block of a predetermined size (for example, an 8 × 8 pixel block). Note that when the parallax for each pixel or each block having a predetermined size is added as the additional information of the stereoscopic image data input to the input unit 2, the processing in the parallax calculation unit 41 may be skipped. When image data and parallax data are sent from the stereoscopic image processing unit 3 as stereoscopic image data, the processing in the parallax calculation unit 41 is skipped. In addition, when additional information includes depth information for each pixel or block of a predetermined size and camera interval and focal length information at the time of shooting, information about the camera interval and focal length is used to determine the depth. May be converted into parallax. The parallax calculation unit 41 sends the parallax to the parallax histogram creation unit 42.

  The parallax histogram creation unit 42 creates a parallax histogram that is a frequency distribution from the parallax data of the entire image obtained by the parallax calculation unit 41. A depth histogram may be used instead of the parallax histogram. In other words, any display may be used as long as the parallax of the display object represented in the stereoscopic image or a frequency distribution of the same amount is represented. The parallax histogram creation unit 42 sends the parallax histogram to the parallax range width calculation unit 43.

  The parallax range width calculation unit 43 obtains the parallax amount of the nearest scene and the parallax amount of the farthest view in the image from the parallax histogram input from the parallax histogram creation unit 42, and uses the width as the parallax feature amount to the stereoscopic image classification unit 5A. Output. In this manner, the parallax range in which a subject in a stereoscopic image exists is hereinafter referred to as a parallax range. Further, the width of the parallax range is referred to as a parallax range width and is denoted as W.

  Details of the processing of the parallax range width calculation unit 43 will be described with reference to FIG. FIG. 4 shows an example of a parallax histogram. The horizontal axis is the amount of parallax, and the vertical axis is the frequency. A point where the amount of parallax is 0 is used as a reference, 0 or more is the far view side, and less than 0 is the close view side. This parallax histogram has a small peak on the near side and a large peak on the far side. For example, when a few people nearby are photographed against a background of a distant landscape, such a histogram is formed. The threshold value Td is a threshold value of the frequency of the parallax histogram. Since the actual parallax histogram has a small peak that is not smooth as shown in FIG. 4, the threshold value Td is used to exclude the influence. The parallax amount of the point having the maximum parallax amount at the intersection of the parallax histogram and the threshold value Td is set as the parallax amount of the farthest view, and the parallax amount of the point having the minimum parallax amount is set as the parallax amount of the nearest scene. That is, even when there are a plurality of intersections as shown in FIG. 4, the parallax amount at the intersection where the parallax is maximum and minimum is set as the parallax amount of the farthest view and the parallax amount of the nearest scene. The parallax range width W is obtained by subtracting the parallax amount of the latest view from the parallax amount of the farthest view.

  Here, the disparity amount of the farthest view and the disparity amount of the closest view are detected using one threshold Td, but may be detected using different threshold values for the disparity amount of the nearest view and the disparity amount of the most distant view. .

  The stereoscopic image classification unit 5A classifies the stereoscopic image using the parallax range width W. In this embodiment, the parallax range width W is classified into two types. Assuming that the threshold for determination is Tw1A, if W <Tw1A, the stereoscopic image is classified as a narrow parallax range, and if Tw1A ≦ W, the stereo image is classified as a wide parallax range. The classification result is output to the image correction unit 7A.

  In order to classify stereoscopic images with a narrow parallax range in this way, for example, Tw1A can be determined as follows. The maximum disparity amount on the distant view side that can be accepted by humans is an amount that the disparity amount on the screen is equal to the distance between the pupils when looking at infinity. This is the far-field maximum parallax amount. On the other hand, the maximum amount of parallax on the near side that humans can tolerate is generally the difference between the angle of convergence when the human eye looks at the screen and the angle of convergence when the apparent position of the subject is viewed. It is said that the amount of parallax that does not exceed 1 ° is safe. From here, the foreground side parallax amount at which the parallax angle difference is 1 ° is set as the foreground side maximum parallax amount. Then, the maximum disparity range width Wmax is obtained by subtracting the distant view side maximum disparity amount from the distant view side maximum disparity amount. Tw1A can be determined based on this Wmax. For example, a value of 10% of Wmax is Tw1A.

  FIG. 5 shows an example of a typical parallax histogram relating to two types of classification by the classification process. The horizontal axis of each parallax histogram is the amount of parallax, the vertical axis is the frequency, and the thin vertical line at the center is a line of parallax 0. In addition, the types of image correction processing to be described later are described in portions below the dotted line of each square.

  The parallax histogram on the left side of FIG. 5 satisfies the W <Tw1A and is classified as a stereoscopic image with a narrow parallax range width. Thus, the parallax histogram is within a narrow range. That is, it is a stereoscopic image with a narrow depth between subjects in the image and a poor stereoscopic effect. The parallax histogram on the right side of FIG. 5 is classified as a stereoscopic image that satisfies Tw1A ≦ W and has a wide parallax range. Thus, the parallax histogram is distributed over a wide range. That is, there are various depth portions in the image, and the stereoscopic image is rich in stereoscopic effect.

  The image correcting unit 7A uses the stereoscopic image classification result based on the size of the parallax range width W, and performs a process of adjusting the parallax so as to enhance the stereoscopic effect when the stereoscopic image is classified as a scene with a narrow parallax range width. . Various methods are conceivable as the parallax adjustment means. In this embodiment, a parallax adjustment method by shifting the left and right images in the horizontal direction is used. This principle will be described with reference to FIGS.

  FIG. 6A shows a stereoscopic image composed of left and right images in which two objects, a circle and a triangle, are shown. FIG. 6B shows how the depth of each object is perceived when a human views this stereoscopic image using a stereoscopic image display device. Due to the positional relationship between the objects on the displayed left and right images, a triangular object is perceived in front of the stereoscopic image display device, and a round object is perceived in the back.

  FIG. 7A shows a stereoscopic image that has been subjected to a shift process in which the left-eye image of FIG. 6A is shifted to the left and the right-eye image is shifted to the right. With this process, the parallax increases uniformly over the entire screen. FIG. 7B shows the perception of the depth of each object by the stereoscopic image of FIG. When FIG. 7B is compared with FIG. 6B, it can be seen that there is an effect that any object appears to move further in the distant view direction. In addition, it can be seen that the round object moves more in the distant direction than the triangular object, and the depth of the two objects is wider. That is, it can be seen that there is an effect that the stereoscopic effect is more emphasized.

  In contrast, FIG. 8A shows a stereoscopic image subjected to a shift process in which the left-eye image in FIG. 6A is shifted to the right and the right-eye image is shifted to the left. By this process, the parallax is uniformly reduced over the entire screen. FIG. 8B shows the perception of the depth of each object by the stereoscopic image of FIG. Comparing FIG. 8B with FIG. 6B, it can be seen that there is an effect that any object appears to move more in the foreground direction. It can also be seen that the round object moves more in the foreground direction than the triangular object, and the depth of the two objects is narrower. In other words, it can be seen that there is an effect that the three-dimensional effect is further weakened.

  In this way, the effect of changing the perceived position of an object in the image simply by performing a shift process that shifts the positional relationship between the right-eye image and the left-eye image in the horizontal direction to increase or decrease the parallax uniformly, By changing the width of the depth, an effect of enhancing or weakening the stereoscopic effect can be obtained. By using this principle, it is possible to adjust an image so that an object that is projected too far forward is shifted to the back, and to adjust an image that is viewed too deeply to the front. It is also possible to intentionally enhance or weaken the stereoscopic effect. In this shift process, the parallax is uniformly increased and decreased, and the parallax range width is not changed, but the depth width is increased and decreased.

  When the stereoscopic image is classified as a scene with a narrow parallax range, that is, when the image correction unit 7A is classified into a square having “Processing A” in FIG. Shift processing is performed to shift the image to the right, and the parallax is increased uniformly over the entire screen. As described above with reference to FIG. 7, this shift process has the effect of widening the depth and enhancing the stereoscopic effect.

  Next, a method for determining the adjustment amount of the shift process for parallax adjustment will be described. As the stereoscopic effect of the stereoscopic image before image correction is poorer, it is desirable to perform an image correction process that emphasizes the stereoscopic effect more strongly, that is, to increase the adjustment amount. When the adjustment amount of the shift process is a, a can be determined by the following method.

  For example, a = c1 / W can be determined using the parallax range width W and the constant c1. The adjustment amount “a” is increased as the stereoscopic image has a narrower parallax range width W and less stereoscopic effect.

  Further, the mode value Hmax can be obtained from the parallax histogram, and a = c2 × Hmax can be determined using the constant c2. FIG. 4 shows Hmax. In general, when the parallax range width is narrow, the frequency Hmax of the mode value of the parallax histogram increases. Therefore, the adjustment amount a is increased as Hmax is increased.

  Alternatively, a = c3 / W × Hmax can be determined by using both the parallax range width W and the mode Hmax of the parallax histogram and the constant c3. Furthermore, the above methods may be combined, for example, a = c1 / W + c2 × Hmax.

  When the stereoscopic image is classified in other manners, that is, when the stereoscopic image is classified into a square having “−” in FIG. 5, the image correction unit 7A does not perform image correction processing. In this case, the parallax range width is medium or more, and a stereoscopic effect can be sufficiently felt without performing image correction processing.

  As described above, the stereoscopic image processing apparatus 1A according to the present embodiment classifies a stereoscopic image into a scene having a narrow depth and a scene having a wide depth by using the parallax range width W of the stereoscopic image. Thereby, it has the effect that the scene classification regarding the content of the stereoscopic image can be performed by the feature of the parallax amount of the stereoscopic image. Such classification is not possible with conventional image classification means using luminance information or the like.

  In addition, the stereoscopic image processing apparatus 1A according to the present embodiment performs image correction processing for enhancing the stereoscopic effect on a scene having a narrow depth, that is, a scene having a poor stereoscopic effect, using the scene classification result of the stereoscopic image. Accordingly, there is an effect that it is possible to selectively perform a stereoscopic effect enhancement process suitable for a scene with little stereoscopic effect.

  Further, the stereoscopic image processing apparatus 1A according to the present embodiment determines the adjustment amount of the shift processing using a feature amount related to the stereoscopic effect of the stereoscopic image, such as the parallax range width W and the mode value Hmax. Thereby, there is an effect that an optimal adjustment amount can be set for each stereoscopic image.

  The second embodiment of the present invention is a stereoscopic image processing apparatus, and in addition to the image correction processing described in the first embodiment, for a scene in which the stereoscopic effect is felt poor because the subject does not pop out. An object of the present invention is to perform image correction processing for emphasizing the three-dimensional effect by popping out the subject. For this reason, as compared with the first embodiment, scene classification is performed by using an amount indicating the ratio of the subject existing in the foreground side in the image as the parallax feature amount.

  FIG. 9 is a block diagram illustrating a configuration of a stereoscopic image processing apparatus 1B according to the second embodiment. The stereoscopic image processing apparatus 1A according to the first embodiment is different in a parallax feature amount calculation unit 4B, a stereoscopic image classification unit 5B, and an image correction unit 7B. The description about the same part is abbreviate | omitted, and it demonstrates centering on a different part.

  FIG. 10 is a block diagram illustrating a configuration of the parallax feature amount calculation unit 4B. Compared to the parallax feature amount calculation unit 4A, a foreground side power total calculation unit 44 is added.

  The foreground-side power sum total calculation unit 44 obtains a foreground-side power sum S that is the sum of the foreground-side frequencies from the parallax histogram input from the parallax histogram creation unit 42, and outputs this to the stereoscopic image classification unit 5B as a parallax feature amount. To do.

FIG. 11 is a diagram showing a parallax histogram for explaining the foreground-side power sum S. FIG. The horizontal axis is the amount of parallax, and the vertical axis is the frequency. A point where the amount of parallax is 0 is used as a reference, and 0 or more is the far side, and less than 0 is the near side. Using the predetermined threshold value Td1, the sum of the frequencies of bins whose parallax amount is equal to or less than Td1 is obtained in the parallax histogram, and this is used as the near-field-side frequency sum S. In FIG. 11, the shaded portion corresponds to the foreground side power sum S. Parallax histogram m from the parallax amount -l (l, m are natural numbers) is written in a range of the width of each bin of the parallax histogram 1, and the frequency of each bin _{h i (-l ≦ i ≦ m} ) Then, the foreground side power sum S can be obtained by the following equation.
S = Σh _i (−1 ≦ i ≦ Td1)
In general, Td1 = 0 is set, and the total frequency up to the parallax amount 0 in the parallax histogram is calculated. However, the present invention is not limited to this, and the total frequency up to a parallax amount other than 0 may be calculated. For example, since there is an estimation error in the parallax estimation, the frequency sum may be obtained by setting Td = 10 in order to obtain the frequency widely including pixels that may have a near-field parallax.

  On the other hand, the parallax range width calculation unit 43 outputs the parallax range width W as a parallax feature amount as described above. Therefore, the parallax feature amount calculation unit 4B outputs two pieces of information, that is, the parallax range width W and the foreground side power sum S as the parallax feature amount, to the stereoscopic image classification unit 5B.

  The stereoscopic image classification unit 5B classifies the stereoscopic image using the parallax range width W and the foreground-side power sum S that are parallax feature amounts. In this embodiment, the parallax range width W is classified into three types, and the foreground side power sum S is classified into three types. Therefore, it can be classified into 3 × 3 = 9 types in total. The classification result is output to the image correction unit 7B.

The parallax range width W is classified as follows using threshold values Tw1B and Tw2B (Tw1B <Tw2B) for determination.
W <Tw1B: Stereo image with a narrow parallax range width Tw1B ≦ W ≦ Tw2B: Stereo image with a medium parallax range width Tw2B <W: Stereo image with a wide parallax range width In order to classify medium images with a medium parallax range width, for example, a value of 10% of the maximum parallax range width Wmax is set as Tw1B, and a value of 50% of Wmax is set as Tw2B.

The near-field-side frequency sum S is classified as follows using threshold values Ts1 and Ts2 (0 ≦ Ts1 <Ts2) for determination.
S <Ts1: Stereo image with almost no subject on the foreground side Ts1 ≦ S ≦ Ts2: Stereo image with a small subject on the foreground side Ts2 <S: Stereo image with a medium or larger subject on the foreground side Thus, in order to determine whether or not there is a small subject on the foreground side, the threshold values Ts1 and Ts2 are both set to small values. For example, Ts1 is set to 5% of the total frequency of the parallax histogram, and Ts2 is set to 10% of the total frequency of the parallax histogram.

  FIG. 12 shows a list of examples of typical parallax histograms of nine types of scenes obtained by the above classification process. Three classifications according to the parallax range width W are described in the row direction, and three classifications according to the foreground-side frequency sum S are described in the column direction, and a total of nine classifications are shown in the matrix. The horizontal axis of each parallax histogram is the parallax amount, the vertical axis is the frequency, and the thin vertical line in the center represents parallax 0. In addition, the types of image correction processing to be described later are described in portions below the dotted line of each square.

  First, the three classifications based on the parallax range width W will be described. The parallax histogram in the left column of FIG. 12 is classified as a stereoscopic image that satisfies W <Tw1B and has a narrow parallax range width. Although the outlines of the parallax histograms are different, the parallax range width is narrow. The parallax histogram in the center column in FIG. 12 is classified as a stereoscopic image that satisfies Tw1B ≦ W ≦ Tw2B and has a medium parallax range width. Although the outlines of the parallax histograms are different from each other, the parallax range width is medium in all cases. The parallax histogram in the right column of FIG. 12 is classified as a stereoscopic image that satisfies Tw2B <W and has a wide parallax range. Although the outlines of the parallax histograms are different, the parallax range is wide.

  Next, three classifications based on the foreground-side frequency sum S will be described. The parallax histogram in the upper row in FIG. 12 is classified as a stereoscopic image that satisfies S <Ts1 and has no subject on the near view side. Although the parallax histograms have different widths, there is almost no frequency of the parallax histogram on the left side of the parallax 0 line. The parallax histogram in the center row of FIG. 12 is classified as a stereoscopic image that satisfies Ts1 ≦ S ≦ Ts2 and has a relatively small subject on the foreground side. Although the widths of the parallax histograms are different from each other, there are a few parallax histograms on the left side of the parallax line. The parallax histogram in the lower row of FIG. 12 is classified as a stereoscopic image that satisfies Ts2 <S and has a medium or higher subject on the near view side. Although the widths of the parallax histograms are different from each other, many portions of the parallax histogram exist on the left side of the parallax line.

  The image correction unit 7B performs image correction processing on a predetermined scene according to the stereoscopic image classification results classified into nine types of scenes as described above. As in the first embodiment, the correction means uses a parallax adjustment method by shifting the left and right images.

  When the stereoscopic image is classified as a scene having a narrow parallax range width, that is, when the stereoscopic image is classified into one of the squares described as “Processing A” in FIG. 12, the image correction unit 7B shifts the left-eye image to the left. A shift process for shifting the right-eye image to the right is performed, and the parallax is increased uniformly over the entire screen. As described above with reference to FIG. 7, this shift process has the effect of widening the depth and enhancing the stereoscopic effect. The adjustment amount in this shift process can be determined by the same method as in the first embodiment.

  When the stereoscopic image is classified as a scene having a medium parallax range width and a relatively small subject on the foreground side, that is, when the stereoscopic image is classified into a square described as “Processing B” in FIG. The correction unit 7B performs a shift process of shifting the left-eye image to the right and the right-eye image to the left, and reduces the parallax uniformly over the entire screen. As described above with reference to FIG. 8, this shift processing has an effect that the entire subject in the image appears to move out to the near side.

  Here, the condition that the parallax range width is medium is based on the following reason. As described above, in a stereoscopic image, when a subject covers the peripheral portion of the image and has a parallax amount that can be perceived in front of the screen, it is difficult to see the subject popping out in front of the screen regardless of the amount of parallax. There is a phenomenon. Here, if adjustment is performed to project the subject to the foreground side with respect to a stereoscopic image with a narrow parallax range width, all or most of the subject will be projected to the foreground side, and the projected subject will appear at the periphery of the image. This possibility increases. Therefore, there is a high possibility that the effect cannot be obtained even if image correction processing for projecting the subject is performed. On the other hand, when the parallax range width is wide, a stereoscopic effect can be sufficiently felt as it is, and therefore image correction processing need not be performed. For the above reason, the condition is that the parallax range width is medium.

  In addition, the reason that there is a relatively small subject on the near view side is that if a very large subject is projected and seen, there is a feeling of pressure and it may be difficult to see. Also, the larger the subject, the higher the possibility that it will be applied to the peripheral part of the image, and there is a high possibility that the effect of increasing the force by popping the subject further forward will not be obtained.

  That is, the above processing selectively performs image correction processing on a scene that is considered to have an effect of image correction processing for projecting a subject.

  The adjustment amount in this shift process can be determined by the following method, for example.

  FIG. 13 shows a parallax histogram before and after the shift process in which the subject moves in the foreground direction. (A) is a parallax histogram before the shift process, and (b) is a parallax histogram after the shift process. The shaded portion in each parallax histogram is the near view side portion in the parallax histogram. By performing the shift process to move in the foreground direction, the parallax histogram moves to the left as a whole, and the value of the foreground side power sum after the shift process also changes. If the subject jumps out of the screen too much after the shift process, it will be difficult to see. Therefore, assuming that the foreground frequency sum after shift processing is S ′ and the predetermined threshold is Ts3, a shift amount where S ′ = Ts3 is obtained, and this is set as an adjustment amount a, thereby increasing the force due to the subject popping out. And the increase in difficulty of viewing.

  Ts3 can be set to 50% of the total frequency of the parallax histogram, for example. When S ′ is 50%, half of the subjects in the image after the shift processing will pop out from the screen and the other half will be behind the screen, so that the foreground and far side subjects are distributed in a well-balanced manner. It is.

  When the stereoscopic image is classified in other manners, that is, when the stereoscopic image is classified into any of the squares marked with “−” in FIG. 12, the image correction unit 7B does not perform the image correction process. Each of these has a medium or larger parallax range width, and a sufficient stereoscopic effect can be felt without performing image correction processing. Further, as in the case of “Processing B” in FIG. 12, the stereoscopic effect is further enhanced. This is because it is not a possible condition.

  As described above, the stereoscopic image processing apparatus 1B according to the present embodiment classifies a stereoscopic image into nine types of scenes using two types of parallax feature amounts, that is, the parallax range width W and the foreground-side power sum S. This has the effect of enabling more detailed and highly accurate classification of the parallax information of the stereoscopic image from various viewpoints.

  In addition, since the stereoscopic image processing apparatus 1B according to the present embodiment enables finer and more accurate classification, the stereoscopic image processing apparatus 1B causes the subject to jump out to the near side with respect to the stereoscopic image in which a relatively small subject exists on the foreground side. This makes it possible to perform image correction processing that increases the power. Accordingly, there is an effect that more various image correction processes can be appropriately performed in accordance with the contents of the scene as compared with the first embodiment. Further, there is an effect that the number of scenes to be subjected to the stereoscopic effect enhancement process is increased.

  The third embodiment of the present invention is a stereoscopic image processing apparatus, and in addition to the two types of image correction processing described in the second embodiment, there is a subject that protrudes to the foreground side at the periphery of the image. The object of the present invention is to perform image correction processing that relatively enhances the stereoscopic effect by moving the subject to the back to make it easier to feel the stereoscopic effect in a scene where it is difficult to feel the stereoscopic effect. For this reason, as compared with the second embodiment, scene classification is performed using a parallax feature amount that further indicates the proportion of the subject that protrudes to the near view side in the peripheral portion of the image in the peripheral portion of the image. In addition, by using this new parallax feature amount, the image correction processing for emphasizing the stereoscopic effect by popping out the subject described in the second embodiment can more accurately classify scenes to be processed. it can.

  FIG. 14 is a block diagram illustrating a configuration of a stereoscopic image processing apparatus 1C according to the third embodiment. The stereoscopic image processing apparatus 1B according to the second embodiment is different in a parallax feature amount calculation unit 4C, a stereoscopic image classification unit 5C, and an image correction unit 7C. The description about the same part is abbreviate | omitted, and it demonstrates centering on a different part.

  FIG. 15 is a block diagram illustrating a configuration of the parallax feature amount calculation unit 4C. Compared with the parallax feature amount calculation unit 4B, an image peripheral part parallax histogram generation unit 45 and an image peripheral part near view side frequency total calculation unit 46 are added.

  The image peripheral part parallax histogram generating unit 45 uses only the parallax data of the peripheral part of the screen among the parallax data of the entire image obtained by the parallax calculating unit 41, and calculates the image peripheral part parallax histogram having the frequency distribution. create. Alternatively, an image peripheral depth may be a histogram instead of the image peripheral parallax histogram. In other words, any display may be used as long as the parallax of the display object represented in the stereoscopic image or a frequency distribution of the same amount is represented. The image peripheral part parallax histogram generating unit 45 sends the image peripheral part parallax histogram to the image peripheral part foreground side frequency sum total calculating unit 46.

  FIG. 16 illustrates the peripheral portion of an image. The shaded portion is the peripheral portion of the image. For example, in an image having a horizontal size of 1920 pixels and a vertical length of 1080 pixels, a portion having a width of 100 pixels around the periphery of the image can be used as the peripheral portion of the image.

  The image peripheral portion foreground side frequency sum total calculation unit 46 uses the image peripheral portion parallax histogram input from the image peripheral portion parallax histogram generation unit 45 to calculate the image peripheral portion foreground side frequency total sum Se, which is the sum of the frequencies on the foreground side. This is obtained and output to the stereoscopic image classification unit 5C as a parallax feature amount. The calculation method is the same as the calculation method of the foreground side power sum S described above.

  On the other hand, the parallax range width calculation unit 43 and the foreground side power total calculation unit 44 output the parallax range width W and the foreground side power total S as the parallax feature amounts as described above. Therefore, the parallax feature amount calculation unit 4C outputs, as the parallax feature amount, three pieces of information of the parallax range width W, the foreground side power sum S, and the image peripheral portion foreground side power total Se to the stereoscopic image classification unit 5C.

  The stereoscopic image classifying unit 5C classifies the stereoscopic image using the parallax range width W, which is a parallax feature amount, the foreground side power sum S, and the image peripheral portion foreground side power Se. In the present embodiment, the parallax range width W is classified into three types, the foreground side power sum total S is classified into three types, and the image peripheral portion near view side power sum Se is classified into two types. Therefore, it can be classified into 3 × 3 × 2 = 18 types in total. The classification result is output to the image correction unit 7C.

  The classification method using the parallax range width W and the foreground-side power sum S is the same as the method described in the second embodiment.

The image peripheral portion near view side frequency total Se is classified as follows using a threshold value Tse1 for determination.
Se ≦ Tse1: A stereoscopic image in which no subject is present on the foreground side in the peripheral portion of the image. Se> Tse1: A stereoscopic image in which a subject is present on the foreground side in the peripheral portion of the image. In order to determine whether or not, the threshold value Tse1 is set to a small value. For example, setting Tse1 = 0 is the strictest condition setting. If the image peripheral portion histogram exists even in the foreground portion, it is determined that the subject exists in the foreground side in the image peripheral portion. In order to relax the condition, Tse1 may be set to a value larger than 0, for example, Tse1 may be set to 2% of the total frequency of the image peripheral parallax histogram.

  The image correction unit 7C performs image correction processing on a predetermined scene according to the stereoscopic image classification results classified into 18 types of scenes as described above. As in the first and second embodiments, the correction means uses a parallax adjustment method by shifting the left and right images.

  FIG. 17 is a table showing 18 types of scenes obtained by the above classification process. Of the three types of parallax-related information, the table is divided into (a) and (b) for two types of classification based on the image peripheral portion near view side frequency sum Se. Each square describes the type of image correction processing described later.

  When the stereoscopic image is classified as a scene with a narrow parallax range width, that is, when the stereoscopic image is classified into one of the squares described as “Processing A” in FIG. 17, the image correction unit 7 </ b> C shifts the left-eye image to the left. A shift process for shifting the right-eye image to the right is performed, and the parallax is increased uniformly over the entire screen. As described above with reference to FIG. 7, this shift process has the effect of widening the depth and enhancing the stereoscopic effect. The adjustment amount in this shift process can be determined by the same method as in the first embodiment.

  When the stereoscopic image is classified as a scene having a medium parallax range width, a relatively small subject on the foreground side, and no subject on the foreground side in the periphery of the image, that is, “Process B” in FIG. In the case where the image correction unit 7C is classified into the squares with "", the image correction unit 7C shifts the left-eye image to the right and shifts the right-eye image to the left to uniformly reduce the parallax over the entire screen. As described above with reference to FIG. 8, this shift processing has an effect that the entire subject in the image appears to move out to the near side.

  Here, the condition that the parallax range width is medium and that there is a relatively small subject on the foreground side is the same as in the case of “Process B” in FIG. 12 described in the second embodiment. Depending on the reason. Also, the condition that the subject does not exist on the foreground side in the peripheral portion of the image is that, as described above, the subject covers the peripheral portion of the image in the stereoscopic image, and the subject is perceived in front of the screen. This is because, when there is a large amount of parallax, there is a phenomenon that it is difficult to see the image by popping forward, regardless of the amount of parallax, and image correction processing is performed to make the subject pop out toward the front only when such a situation does not occur. In this way, compared to the case of “Processing B” in FIG. 12 in the second embodiment, an image that causes the subject to pop out toward the foreground by adding a condition that the subject does not exist on the foreground side in the peripheral portion of the image. It is possible to classify scenes suitable for performing the correction process with higher accuracy.

  The adjustment amount in this shift process can be determined by the following method, for example. By performing the shift process to move in the foreground direction, the parallax histogram is moved in the left direction as a whole, and the value of the foreground side power sum and the value of the image peripheral portion foreground side power sum after the shift process also change. If the subject jumps out of the screen too much after the shift process, it will be difficult to see. Further, if the subject is positioned in front of the screen at the periphery of the screen after the shift process, it is difficult to see. Therefore, the amount of adjustment is determined according to the conditions relating to the value of the foreground-side power sum total after the shift process and the value of the image periphery near-field side power sum total after the shift process.

  As for the foreground frequency summation after the shift processing, the adjustment amount can be determined by the processing using the threshold value Ts3, as in the case of the classification as “processing B” in FIG. 12 in the second embodiment. The adjustment amount thus obtained is set as a1. On the other hand, with respect to the value of the image peripheral portion foreground side frequency sum after the shift processing, Se ′ = Tse2 is obtained as a shift amount, where Se ′ is the image peripheral portion foreground side frequency sum after the shift processing and the predetermined threshold is Tse2. This is the adjustment amount a2. Tse2 uses a small value like Tse1. The final adjustment amount a needs to satisfy both the above-described condition relating to the value of the foreground side power sum after the shift process and the above-described condition relating to the value of the image peripheral portion near-field side power sum. Therefore, the smaller value of a1 and a2 is adopted as the adjustment amount a.

  When the stereoscopic image is classified as a scene having a medium or wide parallax range width and a subject on the foreground side in the periphery of the image, that is, one of the squares described as “Processing C” in FIG. When the images are classified, the image correcting unit 7C performs a shift process for shifting the left-eye image to the left and the right-eye image to the right to increase the parallax uniformly over the entire screen. As described above with reference to FIG. 7, this shift process has an effect that the subject appears to be in a distant view. This process is similar to the case where it is classified into one of the squares described as “Process A” in FIG. 17, but the adjustment amount is different. In this case, the shift process is performed with an adjustment amount necessary to move the subject existing on the near view side in the image periphery to the back of the screen. By this processing, at least the subject located in front of the screen does not exist at the periphery of the image. Therefore, if the subject is on the peripheral part of the image, even if the subject is located in front, the phenomenon of jumping out to the front and making it difficult to see does not occur, and it is easy to feel a stereoscopic effect An effect is obtained. In other words, an effect of relatively enhancing the stereoscopic effect can be obtained.

  Here, the condition that the subject exists on the foreground side in the peripheral portion of the image is that the subject is in the peripheral portion of the image in the stereoscopic image as described above, and the subject is perceived in front of the screen. This is because when there is a large amount of parallax, there is a phenomenon in which it is difficult to see the image by popping forward, regardless of the amount of parallax, so that image correction processing is performed to prevent this phenomenon from occurring. Also, the reason that the parallax range width is medium or wide is that a stereoscopic image with a narrow parallax range width has a poor stereoscopic effect in the first place. Therefore, priority is given to the image correction processing (processing A) that widens the depth. This is to apply.

  The adjustment amount in this shift process can be determined by the following method, for example. Since it is sufficient that the subject does not exist on the foreground side in the image periphery after the shift process, a predetermined threshold value is set as Tse3, and the shift amount at which the image periphery near view side frequency sum Se ′ = Tse3 after the shift process is obtained. The adjustment amount is a. Tse3 uses a small value like Tse1.

  When the stereoscopic image is classified in other ways, that is, when it is classified into one of the squares marked with “−” in FIG. 17, the image correction unit 7C does not perform the image correction process. All of these have a medium or larger parallax range width, and a sufficient stereoscopic effect can be felt without performing image correction processing. Further, as in the case of “Processing B” in FIG. Further, as in the case of “Processing C” in FIG. 17, there is a subject that exists in the foreground side at the periphery of the image, and this does not impede the stereoscopic effect.

  As described above, the stereoscopic image processing apparatus 1 </ b> C according to the present embodiment uses 18 types of stereoscopic images using the three types of parallax feature amounts of the parallax range width W, the foreground side power sum S, and the image peripheral portion foreground side power total Se. It is classified into the scene. This has the effect of enabling finer and more accurate classification than in the second embodiment.

  In addition, since the stereoscopic image processing apparatus 1C according to the present embodiment enables finer and more accurate classification, it is possible to perform image correction processing that emphasizes the stereoscopic effect by popping out the subject described in the second embodiment. There is an effect that the scene can be classified more accurately.

  Furthermore, the stereoscopic image processing apparatus 1C according to the present embodiment can classify whether or not the subject covers the peripheral portion of the image by using the image peripheral portion near view side frequency sum Se as the parallax feature amount. Thus, using this classification, it is possible to perform another image correction process for enhancing the stereoscopic effect. That is, there is an effect that the number of scenes to which the image correction process is applied increases.

  The fourth embodiment of the present invention is a stereoscopic image processing apparatus, and the correction processing is targeted for the image correction processing that emphasizes the stereoscopic effect by popping out the subject described in the second embodiment. The scene can be classified more accurately. For this reason, as compared with the second embodiment, scene classification is performed using another amount indicating the proportion of the subject existing in the foreground side in the image as the parallax feature amount.

  FIG. 18 is a block diagram illustrating a configuration of a stereoscopic image processing apparatus 1D according to the fourth embodiment. The stereoscopic image processing apparatus 1B according to the second embodiment is different in a parallax feature amount calculation unit 4D, a stereoscopic image classification unit 5D, and an image correction unit 7D. The description about the same part is abbreviate | omitted, and it demonstrates centering on a different part.

  FIG. 19 is a block diagram illustrating a configuration of the parallax feature amount calculation unit 4D. Compared with the parallax feature amount calculation unit 4B, a second foreground side frequency total calculation unit 47 is added.

  In the disparity histogram input from the disparity histogram creation unit 42, the foreground-side frequency sum total calculation unit 44 obtains a foreground side frequency summation S that is the sum of the frequencies of bins whose parallax amount is equal to or less than the threshold Td1, as in the second embodiment. This is output to the stereoscopic image classification unit 5D as a parallax feature amount. Further, the second foreground side frequency sum total calculation unit 47 obtains a second foreground side frequency total sum S2 that is the sum of the frequencies of bins whose parallax amount is equal to or less than the threshold Td2 in the parallax histogram input from the parallax histogram creation unit 42. Is output to the stereoscopic image classification unit 5D as a parallax feature amount. Here, Td2 <Td1.

  FIG. 20 is a diagram showing a parallax histogram for explaining the near-field side power sum S and the second foreground-side power sum S2. The horizontal axis is the amount of parallax, and the vertical axis is the frequency. A position where the amount of parallax is 0 is indicated by a solid line. The disparity amount of 0 or more is the far view side, and less than 0 is the close view side. An example of the positions of the threshold values Td1 and Td2 is indicated by a dotted line. FIGS. 20A and 20B are different in the shape of the near view side portion of the parallax histogram.

In each parallax histogram of FIG. 20, the sum of the portion shaded vertically and horizontally and the portion shaded horizontally corresponds to the near-field-side power sum S. Parallax histogram m from the parallax amount -l (l, m are natural numbers) is written in a range of the width of each bin of the parallax histogram 1, and the frequency of each bin _{h i (-l ≦ i ≦ m} ) Then, the foreground side power sum S can be obtained by the following equation as described above.
S = Σh _i (−1 ≦ i ≦ Td1)
On the other hand, the part shaded vertically and horizontally in the parallax histogram corresponds to the second foreground side power sum S2. This can be obtained by the following equation.
S2 = Σh _i (−1 ≦ i ≦ Td2)
Furthermore, FIG. 20 will be used to explain the values of the foreground-side power sum S and the second foreground-side power sum S2 depending on the shape of the parallax histogram. In FIG. 20, it is assumed that Td1 = −5 pixels and Td2 = −15 pixels. Further, the calculated values of S and S2 in FIG. 20A are Sa and S2a, and the calculated values of S and S2 in FIG. 20B are Sb and S2b. In addition, the parallax histogram in FIG. 20A and the parallax histogram in FIG. 20B have the same shape of the portion where the amount of parallax is larger than Td1, but the shape of the portion where the amount of parallax is smaller than Td1 is different. It is assumed that the peak has a lower height, but spreads closer to the foreground. Further, it is assumed that all frequencies of the parallax histogram are equal. Under the above conditions, the sum of the frequencies in the foreground area from Td1 is equal between (a) and (b). That is, Sa = Sb. Moreover, since Td2 <Td1, Sa ≧ S2a and Sb ≧ S2b.

  Here, for example, it is assumed that the value of Sa is 20% of the total frequency of the histogram, and S2a is 8% less than that. On the other hand, the value of Sb is 20% of the total frequency of the histogram as described above. On the other hand, the value of S2b is larger than S2a as is apparent from the figure, and is, for example, 12% of the total frequency of the histogram. That is, when the parallax histogram spreads further toward the foreground side, the value of the second foreground side frequency sum S2 becomes larger.

  Therefore, in this embodiment, by using the second foreground-side power sum S2 in addition to the foreground-side power sum S, the difference in the shape of the near-field parallax histogram can be determined as described above. This determination can be made because Td2 <Td1, that is, the second foreground side frequency sum S2 calculates the frequency sum of only the narrower portion on the foreground side than the foreground side frequency sum S. is there.

  As in the second embodiment, Td1 is generally set to 0 for Td1, and the total frequency up to parallax 0 of the parallax histogram is calculated. However, the present invention is not limited to this, and the frequency up to the parallax amount other than 0 is calculated. The sum may be calculated. Similarly, Td2 may be appropriately set within a range satisfying Td2 <Td1.

  On the other hand, the parallax range width calculation unit 43 outputs the parallax range width W as a parallax feature amount as described above. Accordingly, the parallax feature amount calculation unit 4D outputs, as the parallax feature amount, three pieces of information of the parallax range width W, the foreground side power sum S, and the second foreground side power sum S2 to the stereoscopic image classification unit 5D.

  The stereoscopic image classification unit 5D classifies the stereoscopic image using the parallax range width W, which is a parallax feature amount, the foreground-side power sum S, and the second foreground-side power sum S2. In the present embodiment, the size of the parallax range width W is classified into three types, and the foreground side power sum S and the second foreground side power sum S2 are combined and classified into two types. Therefore, it can be classified into 3 × 2 = 6 types in total. The classification result is output to the image correction unit 7D.

  The classification method using the parallax range width W is the same as the method described in the second embodiment.

The foreground side frequency sum S and the second foreground side frequency sum S2 are classified as follows using threshold values Ts3 and Ts4 (0 ≦ Ts3, 0 ≦ Ts4) for determination.
S> Ts3 and S2 <Ts4: A stereoscopic image in which there is a small subject on the foreground side and the subject does not protrude too far forward. Other: other stereoscopic images. Appropriate values for Ts3 and Ts4 are the near side frequency. Since it varies depending on the values of the thresholds Td1 and Td2 for calculating the sum S and the second foreground side power sum S2, it is necessary to set them appropriately. For example, as in the example in the description of FIG. 20, Td1 = −5 pixels and Td2 = −15 pixels, Sa = Sb = 20% of the total frequency of the histogram, S2a = 8%, and S2b = 12%. Suppose. On the other hand, assume that Ts3 = 15% of the total frequency of the histogram and Ts4 = 10% are used. Then, Sa> Ts3 and S2a <Ts4 for the parallax histogram of FIG. 20A, and Sb> Ts3 and S2b> Ts4 for the parallax histogram of FIG. 20B. Therefore, the stereoscopic image having the parallax histogram of FIG. 20A is classified as a stereoscopic image in which a small subject exists in the foreground side and the subject does not protrude too far, and has the parallax histogram of FIG. The stereoscopic image is classified as any other stereoscopic image. When the shapes of the parallax histograms in FIGS. 20A and 20B are compared, the subject in FIG. 20B protrudes closer to the foreground side. The above determination condition is for determining such a situation.

  The image correction unit 7D performs image correction processing on a predetermined scene according to the stereoscopic image classification result classified into the six types of scenes as described above. As in the first to third embodiments, the correction means uses a parallax adjustment method by shifting the left and right images.

  FIG. 21 is a table showing six types of scenes obtained by the classification process. Three classifications according to the parallax range width W are described in the row direction, and two classifications according to the foreground-side power sum S and the second foreground-side power sum S2 are described in the column direction, and a total of six types of scenes are shown in the matrix. Each square describes the type of image correction processing described later.

  When the stereoscopic image is classified as a scene having a narrow parallax range width, that is, when the stereoscopic image is classified into one of the squares described as “Processing A” in FIG. 21, the image correction unit 7D moves the left-eye image to the left. A shift process for shifting the right-eye image to the right is performed, and the parallax is increased uniformly over the entire screen. As described above with reference to FIG. 7, this shift process has the effect of widening the depth and enhancing the stereoscopic effect. The adjustment amount in this shift process can be determined by the same method as in the first embodiment.

  When a stereoscopic image is classified as having a medium parallax range width, a small subject on the near side, and the subject not protruding too far, that is, “Processing B” in FIG. 21 is described. When the image is classified into squares, the image correction unit 7D performs a shift process for shifting the left-eye image to the right and the right-eye image to the left, thereby reducing the parallax uniformly over the entire screen. As described above with reference to FIG. 8, this shift processing has an effect that the entire subject in the image appears to move out to the near side.

  Here, the condition that the parallax range is medium and that there is a relatively small subject on the foreground side is the same as the case of “Process B” in FIG. 12 described in the second embodiment. Depending on the reason. In addition, the condition is that the small subject on the foreground side does not jump out too far, but if the subject has already popped out in the stereoscopic image before the image correction processing, image correction processing for further popping out the subject is performed. This is because it is likely to be difficult to see. In this way, compared to the case of “Processing B” in FIG. 12 in the second embodiment, by adding a condition that the small subject on the foreground side does not jump too far forward, the subject jumps forward. It is possible to classify scenes that are appropriate to be subjected to image correction processing with higher accuracy.

  The adjustment amount in this shift process can be determined by the same method as in the case of “Process B” in FIG. 17 in the third embodiment.

  When the stereoscopic image is classified in other ways, that is, when the stereoscopic image is classified into any of the squares marked with “−” in FIG. 21, the image correction unit 7D does not perform the image correction process. All of these have a medium or larger parallax range width, and a sufficient stereoscopic effect can be felt without performing image correction processing. Further, this is because it is not a condition for further enhancing the stereoscopic effect as in the case of “Processing B” in FIG.

  As described above, the stereoscopic image processing apparatus 1D according to the present embodiment converts six types of stereoscopic images using the three types of parallax feature amounts of the parallax range width W, the foreground side power sum S, and the second foreground side power sum S2. Classified into scenes. Accordingly, there is an effect that it is possible to more accurately classify a scene to be subjected to an image correction process in which the subject described in the second embodiment is projected and the stereoscopic effect is enhanced.

  In a fifth embodiment of the present invention, a scene change detection unit using a scene classification result is added to the stereoscopic image processing apparatus according to the third embodiment. As a result, when the stereoscopic image is a moving image, it is possible to accurately detect a scene change and perform image correction processing more appropriately.

  FIG. 22 is a block diagram illustrating a configuration of a stereoscopic image processing apparatus 1E according to the fifth embodiment. The image correction unit 7E is different from the stereoscopic image processing apparatus 1C according to the third embodiment, and a scene change detection unit 12 is added. The description about the same part is abbreviate | omitted, and it demonstrates centering on a different part and the added part.

  FIG. 23 is a block diagram illustrating a configuration of the scene change detection unit 12. The scene change detection unit 12 includes a stereoscopic image classification result delay unit 121 and a stereoscopic image classification result comparison unit 122. The scene change detection unit 12 determines whether or not a scene change has occurred in the stereoscopic image using the 18 types of stereoscopic image classification results input from the stereoscopic image classification unit 5C, and the scene change that is the result of the determination. The detection flag Fs is output to the image correction unit 7E.

  The stereoscopic image classification result delay unit 121 stores the input stereoscopic image classification result, and then inputs the stereoscopic image classification result delayed by one frame with respect to the input stereoscopic image classification result to the stereoscopic image classification result comparison unit 122. send.

  The stereoscopic image classification result comparison unit 122 inputs the input stereoscopic image classification result and the stereoscopic image classification result delayed by one frame by the stereoscopic image classification result delay unit 121, and whether there is a change in the classification result between frames. Determine whether. That is, as shown in FIG. 23, it is determined whether or not the (n + 1) th frame stereoscopic image classification result is different from the nth frame stereoscopic image classification result. If the two stereoscopic image classification results are different, 1 is set to the scene change detection flag Fs. This means that a scene change has been detected between n frames and (n + 1) frames of the stereoscopic image. On the other hand, when the two stereoscopic image classification results are the same, 0 is set to Fs. This means that no scene change has been detected.

  Similar to the image correction unit 7C, the image correction unit 7E performs image correction processing on a predetermined classification according to the stereoscopic image classification results classified into 18 types by the stereoscopic image classification unit 5C. As in the first to fourth embodiments, the correction means uses a parallax adjustment method by shifting the left and right images.

  Here, when the stereoscopic image is a moving image, the parallax of the subject changes from frame to frame. Therefore, basically, the amount of parallax adjustment is appropriately set according to the situation of the parallax distribution of the stereoscopic image for each frame. That is, it is desirable to change the shift amount of the parallax adjustment processing for each frame. However, if the change in the shift amount for each frame is large, the depth is frequently changed, which makes it difficult for the observer to follow it, which makes it difficult to see. Furthermore, when an error occurs in the parallax detection process, a change in the shift amount due to the error is added to make it more difficult to see. For this, for example, the shift amount may be smoothed by performing low-pass filter processing on the shift amount in the time direction.

  On the other hand, the parallax of the subject of the stereoscopic image may change discontinuously before and after the scene change. In this case, in order to display both the scene before and after the scene change with a rich stereoscopic effect, even if the change in the shift amount at the time of the scene change becomes large, the shift amount suitable for each scene before and after the scene change It is desirable to use Therefore, when the scene change detection flag Fs input from the scene change detection unit 12 is 1, that is, when a scene change has occurred, the image correction unit 7E temporarily cancels the above-described low-pass filter processing, and shift amount Is allowed to change discontinuously. By performing such processing, it is possible to prevent unsightly by changing the shift amount smoothly by low-pass filter processing for the shift amount except during a scene change, and if a scene change occurs, after the scene change It is possible to realize parallax adjustment processing that is compatible with quickly changing the shift amount of a stereoscopic image. That is, more appropriate parallax adjustment processing can be realized.

  Further, for example, there is a conventional technique in which scene change detection is performed using a change in luminance information of an image, but this method cannot capture a change in the amount of parallax. When the parallax adjustment processing is performed as in the stereoscopic image processing apparatus 1D of the present embodiment, it is desirable to use a scene change detection method that can capture a change in the amount of parallax. For example, if there is no significant change in luminance information before and after a scene change but there is a change in the amount of parallax, it can be detected only by a scene change detection method that can capture the change in the amount of parallax. The scene change detection means by the scene change detection unit 12 enables scene change detection that captures a change in the amount of parallax.

  The stereoscopic image classification unit 5C classifies the input stereoscopic image into 18 types, but the image correction unit 7E has four types of “processing A”, “processing B”, “processing C”, and “−” (no processing). A normal image correction process is performed. That is, some of the 18 types of classifications are grouped together to perform the same image correction process. However, in the scene change, the original 18 types of classification are used, and when there is a change, the scene change is determined. This is because even if the image is classified as a stereoscopic image subjected to the same image correction processing, the correction amount may be different at the time of a scene change, so if the classification is different, it should be determined as a scene change. It is.

  As described above, the stereoscopic image processing apparatus 1E according to the present embodiment performs scene change detection using the scene classification result of the stereoscopic image. This has the effect of making it possible to detect scene changes that capture changes in the parallax of a subject in a stereoscopic image.

  Further, the use of the scene change detection result for the image correction process for the stereoscopic image has an effect that a more appropriate parallax adjustment process can be realized.

  By the way, in this embodiment, the scene change is detected by looking at the change in the stereoscopic image classification result of the (n + 1) th frame of the stereoscopic image and the change of the stereoscopic image classification result of the nth frame. However, the detection method is not limited to this. For example, the change of the (n + 1) th frame stereoscopic image classification result and the (n−1) th frame stereoscopic image classification result, the (n + 1) th frame stereoscopic image classification result, and the (n−2) th frame. It is also possible to detect a scene change by a change in the stereoscopic image classification result between two or more frames, as in the change in the stereoscopic image classification result. In such a case, a scene change between a frame that is later in time and a frame immediately before the two frames used for scene change detection is detected. For example, when a scene change is detected based on a change in the (n + 1) th frame stereoscopic image classification result and the (n−1) th frame stereoscopic image classification result, the (n + 1) th frame later in time and the previous one A scene change between the nth frame and the nth frame is detected.

  The relationship between the frame used for scene change detection and the position of the scene change detected thereby will be further described with reference to FIG. FIG. 24 shows the frames from the (n−2) frame to the (n + 2) frame in chronological order. (N-2) frame to n frame is one scene, there is a scene change between n frame and (n + 1) frame, the stereoscopic image classification result changes, and (n + 1) frame to (n + 2) frame is another scene The situation is illustrated.

When a scene change is detected by a change in the stereoscopic image classification result between adjacent frames, over time,
(1) Judgment of change in stereoscopic image classification result between (n-2) frame and (n-1) frame (2) Judgment of change in stereoscopic image classification result between (n-1) frame and n frame (3) Judgment of change in stereoscopic image classification result between n frames and (n + 1) frames
The process is performed in order. Since there is no change in the processes (1) and (2) and there is a change in the process (3), a scene change is detected in the process (3).

On the other hand, when a scene change is detected based on the distance of multi-dimensional statistics between one frame apart, over time,
(1 ′) Determine change in stereoscopic image classification result between (n−2) frame and n frame (2 ′) Determine change in stereoscopic image classification result between (n−1) frame and (n + 1) frame (3 ') Judgment of change in stereoscopic image classification result between n frames and (n + 2) frames ...
The process is performed in order. Since there is no change in the process (1 ′) and there is a change in the process (2 ′), a scene change is detected in the process (2 ′). In the next process (3 ′), there is a change because the n frame and the (n + 2) frame belong to different scenes. As described above, when a scene change is detected based on a change in the stereoscopic image classification result between frames that are separated by one, there is a change in the stereoscopic image classification result twice for one scene change. Since this is an erroneous detection, for example, when a scene change is detected in a process at a certain time, the process result in the next frame can be ignored.

  As described above, it is possible to detect a scene change based on a change in the stereoscopic image classification result regarding the parallax between a plurality of frames.

  A stereoscopic image processing apparatus 1E according to the present embodiment is obtained by adding a scene change detection unit 12 to the stereoscopic image processing apparatus 1C according to the third embodiment. Similarly, the scene change detection unit 12 is added to the stereoscopic image processing apparatus 1A in the first embodiment, the stereoscopic image processing apparatus 1B in the second embodiment, and the stereoscopic image processing apparatus 1D in the fourth embodiment. It is also possible. In these cases as well, a scene change can be detected by a change in the stereoscopic image classification result.

  In the first to fifth embodiments, the correction process for enhancing the stereoscopic effect has been described. However, in practice, not only such a stereoscopic effect enhancement process, but also a correction process for ensuring the safety of a stereoscopic image as in the prior art may be performed.

  In the parallax range calculation unit 43 in the first to fifth embodiments, the parallax range width is obtained from the parallax amount of the closest view and the parallax amount of the farthest view in the image from the parallax histogram. However, the method of obtaining the parallax range width is not limited to this. For example, the parallax amount of the nearest scene and the parallax amount of the farthest view may be searched directly from the parallax map of the stereoscopic image, and the width may be set as the parallax range width. In other words, whatever method is used, the width of the parallax amount of the most recent view and the parallax amount of the farthest view of the stereoscopic image may be obtained and used as the parallax range width.

  In the above first to fifth embodiments, the parallax feature amount is calculated from the parallax map. However, parallax feature amount data may be added as additional information of the input stereoscopic image data. In this case, the process of obtaining the parallax feature amount in the parallax feature amount calculation units 4A to 4D may be skipped.

  In the first to fifth embodiments described above, when the stereoscopic image data is a still image, the same image is continuously output and handled in the same manner as a moving image. However, the present invention is not limited to this, and when the stereoscopic image data is a still image, the still image may be processed one by one.

  In said 1st-5th Example, the liquid crystal display panel which displays the image for left eyes and the image for right eyes alternately as the display part 9 is combined with the shutter glasses 11, and the three-dimensional image is shown. However, the stereoscopic image display method is not limited to this. A right-eye image and a left-eye image may be displayed using different polarization methods, and a method of observing with glasses having different polarization filters may be used. In addition, a display unit using a lenticular lens or a parallax barrier may be used, and a method that enables stereoscopic viewing without an observer wearing glasses may be used.

  In the first to fifth embodiments described above, the scene classification result of the stereoscopic image is used for performing a parallax adjustment process that enhances the stereoscopic effect. However, the scene classification result of the stereoscopic image according to the present embodiment can be used for other purposes. For example, the above three-dimensional image classification function classifies still images of a plurality of three-dimensional images into scenes, displays thumbnails of the three-dimensional images for each scene, and displays what three-dimensional images to the observer. It is also possible to use a list of whether the scene belongs. Further, it is possible to use such that the images are rearranged and displayed so that the change in parallax between the stereoscopic images is reduced. In addition, when a plurality of stereoscopic images are automatically switched and displayed sequentially, stereoscopic images with poor stereoscopic effect are automatically skipped and displayed, so that only a stereoscopic image with a rich stereoscopic effect is presented to the observer. It is also possible to use it. Furthermore, the above-mentioned stereoscopic image classification function is used for a stereoscopic image recording device, a stereoscopic image editing device, a stereoscopic image editing program, etc., and automatically indexing the scene change points of the stereoscopic image, when reproducing or editing the stereoscopic image. It can also be used as a mark.

  In the second to fifth embodiments described above, the foreground side power sum calculation unit 44 or the second foreground side power sum calculation unit 47 uses the parallax histogram created by the parallax histogram creation unit 42 to use the foreground side power sum S. Or 2nd foreground side frequency sum total S2 was computed. However, the foreground-side power sum S or the like may be calculated directly from the parallax data of the entire image obtained by the parallax calculation unit 41. For example, when the parallax data of the entire image is parallax data for each pixel of the stereoscopic image, the foreground-side power sum S can be calculated by counting the number of pixels whose parallax is Td1 or less.

  In the third and fifth embodiments described above, the image peripheral portion near view side frequency using the image peripheral portion parallax histogram created by the image peripheral portion parallax histogram creating portion 45 in the image peripheral portion foreground side frequency sum total calculating portion 46. Total Se was calculated. However, it is also possible to directly calculate the image peripheral portion near view side frequency total Se using the parallax data of the peripheral portion of the image among the parallax data of the entire image obtained by the parallax calculation unit 41.

  In the first to fifth embodiments, the stereoscopic image processing apparatus has been described. However, the present invention can be applied to a wide range of apparatuses and methods related to stereoscopic images, such as a stereoscopic image display apparatus, a stereoscopic image editing apparatus, a stereoscopic image processing method, a stereoscopic image display method, and a stereoscopic image editing method.

  The first to fifth embodiments have been specifically described above, but the present invention is not limited to them. Embodiments obtained by appropriately combining the technical means disclosed in the five examples described above are also included in the technical scope of the present invention.

  Further, in each of the above-described embodiments, the configuration and the like illustrated in the accompanying drawings are merely examples, and are not limited thereto, and may be appropriately changed within the scope of the effects of the present invention. Is possible. In addition, various modifications can be made without departing from the scope of the object of the present invention.

  In the description of each of the above embodiments, each component for realizing the function is described as being a different part, but it should actually have a part that can be clearly separated and recognized in this way. It doesn't have to be. The stereoscopic image processing apparatus that implements the functions of the above embodiments may configure each component for realizing the function using, for example, different parts, or all the components. May be mounted on a single LSI. That is, what kind of mounting form should just have each component as a function.

  In addition, a program for realizing the functions described in the above embodiments is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. Processing may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  INDUSTRIAL APPLICABILITY The present invention can be used in a wide range of apparatuses and methods related to stereoscopic images, such as stereoscopic image processing apparatuses, stereoscopic image display apparatuses, stereoscopic image editing apparatuses, stereoscopic image processing methods, stereoscopic image display methods, and stereoscopic image editing methods. .

1A to 1E Stereoscopic image processing device 2 Input unit 3 Stereoscopic image processing units 4A to 4D Parallax feature amount calculation units 5A to 5D Stereoscopic image classification unit 6 Image delay units 7A to 7E Image correction unit 8 Display control unit 9 Display unit 10 Glasses synchronization Unit 11 shutter glasses 12 scene change detection unit 41 parallax calculation unit 42 parallax histogram creation unit 43 parallax range width calculation unit 44 foreground side frequency sum total calculation unit 45 image peripheral part parallax histogram generation unit 46 image peripheral part near background side frequency total calculation unit 47 Second foreground side frequency sum total calculation unit 121 Stereo image classification result delay unit 122 Stereo image classification result comparison unit

Claims (15)

A stereoscopic image processing apparatus for classifying a stereoscopic image scene,
An input unit for inputting the stereoscopic image;
A parallax feature amount calculation unit that calculates one or more parallax feature amounts of the stereoscopic image from the stereoscopic image;
A stereoscopic image classification unit that classifies the stereoscopic image into a plurality of scenes based on the parallax feature amount;
A stereoscopic image processing device in which at least one of the parallax feature amounts is calculated from two parallax amounts .   The stereoscopic image processing apparatus according to claim 1, further comprising an image correction unit that performs a stereoscopic image correction process based on the scene.   The stereoscopic image processing apparatus according to claim 2, wherein the parallax feature amount is a parallax range width that is a width of a parallax amount of the most recent view and a parallax amount of the farthest view of the stereoscopic image. A stereoscopic image processing apparatus for classifying a stereoscopic image scene,
An input unit for inputting the stereoscopic image;
A parallax feature amount calculation unit that calculates one or more parallax feature amounts of the stereoscopic image from the stereoscopic image;
A stereoscopic image classification unit that classifies the stereoscopic image into a plurality of scenes based on the parallax feature amount;
The stereoscopic image processing apparatus, wherein the parallax feature amount is a foreground-side power sum determined by a near-field parallax amount from a first constant related to the parallax amount. A stereoscopic image processing apparatus for classifying a stereoscopic image scene,
An input unit for inputting the stereoscopic image;
A parallax feature amount calculation unit that calculates one or more parallax feature amounts of the stereoscopic image from the stereoscopic image;
And a stereoscopic image classification section for classifying the three-dimensional image into a plurality of scenes by the parallax feature amount, the parallax feature amount is viewed from the second constants for smaller parallax amount than the first constant related Saryou the foreground side A stereoscopic image processing apparatus that is a second foreground side power sum determined by the amount of parallax. A stereoscopic image processing apparatus for classifying a stereoscopic image scene,
An input unit for inputting the stereoscopic image;
A parallax feature amount calculation unit that calculates one or more parallax feature amounts of the stereoscopic image from the stereoscopic image;
A stereoscopic image classification unit that classifies the stereoscopic image into a plurality of scenes based on the parallax feature amount;
The stereoscopic image processing device, wherein the parallax feature amount is an image peripheral portion foreground side frequency sum determined by a parallax amount in the peripheral portion of the stereoscopic image closer to the foreground than a third constant related to the parallax amount. The stereoscopic image classification unit performs a process of classifying the stereoscopic image by comparing the parallax range width with a first parallax range width constant;
The stereoscopic image processing apparatus according to claim 3, wherein the image correction unit performs a parallax adjustment process for increasing a parallax of the stereoscopic image in a scene in which the parallax range width is smaller than the first parallax range width constant. The stereoscopic image classification unit compares the parallax range width with a first parallax range width constant and a second parallax range width constant larger than that, and classifies the stereoscopic image, and a first constant related to a parallax amount Performing a process of classifying the stereoscopic image by comparing the near-field side power sum determined by the near-field parallax amount with the first foreground-side power sum constant and a larger second foreground-side power sum constant,
The image correction processing unit is a scene in which the parallax range width is larger than the first parallax range width constant and smaller than the second parallax range width constant, and the near-field side power sum is the first foreground. The stereoscopic image processing apparatus according to claim 3, wherein a parallax adjustment process for reducing parallax of the stereoscopic image is performed on a scene that is larger than a side power sum constant and smaller than the second foreground side power sum constant. The stereoscopic image classifying unit compares the parallax range width with a first parallax range width constant and the second parallax range width constant larger than the first parallax range width constant , From the process of comparing and classifying the near-field-side power sum constant determined by the disparity amount on the foreground side from the constant and the third near-field side power sum constant, and the second constant related to the disparity amount smaller than the first constant related to the disparity amount Performing a process of classifying the stereoscopic image by comparing the second foreground side power sum determined by the amount of parallax on the foreground side and the fourth foreground side power sum constant,
The image correction processing unit is a scene in which the parallax range width is larger than the first parallax range width constant and smaller than the second parallax range width constant, and the foreground-side power sum is the third
A parallax adjustment process for reducing the parallax of the stereoscopic image is performed on a scene that is larger than the foreground side power sum constant of the second scene and the second foreground side power sum is smaller than the fourth foreground side power sum constant. Item 4. The stereoscopic image processing apparatus according to Item 3. The stereoscopic image classification unit compares the parallax range width with a first parallax range width constant to classify the stereoscopic image, and a peripheral portion of the stereoscopic image closer to the foreground than the third constant related to the amount of parallax Performing a process of classifying the stereoscopic image by comparing the image peripheral portion near view side power sum total determined by the amount of parallax in the first image peripheral portion near view side power sum constant,
The image correction processing unit is a scene in which the parallax range width is larger than the first parallax range width constant, and the image peripheral portion foreground side power sum is larger than the first image peripheral portion foreground side power sum constant. The stereoscopic image processing apparatus according to claim 3, wherein a parallax adjustment process for increasing a parallax of the stereoscopic image is performed on a scene.   The stereoscopic image processing apparatus according to claim 1, further comprising a scene change detection unit that detects a scene change of the stereoscopic image based on a change in the scene. A stereoscopic image processing method for classifying a stereoscopic image scene,
Inputting the stereoscopic image;
Calculating one or more parallax feature quantities of the stereoscopic image from the stereoscopic image;
Classifying the stereoscopic image into a plurality of scenes according to the parallax feature amount, and
A stereoscopic image processing method in which at least one of the parallax feature amounts is calculated from two parallax amounts . A stereoscopic image processing method for classifying a stereoscopic image scene,
Inputting one or more parallax feature quantities of the stereoscopic image from the stereoscopic image;
Classifying the stereoscopic image into a plurality of scenes according to the parallax feature amount, and
Wherein at least one of the parallax feature amount, the three-dimensional image processing method which has been calculated from the two parallax amount. A program for causing a computer to execute stereoscopic image processing for classifying a stereoscopic image scene,
Inputting the stereoscopic image;
Calculating one or more parallax feature quantities of the stereoscopic image from the stereoscopic image;
Classifying the stereoscopic image into a plurality of scenes according to the parallax feature amount, and
At least one of the parallax feature amounts is a program calculated from two parallax amounts . A program for causing a computer to execute stereoscopic image processing for classifying a stereoscopic image scene,
Inputting one or more parallax feature quantities of the stereoscopic image from the stereoscopic image;
Classifying the stereoscopic image into a plurality of scenes according to the parallax feature amount, and
At least one of the parallax feature amounts is a program calculated from two parallax amounts.