JP2006332985A - Stereoscopic image format decision device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereoscopic image format decision device capable of automatically deciding a stereoscopic image format. <P>SOLUTION: The stereoscopic image format decision device has an image data division section which divides an image to be decided into two images by a specified division system and a similarity decision section which decides the similarity between the two images, and decides the format of the image to be decided by the division system when the similarity decision section has decided that the two images are similar to each other. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a stereoscopic image format determination apparatus that determines the format of a stereoscopic image.

  Conventionally, various studies have been made on stereoscopic photographs and three-dimensional displays that enable stereoscopic display by displaying different images on the left and right eyes. In the old days, there is a method of adjusting the position of the lens of a camera, arranging a photograph corresponding to what is viewed from the left eye of a human and a photograph corresponding to viewing from the right eye, and viewing it through a binocular adapter.

  Recently, by combining a video camera and a liquid crystal shutter during recording, the right-eye image and the left-eye image are alternately recorded for each frame, and during playback, glasses using two liquid crystal shutters instead of the left and right binocular lenses are used. By turning the liquid crystal shutter on and off alternately, a method that enables stereoscopic display and a slit plate that controls the transmission of light on the front of the liquid crystal panel are installed, and stereoscopic vision can be achieved with the naked eye without special glasses. There are 3D displays that make it possible.

  On the other hand, since a general-purpose stereoscopic image format for describing a stereoscopic image to be displayed on a stereoscopic display is not specifically defined, a left-eye image and a right-eye image are stored in separate image files as a stereoscopic image storage method. As a three-dimensional image, or a method of separately recording the correspondence between the left-eye image and the right-eye image constituting the stereoscopic image, or a composite image of the left-eye image and the right-eye image as a single image. The stereoscopic image is stored by various methods such as a method for storing the image as a stored image.

  In general, when the left-eye image and the right-eye image are stored as separate image files as a stereoscopic image storage method, there is a problem that one of the images is missing or the correspondence between the left and right images is not known. Often, a combination of the image for the right eye and the image for the right eye as a single image is treated as a three-dimensional image, but there is a special rule on how to combine the left and right images as a single image. Does not exist.

  Here, the synthesis method when the left and right images are synthesized into one image is defined by the phrase “stereoscopic image format” and used in the following description.

  As an example, in a stereoscopic image format in which a left-eye image and a right-eye image are combined to form a single image, a left-eye image and a right-eye image are arranged side by side, A method is used in which an image having the same resolution as a one-eye image is reduced in half in the direction.

  In the method of compressing in half in the horizontal direction, the amount of image information per eye is halved, but there is an advantage that an existing moving image compression technique such as MPEG can be used for stereoscopic moving images. In MPEG, since the resolution of images that can be handled is limited, MPEG cannot be used in a composition method in which left and right images are simply arranged as a three-dimensional image, but it can be reduced to the same resolution as a one-eye image. Thus, it becomes possible to handle stereoscopic images using MPEG.

Furthermore, when combining the left-eye image and the right-eye image, not only the left-right arrangement format arranged side by side, but also the vertical arrangement format, the vertical arrangement image compression format, the field sequential method, The method of arranging the left-eye image and the right-eye image alternately in the vertical direction for each line, the method of alternately arranging the left-eye image and the right-eye image in the horizontal direction so that they can be displayed as they are on the autostereoscopic display, etc. A variety of stereoscopic image formats are used as a method for describing a stereoscopic image.
JP 2004-104368 A

  In order to display such various formats of stereoscopic images on various stereoscopic displays, the left-eye image component and the right-eye image component are extracted from the stereoscopic image that is the display source, and then displayed on the stereoscopic display. It is necessary to convert to the format.

  For example, Japanese Patent Application Laid-Open No. 2004-104368 proposes a conversion method for various formats of stereoscopic images.

  However, in the conventional method, it is necessary for the user to determine the stereoscopic image format of the stereoscopic image to be converted in advance and add identification information of the stereoscopic image.

  When there are a large number of stereoscopic images to be displayed, it is very troublesome to visually determine which stereoscopic image format each stereoscopic image has for each of the stereoscopic images and add identification information. It takes a lot of work.

  In addition, it is difficult for a user who has no knowledge of stereoscopic vision to correctly determine the type of input image.

  In addition, since a field for storing stereoscopic image identification information is not provided in the commonly used 2D image format, it is necessary to use a unique image format when adding stereoscopic image identification information. Also, there is a problem that compatibility with the existing 2D image format cannot be achieved.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a stereoscopic image format discrimination device that can automatically discriminate a stereoscopic image format.

  A stereoscopic image format determination apparatus according to the present invention includes an image data division unit that divides an image to be determined into two images by a predetermined division method, a similarity determination unit that determines the similarity between the two images, When the similarity determination unit determines that the two images are similar, the format of the determination target image is determined based on the division method.

  When the image data dividing unit divides the determination target image into an image composed of a left half and a right half, and the similarity determination unit determines that the two images are similar, the determination target image is It is determined that the stereoscopic image data is a left-right composition method.

  When the image data dividing unit divides the image to be determined into images composed of an upper half and a lower half, and the similarity determining unit determines that the two images are similar, the image to be determined is It is determined that the image is stereoscopic image data of the up-and-down composition method.

  When the image data dividing unit divides the image to be determined into an image having an odd field and an even field, and the similarity determining unit determines that the two images are similar, the image to be determined is It is determined that it is field sequential type stereoscopic image data.

  When the image data dividing unit divides the image to be determined into images composed of odd lines and even lines, and the similarity determining unit determines that the two images are similar, the image to be determined is It is determined that the image data is vertical stripe type stereoscopic image data.

  According to the present invention, the stereoscopic image format can be automatically determined.

  An example of the stereoscopic image format will be described with reference to FIGS. In the example shown in FIG. 1A, the left-eye image 101 and the right-eye image 102 are arranged side by side, and one composite image 103 is generated as it is. Hereinafter, this image is referred to as a left / right composite image 103. The horizontal dimension of the left and right composite image 103 is twice the horizontal dimension of the original images 101 and 102.

  An image 104 shown in FIG. 1B is generated by reducing the left-right composite image 103 of FIG. 1A in half in the horizontal direction. Hereinafter, this image is referred to as a left / right composite reduced image 104. The horizontal dimension of the left and right composite reduced image 104 is equal to the horizontal dimension of the original images 101 and 102 shown in FIG.

  In the example shown in FIG. 2A, the left-eye image 201 and the right-eye image 202 are arranged one above the other and a single composite image 203 is generated as it is. Hereinafter, this image is referred to as an upper / lower composite image 203. The vertical dimension of the upper and lower composite image 203 is twice the vertical dimension of the original images 201 and 202.

  An image 204 shown in FIG. 2B is generated by reducing the vertical composite image 203 of FIG. 2A in half in the vertical direction. Hereinafter, this image is referred to as an upper / lower composite reduced image 204. The vertical dimension of the vertically combined reduced image 204 is equal to the vertical dimension of the original images 201 and 202 shown in FIG.

  FIG. 3 shows a field sequential image. As shown in FIG. 3A, a left-eye image is generated by a set of odd fields 301 of a field sequential image, and a right-eye image is generated by a set of even fields 302. Therefore, as shown in FIG. 3B, one field sequential image 303 includes a left-eye image and a right-eye image.

  FIG. 4 shows a vertical stripe image. As shown in FIG. 4A, a left-eye image is generated by a set of odd lines 401 in a vertical stripe image, and a right-eye image is generated by a set of even lines 402. Accordingly, as shown in FIG. 4B, one vertical stripe image 403 includes a left-eye image and a right-eye image.

  With reference to FIG. 5, the structure of the stereoscopic image format determination apparatus according to the present invention will be described. The stereoscopic image format determination apparatus of this example includes an image data input unit 501 that inputs stereoscopic image data from a hard disk, a network, and the like, an image data dividing unit 502 that divides the input image data into two images, and an image data dividing unit. A feature amount calculation unit 503 that calculates the feature amounts of the image data divided into two, a similarity determination unit 504 that determines the similarity between two images from the two feature amounts, and whether the image is reduced It has a reduction determination unit 505 for determining whether or not.

  With reference to FIG. 6, the flow of processing in the stereoscopic image format determination apparatus according to the present invention will be described. In step S1, it is determined whether or not the input stereoscopic image data is the left-right composite image shown in FIG. First, the image data dividing unit 502 divides the input stereoscopic image data into two parts, a left half and a right half. Next, the feature amount calculation unit 503 calculates the feature amounts of the two divided image data. The similarity determination unit 504 calculates the similarity based on the feature amount and determines whether the two images are similar. There is a difference in the amount of information corresponding to parallax between the left-eye image and the right-eye image, but they are basically similar. Therefore, if the two divided images are similar, it is determined that the input stereoscopic image data is the left-right composite image shown in FIG.

  If it is determined in step S1 that the input stereoscopic image data is a left-right composite image, in step S2, the reduction determination unit 505 determines that the left-right composite image is the left-right composite reduced image shown in FIG. It is determined whether or not. The processing in the reduction determination unit 505 will be described later.

  If it is determined in step S1 that the input stereoscopic image data is not a left-right composite image, the process proceeds to step S3.

  In step S3, it is determined whether or not the input stereoscopic image data is the upper and lower composite image shown in FIG. First, the image data dividing unit 502 divides the input stereoscopic image data into two parts, an upper half and a lower half. Next, the feature amount calculation unit 503 calculates the feature amounts of the two divided image data. The similarity determination unit 504 calculates the similarity based on the feature amount and determines whether the two images are similar. There is a difference in the amount of information corresponding to parallax between the left-eye image and the right-eye image, but they are basically similar. Accordingly, if the two divided images are similar, it is determined that the input stereoscopic image data is the upper and lower composite image shown in FIG.

  When it is determined in step S3 that the input stereoscopic image data is an upper / lower composite image, in step S4, the reduction determination unit 505 determines that the upper / lower composite image is the upper / lower composite reduced image shown in FIG. It is determined whether or not. The processing in the reduction determination unit 505 will be described later.

  If it is determined in step S3 that the input stereoscopic image data is not a vertically synthesized image, the process proceeds to step S5.

  In step S5, it is determined whether or not the input stereoscopic image data is the field sequential image shown in FIG. First, the image data dividing unit 502 divides the input stereoscopic image data into an odd field image and an even field image. Next, the feature amount calculation unit 503 calculates the feature amounts of the two divided image data. The similarity determination unit 504 calculates the similarity based on the feature amount and determines whether the two images are similar. Since there is a difference in information amount corresponding to parallax between the left-eye image and the right-eye image, if the two divided images are not similar, the input stereoscopic image data is the field shown in FIG. It is determined that the image is a sequential image. If the two divided images are similar, it is determined that the input stereoscopic image data is not the field sequential image shown in FIG. In this case, the process proceeds to step S6.

  Here, if the image for the left eye and the image for the right eye are similar, if it is determined that the image is a field sequential image, the 2D image is also determined to be a field sequential image. Therefore, in step S5, steps S1 and S3 are performed. On the contrary, since the left-eye image and the right-eye image are not similar, it is determined as a field sequential image.

  In step S6, it is determined whether or not the input stereoscopic image data is the vertical stripe image shown in FIG. First, the image data dividing unit 502 divides the input stereoscopic image data into an odd line image and an even line image. Next, the feature amount calculation unit 503 calculates the feature amounts of the two divided image data. The similarity determination unit 504 calculates the similarity based on the feature amount and determines whether the two images are similar. Since there is a difference in information amount corresponding to the parallax between the left-eye image and the right-eye image, if the two divided images are not similar, the input stereoscopic image data is displayed in the vertical direction shown in FIG. It is determined that the image is a stripe image. If the two divided images are similar, it is determined that the input stereoscopic image data is not the vertical stripe image shown in FIG.

  Here, if it is determined that the image for the left eye is similar to the image for the right eye and it is determined that the image is a vertical stripe image, the 2D image is also determined to be a vertical stripe image. Contrary to S3, the image for the left eye and the image for the right eye are not similar, so that it is determined as a vertical stripe image.

  If it is determined in step S6 that the image is not a vertical stripe image, the stereoscopic image format of the input image is unknown. In general, when there are a plurality of stereoscopic image data, it is highly likely that they are all in the same stereoscopic image format. Therefore, when it is determined that the stereoscopic image format is unknown, it may be determined that it is the same as the stereoscopic image format determined immediately before. Or you may determine with it being a 2D image.

  In this example, the input stereoscopic image data is the left-right composite image in FIG. 1A, the left-right composite reduced image in FIG. 1B, the top-bottom composite image in FIG. 2A, and the top-bottom composite in FIG. Although it can be determined that the image is any one of the reduced image, the field sequential image in FIG. 3 and the vertical stripe image in FIG. 4, a step of determining another stereoscopic image format is further performed under step S6. By connecting, another stereoscopic image format can be determined.

  For example, a stereoscopic image format in which left and right images are arranged obliquely can be determined in the same manner, and it is easy to increase the types of stereoscopic image formats to be determined.

  The processing in the stereoscopic image format determination apparatus in FIG. 6 can be executed by a computer. The present invention may be configured as a computer-readable program for causing a computer to execute the processing of FIG.

  With reference to FIG. 7, the operation of the feature amount calculation unit 503 will be described. The feature amount calculation unit 503 calculates the feature vectors (FV) of the two image data divided by the image data division unit 502 as feature amounts, respectively. As shown in FIG. 7A, one pixel P (x, y) in the image data is composed of sub-pixels having RGB color information. Accordingly, the color information of each sub-pixel of one pixel P (x, y) is R (x, y), G (x, y), and B (x, y).

  Next, as shown in FIG. 7B, the entire image is divided into nine areas by dividing the image vertically and horizontally into three equal parts. Numbers Ai (i = 1 to 9) are assigned to the nine areas sequentially from the upper left to the lower right in the horizontal direction. Then, color averages R_i, G_i, and B_i are calculated for each area Ai (i = 1 to 9). From the color average for each area Ai, a 27-dimensional feature vector FV = (R_1, G_1, B_1, R_2, G_2, B_3,..., R_9, G_9, B_9) is determined.

  Next, the operation of the similarity determination unit 504 will be described. The similarity determination unit 504 normalizes the feature vector FV_l of one image data and the feature vector FV_r of the other image data, and sets the inner product as the similarity sim. The similarity sim is expressed by the following equation.

  The similarity sim takes a value between 0 and 1 (0 ≦ sim ≦ 1). The closer the similarity sim is to 1, the more similar the two images are, and the similarity is 1 (sim = 1) ) Means that the two images are completely coincident. As described above, there is a difference in information amount corresponding to parallax between the left-eye image and the right-eye image, but they are basically similar. The similarity determination unit 504 determines that the two images are similar if the similarity sim is equal to or greater than a predetermined value.

  Here, the predetermined value is statistically determined. In the present invention, the inner product of the feature vectors for determining the similarity between two images is calculated, but other methods may be used as long as the similarity between two images can be determined.

  Here, the process of determining whether or not the image is a vertical stripe image in step S6 will be further described. As described above, the image data dividing unit 502 divides the input stereoscopic image data into an odd line image and an even line image. As shown in FIG. 7A, one pixel is composed of RGB sub-pixels arranged in the horizontal direction. Accordingly, the odd lines and the even lines may each be one pixel (one dot) wide, but may be one subpixel wide. In this case, the input image data is divided into odd lines and even lines for each RGB subpixel, and an odd line image and an even line image are generated for each RGB subpixel. Further, for each RGB sub-pixel, it is determined whether the odd-numbered line image and the even-numbered line image are similar to each other.

  Next, the operation of the reduction determination unit 505 will be described with reference to FIGS. First, with reference to FIG. 8, the process for determining whether or not the input image data is a left-right composite reduced image in step S2 of FIG. 6 will be described. Here, when the aspect ratio of the determination target image is smaller than a predetermined threshold, it is determined that the image is a left-right combined reduced image.

  In general, the aspect ratio (horizontal: vertical or horizontal / vertical) of an image is 4: 3 for a standard television image, 16: 9 for a high-definition image, and 3: 2 for a 35 mm film. FIG. 8 shows a high-definition left-right composite image 103H generated from a high-definition left-right composite image 103H, a standard left-right composite image 103S generated from a standard image (4: 3), and a high-definition left-right composite image 103H. A standard left-right composite reduced image 104S generated by reducing the reduced image 104H and the standard left-right composite image 103S is shown.

  As a threshold for determining whether the determination target image is the left / right composite image 103S, 103H or the left / right composite reduced image 104H, 104S, the image having the aspect ratio of the most horizontally long image of the image group to be determined and the most vertically long image A value less than twice the aspect ratio is set. In the example of FIG. 8, the aspect ratio of the high-definition image is set to 16: 9 or more and the aspect ratio of the standard left / right composite image 103S is set to less than 8: 3.

  For example, an aspect ratio of 16: 9 is set as the threshold value. An image having an aspect ratio larger than 16: 9 is determined to be a combined image that is not reduced, and an image having an aspect ratio smaller than 16: 9 is determined to be a reduced combined image.

  Next, with reference to FIG. 9, the process for determining whether or not the input image data is a vertically synthesized reduced image in step S4 of FIG. 6 will be described. Here, when the aspect ratio of the determination target image is larger than a predetermined threshold, it is determined that the image is a vertically combined reduced image. FIG. 9 shows a high-definition vertical composite image 203H generated from a high-definition image (16: 9), a standard vertical composite image 203S generated from a standard image (4: 3), and a high-definition vertical composite image 203H. A standard vertical composite reduced image 204S generated by reducing the reduced image 204H and the standard vertical composite image 203S is shown.

  As a threshold for determining whether the determination target image is the upper / lower composite image 203S, 203H or the upper / lower composite reduced image 204H, 204S, the image having the aspect ratio of the most vertically long image of the image group to be determined and the most horizontally long image A value less than twice the aspect ratio is set. In the example of FIG. 9, the aspect ratio of the standard image is set to 4: 3 or more and the aspect ratio of the high-definition upper / lower composite image 203H is set to less than 16:18.

  For example, an aspect ratio of 4: 3 is set as the threshold value. An image with an aspect ratio greater than 4: 3 is determined to be a reduced composite image, and an image with an aspect ratio less than 4: 3 is determined to be a non-reduced composite image.

  In this embodiment, an image having an aspect ratio of 4: 3, 16: 9, or 3: 2 that is generally used is assumed as an image to be determined. However, the image is reduced depending on the aspect ratio of the image to be determined. It is necessary to change the threshold for determining whether or not.

  The example of the present invention has been described above, but the present invention is not limited to the above-described example, and various modifications can be made by those skilled in the art within the scope of the invention described in the claims. It will be understood.

  Using the stereoscopic image format determination device of the present invention, for example, a web browser capable of stereoscopically displaying a website in which stereoscopic images are collected on different stereoscopic displays can be created. It can be implemented by analyzing the format of the stereoscopic image data in the website and converting the stereoscopic image in accordance with the stereoscopic display to be displayed.

It is a figure explaining the left-right synthesized image and the left-right synthesized reduced image. It is a figure explaining an up-and-down synthesized image and an up-and-down synthesized reduced image. It is a figure explaining a field sequential image. It is a figure explaining a vertical stripe image. It is a system configuration | structure figure of the stereo image format determination apparatus of this invention. It is a flowchart figure which shows the flow of a process of the stereo image format determination apparatus of this invention. It is a figure explaining the image feature-value of this invention. It is a figure explaining the aspect-ratio of a right-and-left composite image. It is a figure explaining the aspect ratio of an up-and-down synthetic image.

Explanation of symbols

501... Image data input unit, 502... Image data division unit, 503... Feature amount calculation unit, 504 .. similarity determination unit, 505.

Claims (12)

  An image data dividing unit that divides the image to be determined into two images by a predetermined dividing method; and a similarity determining unit that determines the similarity between the two images. The similarity determining unit A stereoscopic image format determination device, wherein when it is determined that the two images are similar, the format of the determination target image is determined based on the division method.   The stereoscopic image format determination apparatus according to claim 1, wherein the image data dividing unit divides the image to be determined into an image composed of a left half and a right half, and the similarity determination unit is similar to the two images. A stereoscopic image format determination apparatus, wherein the determination target image is determined to be stereoscopic image data of a left-right synthesis method.   The stereoscopic image format determination apparatus according to claim 1, wherein the image data dividing unit divides the image to be determined into an image composed of an upper half and a lower half, and the similarity determination unit is similar to the two images. A stereoscopic image format determining apparatus that determines that the determination target image is stereoscopic image data of an up-and-down composition method.   2. The stereoscopic image format determination device according to claim 1, wherein the image data dividing unit divides the image to be determined into an image composed of an odd field and an even field, and the similarity determination unit is similar to the two images. A stereoscopic image format determining apparatus that determines that the determination target image is stereoscopic image data of a field sequential method.   The stereoscopic image format determination apparatus according to claim 1, wherein the image data dividing unit divides the image to be determined into an image composed of odd lines and even lines, and the similarity determination unit is similar to the two images. A stereoscopic image format determination apparatus, wherein the determination target image is determined to be vertical stripe type stereoscopic image data.   The stereoscopic image format determination apparatus according to claim 1, further comprising a reduction determination unit that determines whether or not the image to be determined is reduced, and the reduction determination unit is a stereoscopic image in which the determination target image is a left-right composite method. When it is determined as data, it is determined whether or not the image is reduced in the horizontal direction. When it is determined that the determination target image is stereoscopic image data of the up-and-down composition method, it is determined whether or not the image is reduced in the vertical direction. A stereoscopic image format determination apparatus characterized by determining.   7. The stereoscopic image format determination apparatus according to claim 6, wherein the reduction determination unit reduces the determination target image in the horizontal direction when an aspect ratio of the determination target image is smaller than a horizontal reduction determination threshold. A stereoscopic image format determination device characterized in that determination is made as follows.   8. The stereoscopic image format determination device according to claim 7, wherein the threshold value for determination of reduction in the horizontal direction is a value that is greater than or equal to the aspect ratio of the most horizontally long image and less than twice the aspect ratio of the most vertically long image among the determination target images. A stereoscopic image format determination device characterized by   The stereoscopic image format determination apparatus according to claim 6, wherein the reduction determination unit reduces the determination target image in the vertical direction when an aspect ratio of the determination target image is larger than a vertical reduction determination threshold. A stereoscopic image format determination device characterized in that determination is made as follows.   10. The stereoscopic image format determination apparatus according to claim 9, wherein the threshold value for vertical reduction determination is a value that is not less than the aspect ratio of the most vertically long image and less than twice the aspect ratio of the most horizontally long image among the determination target images. A stereoscopic image format determination device characterized by   2. The stereoscopic image format determination apparatus according to claim 1, further comprising a feature amount calculation unit that respectively calculates a feature amount of the image data divided into two by the image data division unit, and the similarity determination unit includes the feature amount And determining a similarity between the two images based on the three-dimensional image format determination apparatus.   2. The stereoscopic image format determining apparatus according to claim 1, wherein the feature amount is a feature vector of an image, and the similarity is an inner product of feature vectors of two images.

Images