JP3789794B2 - Stereoscopic image processing method, apparatus, and system - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to stereoscopic image processing technology, and more particularly, to a method, apparatus, system, and related computer program and data structure for processing or displaying a stereoscopic image.
[0002]
[Prior art]
Over the past few years, the Internet population has increased rapidly and is entering a broadband era, a new stage of Internet usage. In broadband communication, the communication band is dramatically widened, so the distribution of heavy image data, which has been often avoided in the past, is also popular. Concepts such as “multimedia” and “video on demand” have been proposed for a long time. However, in the broadband era, these words can only be experienced by ordinary users.
[0003]
If the distribution of images, especially moving images, spreads, the user naturally wants to enrich the content and improve the image quality. These are largely due to the digitization of existing video software, the development of an authoring tool therefor, and the pursuit of high-efficiency and low-loss image coding technology.
[0004]
[Problems to be solved by the invention]
Under such circumstances, as one form of image distribution service in the near future, distribution of pseudo three-dimensional images (hereinafter also simply referred to as “stereoscopic images”) has attracted technical attention, and it is considered that a considerable market is acquired. Stereoscopic images fulfill the user's desire for more realistic images, and are especially attractive for applications that pursue a sense of reality, such as movies and games. Furthermore, stereoscopic images are also useful for the realistic display of products in product presentations in EC (electronic commerce), which is considered to be one of the standards for commerce in the 21st century.
[0005]
However, when considering a new net business of stereoscopic image distribution, it may be said that neither an infrastructure for that purpose nor a model for business promotion exists yet. The present inventor has made the present invention paying attention to such a current situation, and an object of the present invention is to provide a stereoscopic image processing technique for enabling the promotion of distribution of stereoscopic images from a technical aspect.
[0006]
[Means for Solving the Problems]
In order to understand the present invention, the following concepts in this specification are first defined.
“Stereoscopic image”: The term “stereoscopic image” refers to an image that is projected on the user's eyes as a result of being displayed stereoscopically rather than image data itself. Image data that can be displayed as a stereoscopic image is called a “multiplexed image” to be described later. That is, when a multiplex image is displayed, a stereoscopic image can be seen.
“Parallax image”: Usually, for stereoscopic viewing with a sense of depth, an image that should be thrown to the right eye (hereinafter simply referred to as a right-eye image) and an image that should be cast to the left eye (hereinafter simply referred to as a left-eye image) so as to generate parallax. It is necessary to prepare. A pair of images that cause parallax, such as a right-eye image and a left-eye image, may be collectively referred to as a parallax image. In this specification, each image that causes a parallax is referred to as a parallax image. That is, both the right eye image and the left eye image are parallax images. In addition to these, generally, images from the respective viewpoints assumed in the stereoscopic image are parallax images.
[0007]
“Basic image”: An image that is subject to processing necessary for stereoscopic vision or an image that has already been processed in order to display a stereoscopic image. As a specific example, in addition to a parallax image in a separate format, which will be described later, an image obtained by combining a plurality of parallax images in some form, such as a multiplex format or a side-by-side format (these are also referred to as “synthesized images”) including.
“Side-by-side format”: One of the basic image configurations. A format in which multiple parallax images are combined in the horizontal direction, vertical direction, or both directions. Usually, the thinned parallax images are juxtaposed. For example, when two parallax images are arranged side by side in the horizontal direction, the respective parallax images are thinned out pixel by pixel in the horizontal direction. The basic image in the side-by-side format is also simply referred to as “side-by-side image”.
[0008]
"Multiplex format": One of the basic image composition modes. Final image data format for displaying stereoscopic images. A basic image in a multiplex format is also simply referred to as a “multiplex image”.
“Separate format”: One of the basic image configurations. Although it is a single two-dimensional image, it is assumed that it is stereoscopically viewed in combination with other two-dimensional images, and indicates each of the plurality of two-dimensional images. The basic image of “separate format” is also simply referred to as “separate image”. Separate images are not composite images, unlike multiplexed images and side-by-side images.
[0009]
“Viewpoint”: A stereoscopic image is assumed to have a viewpoint of viewing it. The number of viewpoints and the number of parallax images are usually equal. When there are two parallax images of a left-eye image and a right-eye image, the number of viewpoints is “2”. However, even if there are two viewpoints, the assumed position of the user's head is one. Similarly, when displaying a stereoscopic image considering the movement of the user in the left-right direction, for example, assuming four viewpoints va, vb, vc, vd in the left-right direction, the parallax images that can be seen from each are Ia, Ib, Ic, Id Then, for example, a stereoscopic image with a sense of depth can be displayed by three sets of parallax images (Ia, Ib), (Ib, Ic), and (Ic, Id). In this state, if four images viewed from the upper direction and four images viewed from the lower direction are used in order to generate a stereoscopic image that wraps in the vertical direction, The number of viewpoints is “8”.
[0010]
Based on the above definition, an aspect of the present invention relates to a stereoscopic image processing method. This method can be understood as a technique on the encoding side that should be called a starting point of distribution of a stereoscopic image. This method adds information (hereinafter also referred to as “stereoscopic information”) to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image to the basic image for displaying the stereoscopic image. is there.
[0011]
“Append” may be incorporated into the basic image, may be incorporated into the header or other area of the basic image, or may be incorporated into another file associated with the basic image. What is necessary is just to establish a relationship. An example of a “predetermined scene in a series of processes for displaying a stereoscopic image” is a scene where, for example, a side-by-side image is converted into a multiplexed image.
[0012]
According to this method, a stereoscopic image can be displayed by an appropriate method by referring to the stereoscopic information. If a large number of basic images are prepared by this method, various information terminals can take out the data and display them stereoscopically, and this method works as a basic technology for distributing stereoscopic images. This method can be used in, for example, a stereoscopic image server.
[0013]
Another aspect of the present invention relates to the structure of image data generated by the method described above. This data structure includes main data of a basic image for displaying a stereoscopic image, information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, sub-data holding stereoscopic information, It is formed as a combination. The main data may be a basic image compressed by a predetermined method. The “combination” is not limited to the case where the two are integrated, and it is only necessary that some kind of association is established between the two. According to this data structure, as described above, stereoscopic display is easily realized on the display side.
[0014]
Still another embodiment of the present invention also relates to a stereoscopic image processing method. This method can be understood as a technique for interpreting and using the above-described data structure, that is, generally a decoding technique for displaying a stereoscopic image. This method detects information to be referred to in a predetermined scene in a series of processing for displaying a stereoscopic image, that is, stereoscopic information added to a basic image for displaying the stereoscopic image. In order to facilitate detection, the stereoscopic information may be added according to a predetermined format agreed with the encoding side in advance. In this method, the configuration of the basic image may be converted into another based on the detected stereoscopic information.
[0015]
Still another embodiment of the present invention also relates to a stereoscopic image processing method. In this method, when a stereoscopic image is handled by a device having a memory, a basic image having a mode different from the configuration of the basic image finally displayed on the screen is stored in the memory, and is appropriately read out. To use. For example, even if a multiplex image is used to display a stereoscopic image, the side-by-side format may be more advantageous than the multiplex format when it is desired to perform enlargement or other processing on the stereoscopic image. In that case, if the side-by-side image is held in the memory, the processing speed can be increased.
[0016]
Still another embodiment of the present invention also relates to a stereoscopic image processing method. This method detects information to be referred to in a predetermined scene in a series of processing for displaying a stereoscopic image, that is, stereoscopic information, added to a basic image for displaying the stereoscopic image, and detects the information. The brightness of the stereoscopic image display screen is adjusted based on the original. For example, information regarding the number of viewpoints assumed in the stereoscopic image may be entered as the stereoscopic information, and the luminance may be adjusted according to the number.
[0017]
If the number of viewpoints is “4”, a multiplexed image is formed by combining four parallax images. The number of pixels that can be seen from any one of the four viewpoints is ¼ that of a normal two-dimensional image. Therefore, the brightness of the screen is theoretically 1/4. For this reason, the process of increasing the brightness of the screen of the display device according to the number of viewpoints is effective. This process is performed by, for example, cooperation between software for detecting the number of viewpoints and a circuit for adjusting luminance.
[0018]
Yet another embodiment of the present invention relates to a stereoscopic image processing apparatus. This device is on the encoding side for preparing a stereoscopic image, and includes an image acquisition unit that acquires a basic image for displaying a stereoscopic image, and a series for displaying the stereoscopic image on the acquired basic image. Information to be referred to in a predetermined scene in the process, that is, an information adding unit for adding three-dimensional information. The image acquisition unit may generate a basic image by itself or input a ready-made basic image.
[0019]
Still another embodiment of the present invention also relates to a stereoscopic image processing apparatus. This device actually displays a stereoscopic image, or performs preprocessing therefor, an image acquisition unit for acquiring a basic image for displaying a stereoscopic image, and an attached to the acquired basic image, It includes information to be referenced in a predetermined scene in a series of processes for displaying the stereoscopic image, that is, an information detection unit that detects stereoscopic information. The image acquisition unit acquires or inputs a basic image via a network or a recording medium, for example. The image acquisition unit may include an image input unit that inputs pre-compressed image data, and an image expansion unit that generates a basic image by expanding the input image data. And a brightness adjusting unit for adjusting the brightness of the display screen of the stereoscopic image.
[0020]
Yet another embodiment of the present invention relates to a stereoscopic image processing system. This system includes a stereoscopic image synthesizing device and a display device, and the synthesizing device provides information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image on the basic image for displaying the stereoscopic image. That is, the three-dimensional information is incorporated, and the display device detects the three-dimensional information, performs appropriate image processing on the basic image based on the three-dimensional information, and displays the three-dimensional image. As an example of image processing, there is a change in the configuration of the basic image. The system may be a server / client system. This system can contribute to the promotion of the distribution of stereoscopic images.
[0021]
Still another embodiment of the present invention relates to a stereoscopic image processing method. In this method, a basic image for displaying a stereoscopic image is acquired, and information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image by examining a part of the basic image, that is, Three-dimensional information is estimated. Unlike the cases described so far, here, the processing when the stereoscopic information is not explicitly added is considered. Therefore, a part of the basic image is actually inspected. As an example, by examining several regions on the base image, it is estimated whether this is a side-by-side image.
[0022]
According to this method, even when stereoscopic information is not explicitly given, it can be known from the basic image. Therefore, past images created by a general method can be used, and software assets can be effectively used.
[0023]
It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium, a transmission medium, etc. are also effective as an aspect of the present invention.
[0024]
DETAILED DESCRIPTION OF THE INVENTION
When an image is displayed on the LCD, the minimum unit of display is usually a dot. However, three dots corresponding to RGB are gathered to form one pixel, and in normal image display or image processing, the pixel is recognized as the minimum unit of processing.
[0025]
However, when a stereoscopic image is displayed on the LCD, another consideration is required. The right eye image and the left eye image reach the user's eyes with parallax through an optical filter such as a lenticular lens or a parallax barrier. If the left and right eye images are alternately arranged in pixel units, that is, in units of 3 dots, an area where both the right eye image can be seen and an area where only the left eye image can be seen is generated, and the colors are mixed, It becomes very difficult to see. For this reason, an alternate arrangement in units of dots, which is a physical minimum display element, is desirable. Therefore, a multiplex image in which right-eye images and left-eye images are alternately arranged in units of dots is often used as a basic image to be stereoscopically displayed.
[0026]
When the parallax image is composed of only two images of the right eye image and the left eye image, that is, when the number of horizontal viewpoints is “2”, it is sufficient for the multiplex image to arrange the right eye image and the left eye image in stripes in dot units. However, when displaying a stereoscopic image in consideration of horizontal viewpoint movement using four parallax images with the number of viewpoints being “4”, the parallax barrier placed in front of the screen 10 as shown in FIG. 12, only the dots of the corresponding parallax images can be seen from the first to fourth viewpoints VP1 to VP4. On the screen 10, “1” is given to the dots of the first parallax image corresponding to the first viewpoint VP1, and the same applies to the following viewpoints. In this example, the first to fourth parallax images are sequentially arranged in stripes in dot units, and a multiplexed image is formed.
[0027]
Further, when considering the viewpoint movement also in the vertical direction, the parallax barrier 12 becomes a pinhole arranged in a matrix rather than a stripe shape, and the multiplexed image also becomes a matrix shape that is replaced in units of dots. FIG. 2 shows an example of the multiplexed image 20 when the number of horizontal viewpoints and the number of vertical viewpoints are both “4”. Here, the region denoted by (i, j) indicates dots that should be visible from the i-th viewpoint in the horizontal direction and the j-th viewpoint in the vertical direction. As shown in the figure, in the horizontal direction, i is cyclically changed to 1, 2, 3, 4, 1,..., And similarly, j is 1, 2, 3, 4, 1, in the vertical direction. , ... and changes cyclically.
[0028]
In consideration of promoting the use of stereoscopic images, the multiplex image 20 shown in FIG. 2 may be transmitted to a necessary terminal. The multiplex image 20 is already in the final format for stereoscopic viewing, so it can be simply displayed on the terminal side. Of course, at this time, the presence of an optical filter such as a parallax barrier is assumed for stereoscopic viewing.
[0029]
However, the inventor has recognized that one problem arises here. That is, the image data should naturally be compressed at the time of transmission. However, in the case of the multiplexed image 20, the ordinary irreversible compression represented by JPEG (Joint Photographic Expert Group) is practically unavailable. This is because a plurality of multiplexed parallax images are images of different viewpoints, so they are essentially irrelevant when considered at the pixel level, and when compressed by a method that relies on a spatial frequency such as JPEG, each Despite the use of the parallax images that are independent from the viewpoint, high-frequency components are removed between these images, and as a result, correct stereoscopic display cannot be performed. In particular, when independent images are alternately arranged in units of pixels, a large number of very fine high-frequency components are generated, so this problem is sometimes fatal. Even if the network bandwidth is widened, if a normal image can be compressed without any problem, if only the basic image for a stereoscopic image cannot be compressed, it will be a drag on the spread.
[0030]
Therefore, it is conceivable that the usage of side-by-side images increases as a format that can be compressed in the case of transmission or storage. FIG. 3 shows a side-by-side image 30 having four viewpoints both horizontally and vertically. Here, the region denoted by (i, j) indicates one parallax image that should be seen from the i-th viewpoint in the horizontal direction and the j-th viewpoint in the vertical direction. That is, the side-by-side image 30 is a combination of parallax images arranged side by side in one direction or both in the horizontal or vertical directions, and each parallax image functions as a single image if it is cut out from the side-by-side image 30.
[0031]
However, since each parallax image only needs to correspond to one of 4 × 4 = 16 viewpoints, it may be 1/16 of the image size to be displayed as a stereoscopic image, and is usually an original size of the same size as the stereoscopic image. From the image, one dot is generated every four dots for each of the horizontal direction and the vertical direction. In an easy-to-understand example, in the case of a multiplex image consisting only of a right-eye image and a left-eye image with the number of viewpoints “2”, only the odd-numbered dots need to be seen from the right eye, and only the even-numbered dots can be seen from the left eye. That's fine. Accordingly, the right-eye image may be any image in which only the odd-numbered columns are extracted from the original image in advance and thinned in half in the horizontal direction, and only the even-numbered columns are similarly extracted from the left-eye image. In general, if the number of viewpoints is “n”, each parallax image constituting the side-by-side image may be 1 / n of the original image size, and if all the parallax images are juxtaposed like tiles, the original image size is restored.
[0032]
In the case of the side-by-side image 30, since each parallax image is independent except for its boundary, even if irreversible compression is performed, an adverse effect occurs only at the boundary portion. For this reason, the side-by-side 30 is usually compressed by JPEG or the like, and can be easily transmitted via a network, or a large number of small storages can be stored. As described above, the side-by-side image 30 is preferable in terms of popularization, but there is also a drawback, which is that a special viewer is required. That is, in any display device, stereoscopic display is not possible unless it is finally converted to a multiplex image, and conversion processing from the side-by-side image 30 to the multiplex image is necessary.
[0033]
In addition to the two formats having the advantages and disadvantages described above, a separate image is conceivable as a third format from the viewpoint of dissemination, particularly image preparation. A separate image can form a stereoscopic image as an aggregate, but it is merely a normal two-dimensional image by itself. FIG. 4 shows the relationship between 16 separate images and a stereoscopic image. Of the 16 images, for example, the separate image 32 described as “viewpoint (4, 2) image” is assumed to have the viewpoint (4, 2), and the image size is the same as the original image. Therefore, the 16 separate images can be easily understood if they are considered as camera images taken while the user is moving.
[0034]
As described above, since the size of the separate image may remain as it is at the time of shooting, it does not require processing such as thinning or combining, and preparation is easy. In addition, since each image remains in its original state, it can be used separately. However, stereoscopic display requires a total of 16 parallax images, which is disadvantageous in terms of transmission and storage, and also requires a special viewer.
[0035]
The above are the three main formats considered for the spread of stereoscopic images. These modifications will be described at the end. Hereinafter, a stereoscopic image processing method for technically realizing spread promotion and stereoscopic display when the basic image is expressed in any of these three formats will be described. In the following, for the sake of simplicity, a case where the number of horizontal viewpoints is “2” and the number of vertical viewpoints is “1” will be exemplified.
[0036]
5, 6, and 7 schematically show the data structures of the side-by-side image 40, the multiplex image 50, and two separate images 60 and 62 among the basic images according to the present embodiment.
[0037]
As shown in FIG. 5, the side-by-side image 40 is obtained by horizontally combining a first parallax image 44 that is a left-eye image and a second parallax image 46 that is a right-eye image, and a header area 42 described later is added to the image data. Has been. Similarly, a header area 42 according to the same format is added to the multiplexed image 50 in FIG. A header area 42 is added to each of the two separate images 60 and 62 shown in FIG. In any case, this data structure includes main data that is a basic image for displaying a stereoscopic image and secondary information that holds information to be referred to in a predetermined scene in a series of processes for displaying the stereoscopic image. It can be considered as a combination with data. If the main data is a basic image compressed by a predetermined method, the sub data may be stored in a header area defined by the compression method. If the header area definition already exists, for example, a user-defined area can be used.
[0038]
FIG. 8 schematically shows a detailed configuration of the header area 42. In the figure, each region holds the following three-dimensional information.
(1) DIM area 70: The dimensions and configuration of the basic image are indicated by 3 bits.
000: Separate images or 2D images that cannot be viewed stereoscopically
001: Multiplex image among 3D images
010: Side-by-side image of 3D image
011: Reserve
1xx: Reserve
Examples of the reserved format include “joint images” in which a plurality of parallax images are juxtaposed as in a side-by-side image, but the parallax images are juxtaposed as they are, and even fields and odd fields are juxtaposed. A “field sequential image” indicating that a parallax image should be displayed alternately in a time division manner is conceivable. “Joint images” are often observed by the parallel method or the intersection method, but they can be converted to side-by-side images by thinning them out in the viewer, or they can be converted directly into multiplexed images, so they are effective as one format. .
[0039]
(2) BDL area 72: 1 bit indicates the presence or absence of side-by-side image boundary processing. Meaningful when DIM is "01x".
0: No boundary processing
1: With boundary processing
As described above, when a side-by-side image is irreversibly compressed, the image is adversely affected at the boundary portion. In order to reduce this, it is shown whether or not the processing shown in the next section has been performed.
[0040]
(3) HDL region 74: 2 bits indicate the contents of the side-by-side image boundary processing. Meaningful when BDL is "1".
00: Insert a white frame
01: Insert a black frame
10: Copy edge pixels
11: Reserve
In order to reduce the adverse effects due to compression, a white frame, a black frame, or the like is added to the boundary portion to reduce mixing of a plurality of parallax images. Copying the end pixels has the same effect.
[0041]
(4) WDT area 76: The number of pixels for the boundary processing of the side-by-side image is designated by 2 bits. Meaningful when BDL is "1".
00-11: Number of pixels
(5) VPH area 78: 8 bits indicate the number of horizontal viewpoints assumed for a stereoscopic image. It may be described manually when creating the basic image, or may be automatically generated by software that generates the basic image.
00000000: Unknown or reserved
00000001-11111111: Number of horizontal viewpoints
(6) VPV area 80: 8 bits indicate the number of vertical viewpoints assumed for a stereoscopic image.
00000000: Unknown or reserved
00000001-11111111: Number of vertical viewpoints
When both VPH and VPV are 00000001, the basic image may be determined as a normal two-dimensional image that cannot be stereoscopically viewed.
[0042]
(7) ODH area 82: One bit indicates the arrangement of a plurality of parallax images in the horizontal direction.
0: Same as the camera array at the time of shooting
1: Reverse of camera alignment
That is, when ODH is “0”, the parallax image captured by the leftmost camera at the time of shooting is recorded as it is on the left side of the basic image as it is, and is recorded in order thereafter. Since it is unlikely that the normal arrangement is random, two types of rules are sufficient.
[0043]
(8) ODV area 84: One bit indicates the arrangement of a plurality of parallax images in the vertical direction.
0: Same as the camera array at the time of shooting
1: Reverse of camera alignment
It should be noted that VPH and VPV related to the number of viewpoints are both 8 bits, and it is considered that they are usually sufficient. Therefore, these most significant bits may be assigned to ODH and ODV, respectively.
[0044]
(9) PSH area 86: 8 bits indicate each viewpoint image in the horizontal direction. Meaningful when DIM is "000".
00000000: Unknown or reserved
00000001-11111111: Horizontal position
Note that an absolute value such as the origin determined on each separate image, for example, the coordinates of the upper left corner point of the image, may be included in the header area 42 separately, which leads to a higher processing speed.
[0045]
(10) PSV area 88: 8 bits indicate each viewpoint image in the vertical direction.
00000000: Unknown or reserved
00000001-11111111: Vertical position
For example, a separate image labeled “viewpoint (4, 2) image” in FIG. 4 has a description of PSH = 4 and PSV = 2.
[0046]
The above is an example of the header area 42. An apparatus for realizing the distribution of stereoscopic images using this area will be described.
FIG. 9 shows the configuration of the image processing apparatus 100 that generates this region. The apparatus 100 includes an image acquisition unit 102 that acquires a basic image for displaying a stereoscopic image, and a predetermined scene in a series of processes for displaying a stereoscopic image on the acquired basic image, for example, a display side later And an information adding unit 104 for adding stereoscopic information to be referred to in a scene where a multiplex image is generated by the apparatus. This configuration can be realized in hardware by a CPU, memory, or other LSI of any computer, and in software, it is realized by a program with a basic image generation function and a stereoscopic information addition function loaded into the memory. However, here, functional blocks realized by their cooperation are depicted. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof. Therefore, hereinafter, what does not explicitly indicate the name of the configuration may be considered to be made by a control unit centered on the CPU, for example.
[0047]
The image acquisition unit 102 receives an original image from an image source such as a network or a user's digital camera, and uses the original image as it is or processes it to generate a basic image. For example, when a separate image is required, the original image may be used as the basic image as it is. On the other hand, when side-by-side images are required, a plurality of original images are juxtaposed and combined. When a multiplexed image is required, the parallax images from each viewpoint are reconstructed in a stripe shape or a matrix shape.
[0048]
The image acquisition unit 102 further compresses the obtained basic image as necessary. Prior to that, it may be determined whether or not the image quality of the stereoscopic image is affected by the compression, and if it is determined that the compression will occur, the compression may be prohibited. For example, when compressing a multiplex image with respect to a spatial frequency component, the compression may be prohibited or may be compressed after being converted into a side-by-side image.
[0049]
The information adding unit 104 adds the above-described header information to the basic image obtained in this way, and records the resulting image data for stereoscopic display in a storage device (not shown) or a predetermined amount via a network. Deliver to the location. As described above, according to this apparatus 100, a basic image with stereoscopic information can be prepared in advance for those who desire stereoscopic display.
[0050]
Note that the image acquisition unit 102 does not necessarily obtain the original image first. Input what is already a multiplex image from a network, etc., detect its 3D information, determine that it is a multiplex image, recognize that it is unsuitable for compression as it is, It is also possible to convert the image into a side-by-side image, compress the image, and rewrite the stereoscopic information. In that case, the apparatus 100 can also be used as a relay point for the distribution of stereoscopic images.
[0051]
On the other hand, FIG. 10 shows a configuration of a decoding-side image processing apparatus 200 that actually performs stereoscopic display. The apparatus 200 refers to a predetermined scene in a series of processes for displaying a stereoscopic image, which is added to the acquired basic image, and an image acquisition unit 202 that acquires a basic image for displaying the stereoscopic image. And an information detection unit 204 for detecting stereoscopic information to be performed. The device 200 is typically a user terminal, and the image acquisition unit 202 acquires a basic image to which stereoscopic information has already been added. The image acquisition unit 202 may generate or reproduce a basic image by decompressing an image data that has been compressed in advance.
[0052]
Subsequently, the information detection unit 204 parses the header area added to the basic image and detects stereoscopic information. If it is determined from the detected stereoscopic information that the basic image is not a multiplexed image, the apparatus 200 converts the basic image into a multiplexed image and displays the stereoscopic image. As an optional function of this apparatus 200, a display screen (not shown) of this apparatus 200 is displayed according to the above-mentioned consideration on luminance based on the number of horizontal viewpoints and the number of vertical viewpoints among the detected stereoscopic information. The brightness may be increased.
[0053]
The apparatus 200 can store and edit a basic image as well as a stereoscopic image. At the time of storage, if the basic image is a multiplex image, it may be converted into a side-by-side image or the like, and the stereoscopic information may be rewritten and stored. When editing, for example, the user may want to enlarge or reduce the image. At this time, since the processing is complicated if it is a multiplex image, it may be converted once into a side-by-side image, then subjected to predetermined image processing, and finally returned to the multiplex image for display.
[0054]
In order to facilitate the editing and other image processing, this apparatus 200 always holds images in a format other than a multiplex image, particularly a side-by-side image, in a memory or other storage device, and appropriately stores this as necessary. What is necessary is just to read and use.
[0055]
As an additional configuration of the apparatus 200, if a characteristic acquisition unit that acquires characteristics such as the number of viewpoints and the optimum observation distance of the display apparatus as data is provided, the convenience is further increased. For example, when the assumed number of viewpoints of the basic image is different from that of the display device, a display image selection unit that automatically selects a parallax image to be displayed from the basic image based on the above characteristics can be provided. If the assumed number of viewpoints of the basic image is “4” and that of the display device is “2”, two are selected from the four parallax images. These two parallax images do not need to have continuous viewpoints, and in order to enhance the stereoscopic effect, two images with the viewpoints skipped may be selected. If the number of viewpoints of the basic image is “2” and that of the display device is “4”, the same parallax image is displayed twice, so that a stereoscopically viewable area can be secured in front of the screen.
[0056]
Further, if the image processing apparatus 200 includes a position detection unit that detects the position of the head of an observer viewing the display screen, the display image selection unit changes the parallax image to be selected in accordance with the head position. , You can also show the image around the viewer.
[0057]
When the optical filter can be replaced, for example, the display unit of the apparatus 200 may be provided with a reading unit that optically reads a display including information such as the number of viewpoints pattern-printed on the optical filter. When the read data suggests a discrepancy with the number of viewpoints of the parallax image, the parallax image may be optimally selected and displayed as described above, or a display that prompts replacement of the optical filter may be performed.
[0058]
FIG. 11 shows a configuration of a network system 300 for distributing stereoscopic images. Here, the synthesizing apparatus 302 is the image processing apparatus 100 of FIG. 9 and functions as a distribution starting point. On the other hand, the display device 304 is the image processing device 200 of FIG. 10 and acts as an end point of distribution. As shown in the figure, the synthesizer 302 has a storage device 306 that records a large number of basic images and acts as an image server, so that the user can easily obtain a desired stereoscopic image via the Internet or other network 308. .
[0059]
The present invention has been described based on the embodiments. It is understood by those skilled in the art that these embodiments are exemplifications, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. By the way. Here are some examples:
[0060]
The functions of the image processing apparatuses 100 and 200 in FIGS. 9 and 10 can be provided to the user in the form of computer programs. If the user wants to generate a basic image himself, the function of the image processing apparatus 100 in FIG. 9 may be provided as an authoring tool after being prepared.
[0061]
The header area 42 shown in FIG. 8 has a large degree of freedom including the number of bits. For example,
・ Copyright information of the basic image observed by the user as a stereoscopic image
・ Requirements for optical filters such as parallax barrier and lenticular lens suitable for viewing basic images
Etc. can be further incorporated. Examples of “conditions to be satisfied by the optical filter” include a distance between viewpoints of a parallax image, that is, a distance between eyes, a field angle of a camera at the time of photographing, and the like. Such a condition is an indispensable parameter for designing an optical filter when it is desired to reproduce a stereoscopic image with a correct scale in the front-rear direction. Further, even in the selection of an image when the number of viewpoints of the basic image is larger than the number of viewpoints of the display device, it can be referred to in order to automatically select an image that provides a more natural stereoscopic effect.
[0062]
In the embodiment, the stereoscopic image to be finally displayed is a multiplexed image, but the image to be displayed varies depending on the observation method. Therefore, various images may be displayed after being converted into images suitable for the observation method.
For example, when liquid crystal shutter glasses are used, the image to be displayed is a field sequential image. In the case of a head-mounted display that has separate display means corresponding to the left and right eyes, the stereoscopic image to be displayed is a separate image and is sent to each display means by separate image output means. The display device can display a side-by-side image on a single head-mounted display. In this case, the image is separated into the left half and the right half by optical means and may be configured to be observed in the horizontal direction. Furthermore, a joint image may be displayed by an observation method such as a crossing method or a parallel method.
[0063]
In the embodiment, the apparatus that displays a stereoscopic image, that is, the image processing apparatus 200 in FIG. 10 starts processing on the premise that stereoscopic information is added to the basic image. However, if there is an existing basic image that does not depend on the present embodiment, this does not have a header area characteristic of the embodiment, so an inspection / estimation processing unit that inspects the basic image is provided, and the image The three-dimensional information may be estimated on the processing device 200 side. For example, whether or not the image is a side-by-side image may be evaluated by dividing the image into m equal parts and n equal parts in the horizontal direction and the vertical direction, respectively, and changing the degree of approximation of the image in each region while changing the values of m and n. . If the degree of approximation of each region or part thereof is high for a certain set of m and n, this can be estimated as a side-by-side image with m horizontal viewpoints and n vertical viewpoints. The evaluation of the degree of approximation is based on, for example, the sum of squares of pixel values. In addition to this, it is conceivable that a boundary line of a region emerges when a differential filter is applied to the basic image, whereby it may be possible to estimate whether the image is a side-by-side image.
[0064]
【The invention's effect】
According to the present invention, the distribution of stereoscopic images can be promoted.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a state in which a user stereoscopically views four parallax images multiplexed in a horizontal direction.
FIG. 2 is a diagram illustrating a multiplexed image.
FIG. 3 is a diagram showing a side-by-side image.
FIG. 4 is a diagram illustrating a set of a plurality of separate images.
FIG. 5 is a diagram schematically illustrating a configuration of a side-by-side image according to the embodiment.
FIG. 6 is a diagram schematically illustrating a configuration of a multiplexed image according to the embodiment.
FIG. 7 is a diagram schematically illustrating a configuration of a separate image according to the embodiment.
FIG. 8 is a configuration diagram of a header area added to a basic image according to the embodiment.
FIG. 9 is a configuration diagram of an image processing apparatus serving as a starting point of image distribution according to the embodiment.
FIG. 10 is a configuration diagram of an image processing apparatus as an end point of image distribution according to the embodiment.
11 is a diagram illustrating a configuration of a network system for image distribution in which the image processing apparatus in FIG. 9 is a synthesizing apparatus and the image processing apparatus in FIG. 10 is a display apparatus.
[Explanation of symbols]
12 parallax barrier, 20, 40 multiplexed image, 30, 50 side-by-side image, 32, 60, 62 separate image, 42 header area, 100, 200 image processing device, 102, 202 image acquisition unit, 104 information addition unit, 204 Information detection unit, 300 network system, 302 synthesis device, 304 display device, 306 storage device.

Claims (9)

An image acquisition unit for acquiring a basic image for displaying a stereoscopic image;
Look including a information estimating unit that estimates a three-dimensional information for displaying a stereoscopic image based on the pixel values of the basic image by the image acquisition unit has acquired,
The information estimation unit divides the basic image acquired by the image acquisition unit into a plurality of regions, and evaluates the degree of approximation of each of the divided regions to determine whether the image is a side-by-side image. A featured stereoscopic image processing apparatus. The basic image is again a side-by-side image,
The information estimation unit divides the side-by-side image acquired by the image acquisition unit into a plurality of regions, and estimates the number of viewpoints of the side-by-side image by evaluating the degree of approximation of each divided region. The three-dimensional image processing apparatus described. The stereoscopic image processing apparatus according to claim 1, wherein the information estimation unit evaluates the degree of approximation based on a sum of squared differences of pixel values. An image acquisition unit for acquiring a basic image for displaying a stereoscopic image;
An information estimation unit that estimates stereoscopic information for displaying a stereoscopic image based on the pixel value of the basic image acquired by the image acquisition unit,
The information estimation unit determines whether or not the image is a side-by-side image by performing differentiation on the basic image acquired by the image acquisition unit. The stereoscopic image processing apparatus according to claim 1, further comprising an information adding unit that adds stereoscopic information to the basic image. The stereoscopic image processing apparatus according to claim 5, wherein the information adding unit adds stereoscopic information to a header area of a basic image. The stereoscopic image processing apparatus according to claim 1, wherein the basic image does not have a header area that holds stereoscopic information. An image acquisition step of acquiring a basic image for displaying a stereoscopic image;
An information estimation step of estimating stereoscopic information for displaying a stereoscopic image based on the pixel value of the basic image acquired by the image acquisition unit,
In the information estimation step, the basic image acquired in the image acquisition step is divided into a plurality of regions, and the degree of approximation of each divided region is evaluated to determine whether the image is a side-by-side image. A feature of a stereoscopic image processing method. On the computer,
An image acquisition step of acquiring a basic image for displaying a stereoscopic image;
An information estimation step of estimating stereoscopic information for displaying a stereoscopic image based on the pixel value of the basic image acquired by the image acquisition unit;
A computer program for executing
In the information estimation step, the basic image acquired in the image acquisition step is divided into a plurality of regions, and the degree of approximation of each divided region is evaluated to determine whether the image is a side-by-side image. A featured computer program.

Images