JP2003070022A - Three-dimensional image compressing method, apparatus and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To overcome the problem wherein a parallax image is compressed to lose pixel information, resulting in the deteriorated quantity of a three- dimensional image. SOLUTION: Two parallax images 11a, 12a are combined to form a stereoscopic image 1c. Three pixels are selected in the order of 11R, 11B, 12G every two pixels of the first parallax image 11a to form one pixel in a first compressed image 11b. Three pixels are selected in the order of 21G, 22R, 22B for each two pixels of the second parallax image 12a offset by one pixel from the first parallax image 11a to form one pixel in a second compressed image 12b. Pixels are alternately read from the first and second compressed images 11b, 12b to form the three-dimensional image 1c.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic image processing technique, and more particularly to a method, an apparatus, a system and a computer program for reducing and compressing a data amount of a stereoscopic image.

[0002]

2. Description of the Related Art In recent years, the population of users of the Internet has rapidly increased, and the broadband era, which can be said to be a new stage of Internet usage, is about to enter. In broadband communication, the communication band is remarkably widened, so heavy image data, which has been often shunned in the past, is becoming popular. image,
Especially when the distribution of moving images is widespread, the user naturally demands enhancement of contents and improvement of image quality. These are largely due to the digitization of existing video software, the development of authoring tools therefor, and the pursuit of highly efficient and lossless image coding technology.

Under such circumstances, as one form of an image distribution service in the near future, the distribution of pseudo three-dimensional images (hereinafter, also simply referred to as "stereoscopic images") has received technical attention and has gained a considerable market. Conceivable. Stereoscopic images fulfill users' desires for more realistic images, and are particularly attractive for applications such as movies and games that seek a sense of presence. Further, the three-dimensional image is also useful for realistic display of products in product presentation in EC (electronic commerce) which is considered to be one of the standards of commerce in the 21st century.

One of the methods for displaying such a stereoscopic image is
There is a parallax barrier system. In this method, the images of the left and right eyes are alternately arranged in stripes, and a stereoscopic image is combined and displayed on a color liquid crystal panel or the like. Observing this stereoscopic image through the parallax barrier enables stereoscopic viewing. A wide range of applications is considered because stereoscopic viewing is possible by a simple method.

[0005]

When synthesizing a stereoscopic image from left and right parallax images by the parallax barrier method, assuming that the parallax image has the same resolution as the stereoscopic image, the original image for synthesizing the stereoscopic image is a stereoscopic image. It has twice as much data as an image. In particular, when synthesizing a multi-view stereoscopic image from parallax images when viewed from three or more viewpoints, the amount of data increases in proportion to the number of viewpoints, so consider the storage area and the capacity of data transfer. It is necessary to reduce the data amount of the original image in some way.

The present invention has been made in view of such a situation, and an object thereof is to reduce the data amount of a parallax image which is a basis for displaying a stereoscopic image and to efficiently synthesize a stereoscopic image. It is to provide an image compression technique.

[0007]

One aspect of the present invention relates to a stereoscopic image compression method. According to this method, display elements that are actually used in a stereoscopic image are selected from the parallax image, and a drawing element is configured by a combination thereof to reduce the information amount of the parallax image.

For stereoscopic image viewing, two images are required when viewed with the right eye and the left eye. These images are generically referred to as parallax images, but more generally, in the case of a multi-view system, the images are extended to three or more images when viewed from three or more viewpoint positions, and are also referred to as parallax images. In the present specification, “parallax image” refers to each of a plurality of images when viewed from a plurality of viewpoints. However, in general, these plural images may be collectively referred to as a parallax image. Further, in the present specification, the “stereoscopic image” refers to an image synthesized using parallax images so that an observer can stereoscopically view the images.

Here, the display element is a pixel (dot) which is the minimum display element of the image, and the drawing element is a pixel (pixel) which is the minimum unit of drawing. For example, when an image is displayed on a liquid crystal display with 3 pixels of RGB, RGB
Three pixels form one picture element. Hereinafter, reducing the amount of data is referred to as “compression” in a broad sense, but this does not mean compression encoding of image data like JPEG compression, but resolution by processing such as thinning or averaging data. To reduce the amount of data. The image here is not limited to a still image, and may be a moving image.
This stereoscopic image compression method may be applied to one frame of a moving image.

Another aspect of the present invention also relates to a stereoscopic image compression method. This method selects a display element that is actually used in a stereoscopic image from each of a plurality of parallax images, forms a drawing element by combining them, and generates a new parallax image, and a plurality of generated parallax images. Extracting the drawing elements from each of the parallax images to synthesize a stereoscopic image. Since the display elements that are not used in the stereoscopic image are discarded, the data amount of the newly generated parallax image is reduced as compared with the original image.

Yet another aspect of the present invention relates to a stereoscopic image compression apparatus. This apparatus selects a display element that is actually used in a stereoscopic image from the input parallax image, configures a drawing element by combining them, and a compression unit that compresses the parallax image, and the compressed parallax image. The storage unit includes a storage unit that stores the image data, and a combining unit that extracts the drawing element from each of the parallax images having different viewpoints stored in the storage unit and combines the stereoscopic images.

The compression unit selects pixels actually used in the stereoscopic image from the parallax image composed of a row of RGB3 pixels, and a new RGB3 is selected by combining them.
The parallax image may be compressed by forming rows of pixels. The new RGB3 pixels may form a new pixel different from the pixel composed of the RGB3 pixels of the original parallax image.

The compression section may select the display element for every n pieces from each of the n pieces of parallax images viewed from the n viewpoints, and combine them to form the drawing element. The row of the display elements may be a row of RGB 3 pixels, and the compression unit selects R, G, B from the row of the display elements.
Pixel of every n pixels is thinned out every n pixels to create a new RGB
It is also possible to select three pixels and construct a new picture element from these new RGB3 pixels.

Yet another aspect of the present invention relates to a computer program. This program selects a display element that is actually used in a stereoscopic image from each of a plurality of parallax images, forms a drawing element by combining them, and generates a new parallax image, and a plurality of generated parallax images. Causing the computer to execute the steps of extracting the drawing element from each of the parallax images and synthesizing a stereoscopic image.

Yet another aspect of the present invention relates to a stereoscopic image compression system. This system includes a terminal and a server.
The server selects a display element that is actually used in the stereoscopic image from the input parallax image, configures a drawing element by combining them to compress the parallax image, and a plurality of compressed viewpoints different from each other. And a transmitting unit that transmits the parallax image to the terminal. The terminal includes a receiving unit that receives the plurality of parallax images transmitted by the server, and a combining unit that extracts the drawing element from each of the plurality of received parallax images and combines the stereoscopic images.

It should be noted that any combination of the above components, the expression of the present invention may be represented by a method, an apparatus, a system, a recording medium,
A program converted between computer programs and the like is also effective as an aspect of the present invention.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION First, a general configuration for reducing the data amount of parallax images will be described, and then an improved technique thereof will be described.

It is assumed that the left and right parallax images are used to synthesize a stereoscopic image for the parallax barrier system. It is assumed that the two parallax images have the same resolution as the stereoscopic image. Compared with the resolution when actually displayed as a stereoscopic image,
The parallax image that is the original image has twice the amount of information, and different parallax image information is used for each pixel when these original images are combined into a stereoscopic image.

However, when the purpose is only for stereoscopic display, it is wasteful that the original image has twice the amount of information, and therefore the image is compressed and held in the horizontal direction. A color liquid crystal panel holds three pieces of display elements (hereinafter referred to as “pixels”) of different colors, which are normally arranged in the horizontal direction, as one drawing element (hereinafter referred to as “picture element”), and collectively holds information. Each picture element is displayed based on the information. Therefore 2
When the eye-type parallax image is compressed in the horizontal direction, the compression is performed in units of picture element columns. For example, as shown in FIG. 6A, when one pixel is composed of three RGB pixels in the parallax image 51a, thinning processing is performed in units of the one pixel, and pixel data is obtained every other pixel. By discarding, the compressed image 51b is formed by the remaining picture element data columns. Alternatively, as another method, as shown in FIG.
A compressed image 52b in which the picture element data to be left is determined from the average value of adjacent picture elements in a and the picture element sequence is reduced to half
To generate. In any of the methods, in stereoscopic display, the stereoscopic image is synthesized by alternately extracting and arranging pixel data for each pixel from the two parallax images compressed in this manner.

The parallax barrier type stereoscopic image can be configured by a multi-view type. The multi-view stereoscopic image referred to here is a stereoscopic image that is synthesized by using parallax images when viewed from three or more different viewpoints as an original image. It is obtained by shooting things. When an observer observes the displayed stereoscopic image from different positions, different stereoscopic images can be seen according to the number of viewpoints at the time of shooting. Such a multi-view stereoscopic image is effectively used to display a wraparound stereoscopic image. For example, by using a parallax image when viewed from the left or right relative to the object, it is possible to obtain the sensation of turning the object left and right. Further, if the parallax image when viewed from the relatively upper direction and the parallax image when viewed from the relatively lower direction are used, the number of viewpoints is further doubled, and the observer moves the observation position up and down. By changing the direction, it is possible to display a stereoscopic image that also wraps around in the vertical direction.

Also in the case of a multi-view stereoscopic image, the data amount can be reduced as in the above case. FIG. 7 shows an example of a four-eye system. Four parallax images 61a, 62a, which form the stereoscopic image 6c,
63a and 64a are 1 by the thinning processing described above.
Compressed to / 4. Further compressed four parallax images 6
1b, 62b, 63b, 64b are arranged side by side in the horizontal direction and are held as one image 6b. This is called “side-by-side compression” of parallax images.

The parallax barrier stereoscopic image is composed by reading out pixel data from each of the parallax images pixel by pixel and alternately arranging the pixel data. Therefore, when a stereoscopic image is combined using parallax images that have been thinned out in picture element units, different pixel data is used than when a stereoscopic image is combined from uncompressed parallax images, and inconsistency occurs. That is, originally, it was necessary to select pixel data on a pixel-by-pixel basis and combine it into a stereoscopic image. Will not be obtained. In the case of the n-eye type, (n-1) picture elements are thinned out, and when one picture element is composed of three RGB pixels, 3 * (n-1) picture element data is lost. Therefore, the effect of thinning out in picture element units increases as the number of viewpoints n increases. The present inventor has recognized that such a problem occurs, and has devised a further improved technique for compressing a parallax image as described below.

FIG. 1 shows the configuration of a stereoscopic image compression apparatus 100 according to the first embodiment. This configuration can be realized in terms of hardware by a CPU, memory, or other LSI of an arbitrary computer, and in terms of software, it can be achieved by a program having a stereoscopic image compression function loaded in the memory. Then, the functional blocks realized by those collaborations are drawn. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof.

The parallax image input unit 102 receives input of parallax image data when viewed from different viewpoints. For example, a parallax image captured by the camera while changing the capturing position is input. The parallax image compression unit 104 compresses the input parallax image by the method described below. The compressed parallax image is held in the storage unit 106. Stereoscopic image composition unit 10
Reference numeral 8 reads pixel information from the plurality of parallax images held in the storage unit 106 and synthesizes a stereoscopic image by a method described later. The display unit 110 outputs the combined stereoscopic image to a display device such as a display. The displayed stereoscopic image is observed through a parallax barrier or a lenticular lens.

FIG. 2 is a diagram for explaining a method of compressing parallax images by the stereoscopic image compression apparatus 100. Here, a twin-lens type example will be described. Two original images 11a and 1 having parallax
It is assumed that 2a is input. When the stereoscopic image 1c is composed from these uncompressed original images, 11R, 21
Pixel columns are selected and combined in the order of G, 11B, 22R, 12G, 22B. Focusing on the first original image 11a, pixels are selected in the order of 11R, 11B, and 12G. One pixel is shifted from the second original image 12a,
Pixels are selected in the order of 21G, 22R, 22B, and so on. In this way, the stereoscopic image 1c is selected from the parallax images in units of dots, and the pixels are alternately arranged to be combined.

The parallax image compression unit 104 pays attention to a pixel row actually used in stereoscopic display, and performs compression by constructing picture elements from these used pixels. For example, the picture elements in the first column of the compressed image 11b for the first original image 11a are the three pixels 11 that are actually used in the original image 11a.
It is composed of R, 12G, and 11B. These are pixel information of different points which originally belong to different picture elements, but the fact is ignored to create a composite picture element. Then, the information of 11G, 12R, and 12B that is not used for the stereoscopic image display is removed without being used at all. The compressed image 11b thus obtained is held in the storage unit 106. The same applies to the compressed image 12b with respect to the second original image 12a,
3 pixels 21G, 22 actually used in the original image 12a
R and 22B form a pixel in the first column of the compressed image 12b.

The stereoscopic image synthesizing unit 108, as shown in FIG.
The two compressed images 11b and 12 held in the storage unit 106
Pixels are alternately read from b and the stereoscopic image 1c is synthesized. First compressed image 11b to 11R, second compressed image 12
b to 21G, first compressed images 11b to 11B, second compressed images 12b to 22R, first compressed images 11b to 12
Pixel information is read in the order of G and the second compressed image 12b to 22B. The stereoscopic image 1c thus synthesized has the same pixel information as the stereoscopic image 1c obtained by synthesizing the uncompressed parallax images shown in FIG.

FIG. 4 is a diagram for explaining a compression method in the case of the 8-eye type. Of the eight original images having parallax, the pixel row of one original image 31a is shown. Of the pixel rows of the original image 31a, 8 are actually used for the stereoscopic display image 3c.
Since the pixels are thinned out 11R, 13B, 16G for each pixel, the picture element in the first column of the compressed image 31b is composed of the information of these three pixels as shown in the figure. Similarly, for the other seven original images, compressed images are formed by thinning out every eight pixels.

FIG. 5 is a flow chart for explaining the procedure of compression of parallax images and composition of stereoscopic images by the stereoscopic image compression apparatus 100. Here, n is the number of parallax images, m is the number of horizontal picture elements, c is the number of pixels forming one picture element, and dat
a [f, g, h] is the h-th pixel data in the g-th pixel of the f-th original image, and ndata [u, v, w] is the w-th pixel in the v-th pixel of the u-th compressed image. Pixel data, cdata [p, q] is the q-th pixel data in the p-th picture element of the combined stereoscopic image. X / Y is a quotient (integer) when X is divided by Y, and X%
Y is a remainder (integer) when X is divided by Y.

FIG. 5A shows a parallax image compression procedure.
In the two-lens system, n = 2, and in the eight-lens system, n = 8. Further, in the liquid crystal panel, one pixel is composed of three RGB pixels, so that c = 3. First, the variable p indicating the original image number is initialized to zero (S10). P% in the variable db
The value of c is substituted (S14). A variable i indicating the pixel number in the horizontal direction is initialized to zero (S16).

For the current value of the variable i, (i
The value of + p) / c is substituted (S18), and the variable b is (i +
The value of db)% c is substituted (S20). Further, the value of i / (c × n) is substituted into the variable e (S22). The value of the pixel data data [p, a, b] selected from the original image is substituted into the pixel data ndata [p, e, b] of the compressed image (S24). The variable i is incremented by the number n of parallax images (S26). If the variable i is smaller than c × m (Y of S28), steps S18 to S26 are repeated for the incremented variable i. In the case of an image of 3 pixels of RGB, by repeating steps S18 to S26, one pixel of RGB is selected from the original image for each number n of parallax images and rearranged in the order of RGB to form one compressed image. The element is composed.

When the variable i becomes c × m (N of S28),
In step S30, the variable p is incremented by 1. If the variable p is smaller than n (Y in S32), the process returns to step S14 and the subsequent processes are repeated for the next original image. When the variable p becomes n (N in S32), the compression process ends.

FIG. 5B shows a procedure for synthesizing a stereoscopic image from a compressed image. First, the variable i indicating the pixel number in the horizontal direction is initialized to zero (S46).

With respect to the current value of the variable i, i /
The value of c is substituted (S48), and the value of i% c is substituted for the variable b (S50). Further, the value of i / (c × n) is substituted into the variable e (S52). Variable k indicating the number of compressed image
The value of i% n is substituted into (S53). The value of the pixel data ndata [k, e, b] of the k-th compressed image is substituted into the pixel data cdata [a, b] of the stereoscopic image (S.
54). The variable i is incremented by 1 of the parallax image (S56). If the variable i is smaller than c × m (S58)
Y), and steps S48 to S56 are repeated for the incremented variable i.

When the variable i becomes c × m (N of S58),
The stereoscopic image synthesizing process ends. This makes RGB 3
In the case of a pixel image, one of RGB pixels is alternately selected from the n compressed images, and the RGB image in the stereoscopic image is selected.
Pixel data is formed by rearranging in order of RGB so that one picture element is formed in the order of.

In the case of an image having a parallax in the vertical direction, in general, the picture element is rarely divided into pixels in the vertical direction. You only need to leave the picture elements in the necessary rows.

FIG. 8 is a block diagram of a stereoscopic image compression system according to the second embodiment. In this system, a user terminal 120 and a server 140 are connected to each other via a network such as the Internet (not shown).
0 transmits the parallax image thinned and compressed to the user terminal 120 when there is a request from the user terminal 120.
The user terminal 120 synthesizes a stereoscopic image from the parallax image transmitted from the server 140 and displays it on a display or the like.

The server 140 includes a parallax image input unit 102,
Parallax image compression unit 104, storage unit 106, and transmission unit 1
42 is included. The same operations as those in the first embodiment are performed for the components designated by the same reference numerals as those in the first embodiment. The transmission unit 142 reads out the parallax image compressed by the parallax image compression unit 104 from the storage unit 106, and outputs the parallax image to the user terminal 120.
Send to.

The user terminal 120 includes a receiving unit 122, a stereoscopic image synthesizing unit 108, and a display unit 110. Receiver 1
22 receives the parallax image after compression from the server 140.
Similar to the first embodiment, the stereoscopic image synthesis unit 108 synthesizes the stereoscopic image from the received compressed image, and the display unit 110 displays the synthesized stereoscopic image.

By thus configuring the server / client system and distributing parallax image compression and stereoscopic image composition to the server and client, it is possible to distribute the load and reduce network traffic. For example, in the case of an eight-eye type stereoscopic image, each of the eight parallax images is compressed to ⅛ in the server 140 and transmitted to the user terminal 120, so the data transfer amount is 1
It can be suppressed to the same amount as when sending one original image.
Further, the process of synthesizing a stereoscopic image from a compressed image requires rearrangement of pixels, which increases the load on the CPU, but since this process is performed on the user terminal 120 side, the server 14
It is possible to prevent the load from being concentrated on 0.

As described above, according to the embodiment of the present invention, when the parallax images are thinned out, the pixels actually used in the stereoscopic image are left, so that the parallax images are synthesized from the compressed parallax images. The three-dimensional image has the same pixel information as the three-dimensional image synthesized from the original image before compression, and an appropriate three-dimensional display is possible. In addition, since pixel information necessary for displaying a stereoscopic image is left and unnecessary pixel information is removed and compressed, no data loss occurs. By this compression, it is possible to reduce the storage capacity required for storing image data, and it is also possible to reduce network traffic when operating in a server / client system.

The present invention has been described above based on the embodiments. It is understood by those skilled in the art that these embodiments are mere examples, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and that such modifications are also within the scope of the present invention. By the way.

As such a modification, in the second embodiment, the user terminal 120 may not have the function of compressing the parallax image, but may simply display the received stereoscopic image. In that case, the server 140 has a role of providing a stereoscopic image prepared in advance to the user terminal 120. The parallax image compressing unit 104 is added to the server 140.
May be provided, and the parallax image may be compressed on the server 140 side and stored in the storage unit 106. In that case, the user terminal 120 may transmit the parallax image before compression to the server 140.

Further, in any of the embodiments, the data format of the compressed image may be a side-by-side format in which the data of a plurality of parallax images after compression are arranged side by side to form one image data. Furthermore, an encoding unit that compresses and encodes a side-by-side format compressed image by JPEG compression or the like may be provided, and the parallax image that has been thinned and compressed may be stored in the storage unit 106 in a compression-encoded form. The server 140 in the second embodiment
The transmitting unit 142 may transmit the compression-coded parallax image to the user terminal 120. In this case, the user terminal 120 performs a process of once decoding the received parallax image and then combining the parallax image with the stereoscopic image. The server 140 may have a receiving unit that receives the side-by-side format compressed image from the user terminal 120, and may be configured to receive the parallax image from the user. Alternatively, the user terminal 120
The parallax image compressing unit 104 may perform a compression process after the receiving unit receives a plurality of parallax images from the user and receives the plurality of parallax images from the user.

Further, in the above description, the observer stereoscopically viewed the stereoscopic image synthesized using the parallax barrier or the lenticular lens, but as means for observing the stereoscopic image,
Any optical filter may be used as long as it is an optical filter that acts as a video separation means for separating the image for the right eye and the image for the left eye, and it can be applied regardless of the specific method of stereoscopic vision.

[0046]

According to the present invention, it is possible to hold image information of a stereoscopic image and compress the image data.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a stereoscopic image compression apparatus according to a first embodiment.

FIG. 2 is an explanatory diagram of a parallax image compression method by the stereoscopic image compression device.

FIG. 3 is an explanatory diagram of a stereoscopic image synthesizing method by the stereoscopic image compression device.

FIG. 4 is an explanatory diagram of a compression method in the case of an 8-eye type.

5A is a flowchart showing a procedure for compressing a parallax image by the stereoscopic image compression apparatus, and FIG. 5B is a flowchart showing a procedure for composing a stereoscopic image by the stereoscopic image compression apparatus.

6A and 6B are explanatory views of a conventional parallax image compression method.

[Fig. 7] Fig. 7 is a diagram for describing a conventional compression example of a parallax image in the case of a four-eye system.

FIG. 8 is a configuration diagram of a stereoscopic image providing system according to a second embodiment.

[Explanation of symbols]

100 stereoscopic image compression device, 102 parallax image input unit, 104 parallax image compression unit, 106 storage unit,
108 stereoscopic image composition unit, 110 display unit, 120
User terminal, 122 receiver, 140 server,
142 Transmitter.

Claims (7)

[Claims] 1. A stereoscopic image compression method, comprising: selecting display elements that are actually used in a stereoscopic image from a parallax image and configuring a drawing element by a combination thereof to reduce the information amount of the parallax image. . 2. A step of selecting a display element that is actually used in a stereoscopic image from each of the plurality of parallax images and forming a drawing element by a combination thereof to generate a new parallax image, and the plurality of generated plurality of parallax images. And extracting the drawing element from each of the parallax images to synthesize a stereoscopic image. 3. A compression unit that selects display elements that are actually used in a stereoscopic image from the input parallax image, forms drawing elements by combining them and compresses the parallax image, and the compressed parallax image. A stereoscopic image compression apparatus including: a storage unit that stores the stereoscopic image; and a combining unit that extracts the drawing element from each of a plurality of parallax images with different viewpoints stored in the storage unit and combines the stereoscopic images. 4. The compression unit selects pixels that are actually used in a stereoscopic image from a parallax image composed of a row of RGB 3 pixels, and a new RGB is selected by a combination thereof.
The stereoscopic image compression apparatus according to claim 3, wherein the parallax image is compressed by forming a column of 3 pixels. 5. The compression unit selects the display elements for each n pieces from each of the n pieces of parallax images viewed from the n viewpoints, and forms the drawing element by combining them. The stereoscopic image compression apparatus according to claim 3, which is characterized in that. 6. A step of selecting a display element that is actually used in a stereoscopic image from each of the plurality of parallax images and forming a drawing element by a combination thereof to generate a new parallax image, and the plurality of generated parallax images. And a step of taking out the drawing element from each of the parallax images and synthesizing a stereoscopic image, the computer program being executed. 7. A stereoscopic image compression system including a terminal and a server, wherein the server selects a display element that is actually used in the stereoscopic image from the input parallax image and configures a drawing element by a combination thereof. And a compression unit that compresses the parallax image, and a transmission unit that transmits the plurality of compressed parallax images with different viewpoints to the terminal, the terminal receiving the plurality of parallax images transmitted by the server. A stereoscopic image compression system, comprising: a reception unit that performs the stereoscopic image and a synthesis unit that extracts the drawing element from each of the plurality of received parallax images and synthesizes the stereoscopic image.