KR20130132922A - Multi-sample resolving of re-projection of two-dimensional image - Google Patents

Abstract

In accordance with the present invention, multiple sample analysis of reprojection of a two-dimensional image is disclosed. One or more samples of the two-dimensional image are identified for each pixel of the three-dimensional reprojection. The amount of one or more sample coverages is determined for each pixel of the reprojection. The amount of each coverage identifies the area of the pixel covered by the corresponding two-dimensional sample. The final value is analyzed for each pixel of the reprojection by combining each two-dimensional sample associated with the pixel according to the weighted sample coverage amount.

Description

Multi-Sample Analysis of Reprojection of Two-Dimensional Images {MULTI-SAMPLE RESOLVING OF RE-PROJECTION OF TWO-DIMENSIONAL IMAGE}

Cross-reference to related application

This application is filed and filed on January 7, 2011, entitled "DYNAMIC ADJUSTMENT OF PREDETERMINED THREE-DIMENSIONAL RE-PROJECTION SETTINGS BASED ON SCENE CONTENT," ).

This application is filed and filed on January 7, 2011, entitled "SCALING PIXEL DEPTH VALUES OF USER-CONTROLLED VIRTUAL OBJECT IN THREE-DIMENSIONAL SCENE," SCEA10053US00).

This application is filed on January 7, 2011, entitled "MORPHOLOGICAL ANTI-ALIASING (MLAA) OF A RE-PROJECTION OF A TWO-DIMENSIONAL IMAGE," Control number SCEA10054US00).

Technical field

Embodiments of the present invention relate to a method for analyzing multiple samples of three-dimensional reprojection of two-dimensional images.

The ability to perceive two-dimensional images in three dimensions by many different techniques has been fairly widely known over the years. By providing aspects of depth to the two-dimensional image, it potentially creates a sense of presence for any illustrated scene. Introducing this three-dimensional visual representation greatly enhances the observer experience, especially in the area of video games.

There are a number of techniques for three-dimensional rendering of a given image. Most recently, a technique has been proposed for projecting two-dimensional image (s) in three-dimensional space, known as depth-image-based rendering (DIBR). In contrast to this proposal, which often relied on the basic concept of "stereo" video, namely, two separate video streams, one for the left eye and the other for the right eye, often this idea Is based on a more flexible monoscopic video (ie, a single video stream) and associated transmission of associated per-pixel depth information. From this data representation, one or more "virtual" views of the 3D scene can be generated in real time at the receiver side by so-called DIBR techniques. This new approach to 3D image rendering offers several advantages over previous approaches.

There are generally two ways of presenting two separate images to the observer to produce an illusion of depth. In a system commonly used to project 3D images onto a screen, there are two separate synchronous projectors for the left eye image and the right eye image. Images for both eyes are simultaneously projected onto the screen, but there is orthogonal polarization, for example vertical polarization for the left eye image and horizontal polarization for the right eye image. The observer wears special polarized 3D viewing glasses with lenses that are properly polarized for the left and right eyes (eg, vertically polarized for the left eye and horizontally polarized for the right eye). Because of the polarization of the lens and image, the observer perceives only the left eye image with the left eye and only the right eye image with the right eye. The degree of depth illusion is partly a function of the offset between the two pictures on the screen.

In a 3D video system, the left eye picture and the right eye picture are displayed by the video display screen but are not displayed correctly at the same time. Instead, the left eye image and the right eye image are displayed in an alternating manner. The observer wears active shutter glasses that block the left eye when the right eye image is displayed and vice versa.

The experience of 3D video may depend somewhat on the characteristics of human vision. For example, the human eye has a distinct number of light receptors, but a human cannot distinguish any pixels even in the surrounding field of view. More surprisingly, the number of color-sensitive cones in the human retina can vary dramatically between individuals and up to 40. Nevertheless, people seem to perceive colors in the same way, and we essentially use the brain. In addition, the human watch has the ability (super vision) to determine the alignment of objects at a fraction of the cone's width. This explains why spatial aliasing artifacts (ie, visual irregularities) are more pronounced than color errors.

Because of this fact, graphics hardware vendors make great efforts to compensate for aliasing artifacts by adjusting color accuracy to spatial continuity. As with the integration nature of digital cameras, a number of techniques are supported in hardware based on mixing weighted color samples.

Of course, any aliasing artifacts will eventually disappear as the display resolution and sampling rate increase. Aliasing artifacts can also be handled at low resolutions by computing and averaging multiple samples per pixel. However, for still most image rendering algorithms, these solutions may not be very feasible due to hardware constraints.

Embodiments of the invention have been made within this context.

1A-1D are schematic diagrams illustrating the effect that distances between corresponding three-dimensional pixels may have on an observer's perceived depth;
2A-2C are schematic diagrams illustrating the effect of depth tearing on three-dimensional reprojection of a two-dimensional image;
3A-3C are schematic diagrams illustrating a multi-sample analysis method of three-dimensional reprojection of a two-dimensional image according to the prior art;
4 is a flowchart illustrating a multi-sample analysis method of three-dimensional reprojection of a two-dimensional image according to an embodiment of the present invention;
5A-5B are schematic diagrams illustrating a multi-sample analysis method of three-dimensional reprojection of a two-dimensional image according to an embodiment of the present invention;
5C is a schematic diagram illustrating superposition between pixel samples in multi-sample analysis of three-dimensional reprojection of a two-dimensional image in accordance with one embodiment of the present invention;
6 is a block diagram showing an apparatus for analyzing multiple samples of three-dimensional reprojection of a two-dimensional image according to an embodiment of the present invention;
7 is a block diagram showing an example of a cell processor implementation of a multi-sample analysis apparatus of three-dimensional reprojection of a two-dimensional image according to an embodiment of the present invention;
8 illustrates an example of a non-transitory computer readable storage medium having instructions for implementing multiple sample analysis of three-dimensional reprojection of a two-dimensional image in accordance with an embodiment of the present invention.

Introduction

Three-dimensional reprojection of a two-dimensional image can be observed by displaying two separate views of the image (ie, one for each eye) so that the viewer recognizes the illusion of depth. When viewing a three-dimensional image, the left and right eyes converge two corresponding pixels (ie, left eye pixel and right eye pixel) to simulate a single pixel with depth of field.

1A-1D are schematic diagrams illustrating the effect that distances between corresponding pixels may have on an observer's perceived depth. Pixel size is a fixed boundary imparted by display resolution that is inevitably limited by hardware in visual displays (eg, three-dimensional television). As the separation distance of the corresponding pixel increases, the perceived depth position gradually moves further away from the screen.

1A shows two corresponding pixels overlapping. These two pixels may form a perceived depth 101 lying on the side of the screen (eg, a visual display). As the corresponding pixels grow apart (ie, located adjacent to each other) in FIG. 1B, the perceived depth 101 is enhanced and lies some distance to the screen. When corresponding pixels grow farther apart (ie separated by pixel width) in FIG. 1C, the perceived depth 101 appears to lie far further into the screen. By directing the left eye pixel to the right of the right eye pixel in FIG. 1D, the perceived depth can be seen closer than the screen. It is important to note that the perceived depth is not linearly proportional to the distance between the corresponding pixels. Thus, a small increase in the separation distance of the corresponding pixel can result in a significant increase in the perceived depth associated with the corresponding pixel.

The interaction between the separation distance of the corresponding pixel and the perceived pixel depth makes the smooth transition of the depth value rather difficult. This is because each pair of corresponding pixels is projected to a particular depth plane determined by the separation distance between the pixels. 2A-2C are schematic diagrams illustrating this effect, referred to herein as depth tearing.

Three-dimensional visual artifacts, known as depth tiering, occur when attempting to locate three-dimensional element 201 having a sloped depth as shown in FIG. 2A. Rather than recognizing element 201 as being located on a single diagonal surface, it is likely that the element is perceived to be located on multiple sides 203 parallel to the screen as shown in FIG. 2B.

In an ideal situation, simply increasing the resolution of the displayed image may solve the problem of depth tearing. However, this solution is not practical because the process of three-dimensional reprojection is limited by hardware constraints. Instead, the effect of depth tearing is minimized by mixing / blending the colors or other pixel values of adjacent three-dimensional pixels. The result is a smoothed waveform 205 rather than a straight diagonal as shown in FIG. 2C. Although incomplete, this provides the observer with a much more natural three-dimensional image.

To blend three-dimensional pixels to form a smoother depth plane, an anti-aliasing solution is used. Prior art anti-aliasing solutions include temporal anti-aliasing which requires the storage of additional buffered data. For example, in the case of a 3D picture, this means storing a pre-reprojected color and depth buffer or two color buffers of left and right eye images. In a 720P picture this requires about 7 megabytes of memory space for data. Considering that the 3D game requires an already increased storage buffer for the second eye picture, this further exacerbates the already high memory pressure.

Other prior art solutions known as full screen anti-aliasing (FSAA) also require larger buffers to increase memory pressure. FSAA inherently involves increasing the resolution and then applying smart downsamples. In addition to still having depth tearing, the big problem is reduced performance at higher resolutions.

An additional solution is known as multisampling anti-aliasing (MSAA), similar to FSAA, but the selection of color values is done at a smaller resolution than the sample resolution. This results in a much cheaper implementation in terms of processing load and memory. In general, MSAA requires an analysis to be performed before additional work is done on the picture. In general, this means that any additional sample data lost due to time reprojection due to post-processing of the source picture is applied. Even if sample information is stored (due to the fixed sample location described above), this information has depth tiering. Memory requirements for FSAA and MSAA are more transient than temporal anti-aliasing.

3A-3D illustrate prior art multi-sampling anti-aliasing methods for multi-sampling with the problems associated with the prior art. Each three-dimensional pixel of the three-dimensional image (ie, left eye view or right eye view) may be represented by a bucket. In general, a pixel bucket contains only a single sample. For example, without limiting the invention, this sample may be a pixel, subpixel, or any group of pixels from the corresponding two-dimensional image to be reprojected in three dimensions. The sample can be characterized by a set of values describing the color profile. Thus, in a reprojection structure that does not implement multiple sampling, the color profile of the three-dimensional pixel may be characterized as comprising a single sample.

In multiple sampling, each pixel bucket is allowed to contain more than one sample. In a typical multisampling structure, sample positions are assigned within three-dimensional pixels and samples are recorded at those positions if covered by the current rendering shape. 3A shows a row of three-dimensional pixels 301 implementing 2X multi-sampling. Black dots represent sample positions to which each three-dimensional pixel 301 to be used during multiple sampling is assigned. In 2X multiple samples, each three-dimensional pixel 301 may be characterized by up to two samples from the corresponding two-dimensional image. Although the example shown is limited to using two samples per three-dimensional pixel, it is important to note that any number of samples can be used in a multiple sampling structure. Furthermore, the sample may be derived from any direction depending on the reprojection direction of movement. If reprojection movement occurs only in the horizontal direction (ie using a parallel movement camera), the sample can be derived from the horizontal scan of the two-dimensional image. This technique requires only one scanline to be stored at a time, requiring relatively little memory overhead.

Multisampling assigns a sample from a two-dimensional image to the sample position for each pixel of the three-dimensional reprojection. However, the actual process of combining multiple samples to obtain a single color profile is known as resolving. Two steps are necessary to achieve a smooth waveform for the diagonal described above. The general analysis process assumes that the coverage of each sample is proportional to the total number of samples per 3D pixel. Thus, each sample in a given 3D pixel in 2X multiple sampling / analysis is given a 50% weight, and these values are added together. So if the left sample is 100% red and the right sample is 100% blue, the 3D pixel takes a color profile of 50% red and 50% blue (or some sort of purple color). Similarly, when processed with 4X multiple sampling / analysis, each sample carries 25% weight. Other blend combinations may be used but may deviate from the fixed position of the sample point from the pixel being analyzed.

The problem with the multiple sampling method of the prior art is that there is an actual position of the sample position within a given three-dimensional pixel. 3b to 3d illustrate this problem in the context of 2 × multiple sampling / analysis. In this example, each sample from the two-dimensional image can be a separate pixel from the two-dimensional image. Each pixel of the three-dimensional reprojection 301 may be represented by up to two different samples. The sample position in each three-dimensional pixel is fixed.

In FIG. 3B, two black pixel samples 303A and 303B are followed by two white pixel samples 305A and 305B of the two-dimensional image that are reprojected to the right. The first black pixel 303A covers the left sample position of the first (leftmost) reprojection pixel 301A in row 301 . The second black pixel 303B covers the right sample position of the first reprojection pixel and the left sample position of the second reprojection pixel 301B . The first white pixel 305A covers the right sample position of the second reprojection pixel 301B and the left sample position of the third reprojection pixel 301C . The second white pixel 305B covers the right sample position of the third reprojection pixel 301C and the left sample position of the fourth reprojection pixel 301D . For this example, the right sample position of the fourth reprojection pixel 301D may be assigned an initial value (eg, colorless or white) since no corresponding pixel value is assigned from the two-dimensional image. . Given each sample position, the 50% weight causes the first reprojected pixel 307A to produce black because the sample position for this pixel is covered with a black pixel value. The second reprojection pixel 307B is analyzed to be gray (ie 50% white and 50% black) because one sample location for this pixel is covered with black pixels and the other with white pixels. Because it becomes. In the analyzed pixel row 307 the third reprojection pixel 307C is analyzed as white because the two sample positions for this pixel are covered with white pixel values. In the pixel row 307 being analyzed, the fourth reprojection pixel 307D is analyzed as white because one sample location is covered by the white pixel 307D , and in this example, the other sample location is a white value. Because it is initialized. It is noted that in a typical MSAA implementation all sample points need to be covered with some initial value, for example black, white or grey. Specifically, white pixel samples 305A and 305B may be ignored when the buffer is initially cleared to white.

In Fig. 3C, the pixels 303A, 303B, 305A, and 305B have been slightly moved to the right. However, this movement is small enough that the coverage of the sample position in row 301 does not change. Specifically, the sample position in the first reprojection pixel 301A is covered by the black pixels 303A and 303B, and the sample position in the second reprojection pixel is covered by the black pixel 303B and the white pixel 305A , In the third reprojection pixel 301C , the sample position is covered with the white pixels 305A, 305B , and in the fourth pixel 301D , one sample position is covered with the white pixel 305B , and the other is not covered, but white Is initialized to As a result, the pixels 307A, 307B, 307C, and 307D in the analyzed pixel row 307 are again analyzed as black, gray, white, and white as shown in FIG. 3B. Since the reprojection pixels 301A, 301B, 301C, and 301D have fixed sample positions, small movements in the reprojection of the two-dimensional image are caused when the two-dimensional pixels 303A, 303B, 305A, and 305B move to the right. It does not change the value profile of the projection pixel.

However, if the two-dimensional pixels 303A, 303B, 305A, and 305B move slightly further to the right than in FIG. 3C, the coverage of the sample position in the reprojected pixels 301A, 301B, 301C, and 301D changes drastically. Specifically, as a result of slightly shifting to the right as shown in FIG. 3D, the two sample positions in the first and second reprojection pixels 301A and 301B are now covered by the black pixels 303A and 303B , respectively. The two sample positions in the third and fourth reprojection pixels 301C, 301D are covered by white pixels 305A, 305B , respectively. As a result, the first two analyzed reprojected pixels 307A, 307B are analyzed in black, and the second two analyzed reprojected pixels 307C, 307D are analyzed in white.

3b to 3d show that there is a difference in values only in the analyzed pixels when there is a discrete shift in the reprojection of a two-dimensional image in a typical 2X multisample / analysis structure. Rather than gradually converting the values of the analyzed three-dimensional pixels to each gradual movement in reprojection, the general multiple sampling / analysis method allows only discrete movements in three-dimensional pixel values such as color values. Since the human eye can match up the colors fairly accurately, this results in an increase in the perceived depth plane (ie, deeper tearing).

Increasing the sample count allows for a more progressive transition of the three-dimensional pixel value profile while increasing the number of depth planes perceived by the observer. Visual artifacts are not so bad in a single scanline (observers only notice that the entire section is slightly shifted). However, the problem is amplified in full images where multiple scan lines are used. In multiple scan lines, the observer can recognize that one line is shifted in depth while the other line remains at the same depth.

Increasing the number of sample positions per three-dimensional pixel significantly reduces the number of visual artifacts recognized by the viewer. However, increasing the sample count significantly is not feasible due to the memory constraints of the system. For example, a typical 720p buffer using RGBA reaches about 3.5M. Adding a second sample location to the three dimensional pixel doubles the amount of memory required.

Embodiments of the present invention seek to limit the number of two-dimensional samples required for a given three-dimensional pixel while forming a more gradual transition in the three-dimensional pixel color profile for movement in reprojection. Embodiments of the invention may use more than two samples. However, in embodiments with reprojection moving only in the horizontal direction, visual improvement was observed to be negligible when four samples were used instead of two. Without being limited to any theory of operation, this may be due to all pixels being the same size upon reprojection in the embodiment being tested. When a typical scene allows up to four samples, there are few patterns of samples that do not overlap in such a way that it is necessary to hold more than two samples. Embodiments of reprojecting pixels horizontally and vertically can yield improved visual quality using more than two samples. In an embodiment of the present invention, the pixel value for each pixel in three-dimensional reprojection is calculated based on the amount of "coverage" of the corresponding sample. In fact, the amount of coverage can be used to determine the weight to apply to each two-dimensional sample value when calculating the corresponding three-dimensional pixel value in reprojection.

Example

4 is a flowchart illustrating a multi-sample analysis method of three-dimensional reprojection of a two-dimensional image according to an embodiment of the present invention. Before the two-dimensional image 401 is reprojected in three dimensions, the sample must be identified for use with the corresponding three dimensional pixel (step 403 ). Three-dimensional reprojection is a combination of multiple distinct views (ie, right eye view and left eye view) that converge to produce a real depth illusion. For each view of the three-dimensional reprojection, it must first be determined whether the two-dimensional sample corresponds to each three-dimensional pixel for this particular view. As mentioned above, a sample refers to a pixel, sub-pixel, or any other group of pixels in a two-dimensional image. The three-dimensional pixel may be characterized in that any number of samples depends on the type of reprojection performed on the two-dimensional image. In addition, any two-dimensional sample may overlap with more than one three-dimensional pixel.

Since there may be more than the maximum allowable number of samples per three-dimensional pixel, there may be a choice of samples to use. When reprojecting the sample may be selected to keep the foreground object. This can be done in a depth test for each sample or in the case of a camera that has been moved in parallel and can select the particular reverse order of the two-dimensional pixel samples so that the first or last sample can be recorded. Maintaining multiple samples can be accomplished by choosing through a common cache eviction policy such as FIFO.

In order to allow a simple selection of samples, this embodiment designates two types of samples, a "leading" sample and a "trailing" sample. The leading sample is a sample that crosses the left edge of the three-dimensional pixel. Similarly, the trailing sample contacts or crosses the right edge of the three-dimensional pixel. This classifies the sample as a trailing sample that does not traverse any edge. In an embodiment where the width of the two-dimensional pixel is equal to the width of the three-dimensional pixel when reprojected, it is guaranteed to contact or traverse the edge. It is noted that other embodiments do not need to require the width of the two-dimensional pixel to be the same as the width of the pixel in reprojection. For example, in FIG. 5A, the black pixel sample 503 is the last sample in the first pixel of the pixel row 501 and the leading sample in the second pixel in this row. If four samples are used, multiple leading and multiple trailing samples may be maintained and the coverage amount may be adjusted based on the other leading and trailing samples. This was observed to have little visual impact.

Once the two-dimensional sample is determined to correspond to each three-dimensional pixel, sample coverage for each sample must be determined (step 405 ). The sample coverage amount refers to the area of the pixel covered by the corresponding two-dimensional sample. Depending on the reprojection direction of movement the sample may be derived from any direction. For example, a three-dimensional reprojection system using a parallel mobile camera without limitation can derive a sample from a horizontal scan of a two-dimensional image since the reprojection movement occurs only in the horizontal direction. In contrast to the prior art in which sample positions are used to analyze three-dimensional pixel values, embodiments of the present invention track the actual amount of coverage of each sample associated with a given three-dimensional pixel. The coverage amount of one sample may be affected by another sample. The main occurrence of this comes from overlapping samples. Since the depth value is usually the difference between two two-dimensional pixels that are reprojected, the pixels often overlap. As mentioned above, information about the foreground object is usually more important than the background object.

Identifying the sample in step 403 and determining the coverage amount in step 405 may occur simultaneously and selecting the sample may be based on the coverage amount (such as making a choice to keep the sample with the higher coverage amount). It is noted that.

5C illustrates the superposition effect of two different samples 512, 514 and the concept of a "leading" sample and a "end" sample. The area between the two dashed lines represents the overlapping area between two samples. The black sample 512 is the leading sample for the second pixel in row 501 . Gray sample 514 is the last sample for the second pixel. In the sample shown in FIG. 5C, the gray sample 514 also represents a sample closer to the viewer. In this case the coverage of the black leading sample 512 can be reduced by the amount of overlap of the gray sample with the coverage. Coverage of one sample is possible to completely remove the other sample (or to reduce the coverage of another sample to zero).

After determining coverage amounts for all samples corresponding to all three-dimensional pixels, analyzing the final pixel values (eg, color) according to these coverage amounts may be performed (step 407 ). For a given three-dimensional pixel, each sample corresponding to this pixel can be weighted according to the amount of coverage and then combined to produce the final pixel value. The final pixel value may be the final value for any data channel used to define the pixel in the image. For example, without limitation, the final pixel value may be the final color value. However, embodiments of the present invention are not limited to implementations where the final value is a color value.

The weight of the sample used to analyze the final pixel value may cause hole-filling when the total reproducing of 100% is not performed. For example, without limitation, the coverage amount can be divided into the total coverage of the three-dimensional pixels to help balance any gap. If there are no samples for a given three dimensional pixel, such a pixel may be allowed to pass through hole filling. In a particular embodiment, the method shown in FIG. 4 may include an optional hole filling step indicated in step 408 . In general, hole filling can adjust the final value for any pixel where the total weighted coverage amount for all samples contributing to the final value is less than 100%. For example, without limitation, it may involve copying adjacent pixel background values to a given pixel in the hole filling step, as this is usually less noticeable. In some implementations, this may be performed as part of the final pixel value analysis at step 407 through a specific processing sequence during analysis.

Once the three-dimensional pixel values have been analyzed for three-dimensional reprojection, the reprojected image can be displayed as described in step 409 .

It is noted that the coverage amount for the sample can be stored temporarily in a number of different ways. For example, coverage values may be stored in an array. For example, in the case of RGBA data stored as a 32-bit value, there may be two 32-bit values per reprojected pixel (since two samples are used). This does not prevent embodiments of the present invention from using lists or indexes to reduce memory overhead. However, the two sample lists are more likely to introduce additional overhead and are unlikely to reduce it. This can be overcome, for example, by using a list of variable size.

It is noted that in the case of three-dimensional stereoscopic left and right eye images, the pixel values for the left and right eye images can be analyzed as described in FIG. In this case there is a separate set of samples for each reprojection target (ie left and right eyes) and each target needs to be analyzed independently. Both targets can be analyzed simultaneously for performance, but can still be analyzed independently. It is noted that the pixel values for stereoscopic images (eg, images interlaced from different views as used in lenticular lens array displays) can also be analyzed independently. If desired, such images can be analyzed simultaneously, independently, for example using suitably configured parallel processing systems and software.

The left eye and right eye images may be displayed sequentially or simultaneously depending on the nature of the display. For example, left and right eye images may be displayed sequentially in the case of a 3D television display used with active shutter glasses. Alternatively, the left eye and right eye images may be displayed simultaneously in the case of a dual projection type display used with manual 3D view glasses having left and right eye lenses of different colors or different polarizations.

5A-5B further illustrate the use of coverage in multi-sample analysis of three-dimensional reprojection of two-dimensional images. By way of example and not by way of limitation, Figures 5A-5B illustrate a 2X multisample analysis method of three-dimensional reprojection of a two-dimensional image in accordance with one embodiment of the present invention. In a 2X multisample, the reprojection pixel given in row 501 may have up to two two-dimensional samples associated with it. In FIG. 5A, immediately after the two black pixel samples 503A and 503B are followed by two white pixel samples 505A and 505B of the two-dimensional image which are reprojected to the right. It is noted that it is reasonable to expect that most of the pixels in the reprojection of a general scene remain adjacent to adjacent pixels and do not move. Some overlap of the shifted pixels is allowed as a gap between the shifted pixels. The coverage amount of each two-dimensional sample 503A, 503B, 505A, 505B is then determined. In the example shown in FIG. 5A, the first black sample 503A has 50% coverage of the first reprojected pixel 501A . The second black sample 503B has 50% coverage of the first reprojected pixel 501A and 50% coverage of the second reprojected pixel 501B . The first white sample 505A has 50% coverage of the second reprojection pixel 501B and 50% coverage of the third reprojection pixel 501C . The second white sample 505B has 50% coverage of the third 3D pixel 501C and 50% coverage of the fourth reprojection pixel 501D . For this example, it is assumed that the reprojected pixel has been initialized to have no sample or to have zero coverage. Using the coverage amount as a weight and applying the total coverage division as described above to the fourth reprojection pixel, the analyzed reprojection in the analyzed reprojection row 507 , each having a value of black, gray, white and white, respectively. The pixels 507A, 507B, 507C, and 507D are obtained.

However, the situation is different when the coverage is only slightly different. Specifically, in FIG. 5B the two-dimensional pixels 503A, 503B, 505A, 505B are further reprojected to the right, resulting in different coverage amounts for the two-dimensional pixels 503A, 503B, 505A, 505B . In the example shown in FIG. 5B these coverages are as follows. The first black sample 503A has 75% coverage of the first reprojected pixel 501A . The second black sample (503B) is provided with a coverage of 75% to 25% coverage and a second re-projected pixel (501B) of the first re-projected pixel (501A). The first white sample 505A has 25% coverage of the second reprojection pixel 501B and 75% coverage of the third reprojection pixel 501C . The second white sample 505B has 25% coverage of the third 3D pixel 501C and 75% coverage of the fourth reprojection pixel 501D . Again, for this example, it is assumed that the reprojection pixel (including the fourth reprojection pixel 501D ) has been initialized as having no sample or having zero coverage. Using the coverage amount as a weight and applying the total coverage division to the fourth reprojection pixel, the analyzed reprojection pixel 507A in the analyzed reprojection row 507 having values of black, dark gray, white and white, respectively , 507B, 507C, 507D ). A higher gray value for the second analyzed reprojection pixel 507B results in a higher percentage coverage of the second reprojection pixel 501B by the black pixel 503B compared to the white pixel 505A . do.

The prior art method relies on the sample position in the three-dimensional pixel to determine the final three-dimensional pixel value, while the method of the present invention finds the sample coverage of the three-dimensional pixel when analyzing the final color. This allows for a more accurate analysis step when compared to the prior art. As described above, the prior art only allows the transition of pixel values for three-dimensional pixels in which discrete movement occurs upon reprojection. In contrast, the method of the present invention analyzes the final value of each three-dimensional pixel using a sample coverage amount that allows for smooth color transition in response to any movement upon reprojection.

Thus, embodiments of the present invention may not only solve the depth tearing problem during three-dimensional reprojection, but also create a more natural looking three-dimensional image.

When using a stereoscopic display such as a lenticular lens array to display the final three-dimensional image, the amount of movement between the image corresponding to each viewpoint and the image of another viewpoint is expected to be quite small. Embodiments of the present invention also assist in generating recognized jumps while moving between viewpoints. Different objects in the prior art 3D systems tend to "snap" to that location, while the observer changes between different points of view. Embodiments of the present invention allow for more natural movement of all objects in the three-dimensional scene as the viewer moves position.

5A-5B show an example where the samples 503, 505 are moved horizontally with respect to the three-dimensional pixel 501 but not vertically. However, embodiments of the invention may include implementations in which the sample is moved vertically or implementations in which the sample is moved vertically and horizontally.

6 shows a block diagram of a computer device that can be used to implement a method for multiple sample analysis of three-dimensional reprojection of two-dimensional images. Apparatus 600 may generally include a processor module 601 and a memory 605 . The processor module 6 01 may include one or more processor cores. An example of a processing system using multiple processor modules is a cell processor, an example of which is described in detail in, for example, the Cell Broadband Engine Architecture, which is incorporated herein by reference http: // Available online at www306.ibm.com/chip/techlib/techlib.nsf/techdocs/1AEEE 1270EA2776387257060006E61BA / $ file / CBEA_01_pub.pdf.

The memory 605 may be in the form of an integrated circuit, and may be, for example, RAM, DRAM, ROM, or the like. Memory 605 may also be main memory accessible by all of the processor modules. In some embodiments, processor module 601 may have local memories associated with each core. Program 603 may be stored in main memory 605 in the form of processor readable instructions that may be executed in processor modules. Program 603 may be configured to perform multiple sample analysis of three-dimensional projection of a two-dimensional image. The program 603 may be written in any suitable processor readable language, for example C, C ++, Java, assembly, Matlab, Fortran and many other languages. Input data 607 may also be stored in memory. This input data 607 may include information about the identification of the sample to be used and the amount of sample coverage. During execution of program 603 , portions of program code and / or data may be loaded into local storage or memory of processor cores for parallel processing by multiple processor cores.

Device 600 also includes well-known support, such as input / output (I / O) element 611 , power supply (P / S) 613 , clock (CLK) 6 15 , and cache 617 . May include functions 609 . The apparatus 600 may optionally include a mass storage device 619 , such as a disk drive, CD-ROM, tape drive, for storing programs and / or data. The device 600 may optionally include a display unit 621 and a user interface unit 625 to facilitate interaction between the user and the device. For example, display unit 621 may be in the form of a 3D ready television set that displays text, numbers, graphic symbols, or other visual objects as stereoscopic images perceived by 3D view glasses 627 , although It is not limited to the example.

User interface 625 may include a keyboard, mouse, joystick, light pen, or other device that may be used with a graphical user interface (GUI). In addition, the apparatus 600 may include a network interface 623 that enables the device to communicate with other devices via a network, such as the Internet.

Processor 601, memory 605, support functions 609, components of the mass storage device 619, user interface 625, network interface 623 and the system including a visual display 621, 600 are It may be operatively connected to each other via one or more data buses 629 . Such components may be implemented in hardware, software, firmware, or some combination of two or more thereof.

There are many additional ways to simplify parallel processing using multiple processors in the device. For example, one can "unroll" processing loops by duplicating code on two or more processor cores and having each processor core implement code for processing different data. Such an implementation may avoid latency associated with establishing a loop. As applied to the present invention, multiple processors can identify two-dimensional samples corresponding to pixels in three-dimensional reprojection in parallel. Additionally, many processors can determine the amount of three-dimensional pixel sample coverage in parallel or analyze the final color for the three-dimensional pixels in parallel. The ability to process data in parallel saves significant processing time, making the system for multiple sample analysis of three-dimensional reprojection of two-dimensional images more efficient and simplified.

In particular, an example of a processing system capable of implementing parallel processing on three or more processors is a cell processor. There are a number of different processor architectures that can be classified as cell processors. For example, FIG. 7 illustrates one type of cell processor, but is not limited to this example. The cell processor 700 includes a main memory 701 , a single power processor element (PPE) 7 07 , and eight synergistic processor elements (SPEs) 7 11 . In the alternative, the cell processor may be configured in any number of SPEs. With respect to FIG. 7, the memory 701 , the PPE 707 , and the SPE 711 can communicate with each other and the I / O device 715 via a ring type interconnect bus 717 .

The memory 701 includes input data 703 having features in common with the above-described program. At least one of the SPEs 711 may be multi-sample analysis of three-dimensional reprojection of the two-dimensional image command 713 to its local storage LS and / or input data to be processed in parallel, for example as described above. May include some. PPE 707 may include, in its L1 cache, multi-sample analysis of three-dimensional reprojection of two-dimensional picture instruction 709 having features common to the above-described program. Further, instruction 705 and data 703 may be stored in memory 701 for access by SPE 711 and PPE 707 as needed. It should be noted that any number of processes included in the method of multi-sample analysis of three-dimensional reprojection of two-dimensional images of the present invention can be parallelized using a cell processor.

For example, PPE 707 may be a 64-bit Power PC Processor Unit (PPU) with an associated cache. PPE 707 may include an optional vector multimedia expansion unit. Each SPE 711 includes a Synergy Processor Unit (SPU) and Local Storage (LS). In some implementations, local storage can have a capacity of about 256 kilobytes of memory, for example, for data and programs. SPUs are computation units that are less complex than PPU in that they typically do not perform system management functions. SPUs may have a single instruction multiple data (SIMD) capability and typically process data to perform assigned tasks and initiate any necessary data transfer (to access properties set by the PPE). SPUs enable the system to implement applications that require higher computational unit densities and can effectively use the provided instruction set. A significant number of SPUs in a system managed by the PPE allow for cost effective processing over a wide range of applications. For example, the cell processor may feature an architecture known as cell broadband engine architecture (CBEA). In a CBEA-compatible architecture, multiple PPEs can be combined into PPE groups and multiple SPEs can be combined into SPE groups. For example, a cell processor is shown as having a single SPE group and a single PPE group together with a single SPE and a single PPE. In the alternative, the cell processor may include multiple groups of power processor elements (PPE groups) and multiple groups of synergy processor elements (SPE groups). CBEA-compatible processors are described in detail in, for example, the Cell Broadband Engine Architecture available online at https://www306.ibm.com/chips/techlib/techlib.nsf/techdocs/1AEEE1270EA277638725706000E61BA/$file/CBEA_01_pub.pdf. Which is incorporated herein by reference.

According to another embodiment, instructions for multiple sample analysis of three-dimensional reprojection of a two-dimensional image may be stored in a computer readable storage medium. For example, FIG. 8 shows an example of a non-transitory computer readable storage medium 800 according to one embodiment of the invention, but is not limited to this example. Storage medium 800 includes computer readable instructions stored in a format that can be retrieved, converted, and executed by a computer processing device. For example, the computer readable storage medium may be a computer readable memory such as RAM or ROM, a computer readable storage disk (eg, a hard disk drive) for a fixed disk drive, or a removable disk drive. However, such examples are not limited. Computer readable storage medium 800 may also be a flash memory device, computer readable tape, CD-ROM, DVD-ROM, Blu-ray, HD-DVD, UMD, or other optical storage medium.

Storage medium 800 includes instructions 801 for multisample analysis of three-dimensional reprojection of a two-dimensional image. The instructions 801 for multisample analysis of three-dimensional reprojection of a two-dimensional image may be configured to implement multisample analysis in accordance with the methods described above. Specifically, the multisample analysis instruction 801 may include identifying a two-dimensional sample instruction 803 used to determine one or more samples from a two-dimensional image corresponding to a pixel of three-dimensional reprojection.

The multisample analysis instruction 801 may further include an instruction 805 for determining a sample coverage amount configured to determine the area that a given two-dimensional sample occupies in the three-dimensional pixel. In three-dimensional reprojection that implements parallel camera movement, this sampling is performed only in the horizontal direction (ie, the sample is uniform in the vertical direction for any three-dimensional pixel). Multiple sampling can be accomplished in any number of different directions.

In addition, the multisample analysis instruction 801 may further comprise a final pixel color analysis instruction 807 configured to determine the three-dimensional pixel color by combining the weighted sample coverage amount. For example, if a given three-dimensional pixel has a red sample with 80% coverage and a blue sample with 20% coverage, the final pixel color analysis command 807 is purple with a higher intensity of red due to higher coverage. 3D pixel color can be analyzed by being a shade of. It is important to note that any number of weighting methods can be used to analyze the final color based on the sample coverage amount.

The multisample analysis instruction 801 may optionally include a hole fill instruction 808 that , when executed, may adjust the final value for any pixel having a total weighted coverage amount of less than 100% for all samples contributing to the final value. Can be.

The multisample analysis command 801 may further include a 3D reprojection display command 809 that displays a 3D reprojection of the 2D image after the multisample analysis has occurred.

Although embodiments have been described for viewing stereoscopic 3D images using passive or active 3D view glasses, embodiments of the present invention are not limited to these embodiments. In particular, embodiments of the present invention may be applied to stereoscopic 3D video techniques that do not rely on head tracking or passive or active 3D view glasses. Examples of such “glasses free” stereoscopic 3D video technologies are sometimes referred to as autostereoscopic techniques or autostereoscopy. Examples of such techniques include, but are not limited to, techniques based on the use of lenticular lenses. A lenticular lens is an array of magnifying lenses designed to magnify different images when viewed from slightly different angles. Different images may be selected to provide a three-dimensional viewing effect when viewing the lenticular lens from different angles. The number of generated images increases in proportion to the number of viewpoints for the screen.

More specifically, in a lenticular lens video system, reprojected images of a scene at slightly different viewing angles may be generated from depth information and an initial 2D image for each pixel of the image. By using the reprojection technique, progressively different views of the scene at different viewing angles can be generated from the initial 2D image and depth information. Images representing different views can be divided into strips and displayed in an interlaced manner on an autostereoscopic display having a display screen disposed between the lenticular lens array and the view position. The lenses that make up the lenticular lenses may be cylindrical magnifying lenses that are aligned with the strips, and are generally twice the width of the strips. The viewer perceives different views of the scene according to the angle of view of the screen. Different views may be selected to provide depth welcome in the scene being displayed.

In addition, some embodiments of the present invention may solve the depth tearing problem in the case of three-dimensional reprojection of a two-dimensional image and include generating more than one image for reprojection, although embodiments are non-reprojected. More generally applicable in 3D case. Multiple sample analysis of the reprojected pixels described herein can produce better image quality in the reprojected image due to the gradient of movement. Such multiple sample analysis can be implemented, for example, during normal rasterization of the reprojected image.

 In addition, in some three-dimensional implementations, it may not be necessary to generate more than one image. For example, in the case of a stereoscopic display, it may not be necessary to generate both left and right eye images via reprojection. Alternatively, only one new picture may be generated. For example, you can start with color and depth information for each pixel for a left eye image, and create a corresponding right eye image through reprojection (or vice versa), so that in a stereoscopic display Sufficient images may be generated for the display of. This involves generating only a reprojected single picture.

While the foregoing is a complete description of the preferred embodiments of the present invention, it is possible to use various alternatives, modifications, and equivalents.

Although the invention has been described in great detail with respect to some preferred versions of the invention, other versions are possible. Accordingly, the spirit and scope of the claims should not be limited to the description of the preferred versions contained herein. Instead, the scope of the invention should be determined with reference to the claims, along with the full scope of equivalents of the claims.

)에서 특정한 바와 같이 "수단" 또는 "단계" 절로서 해석되어선 안 된다. 구체적으로, 본 명세서의 청구범위에서 "하는 단계"라는 사용은 미국 특허법(

)의 규정을 적용하려는 것이 아니다.All features disclosed in this specification (including claims, summaries, and drawings) may be replaced by alternative features that serve the same, equivalent, or similar purpose unless explicitly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is a general example of a series of equivalent or similar features. Any feature, whether preferred or not, may be combined with any other feature, whether desired or not. In the following claims, the indefinite article "a" or "an" refers to one or more quantities of items following the phrase, unless expressly stated otherwise. Any element in a claim that does not expressly refer to "means for" performing a particular function is described in US Patent Law (

) Shall not be construed as a "means" or "step" clause as specified in. Specifically, the use of "steps" in the claims herein refers to US patent law (

Is not intended to apply.

Those skilled in the art are interested in paying attention to all papers and documents submitted concurrently with this specification and open for examination by the public with this specification, the contents of which are incorporated herein by reference.

Claims (32)

A method for multi-sample resolving of reprojection of two-dimensional images,
a) identifying one or more samples of a two-dimensional image for each pixel in the reprojection;
b) determining one or more sample coverage amounts for each pixel of the reprojection, wherein each sample coverage amount identifies an area of a pixel covered by a corresponding two-dimensional sample;
c) analyzing a final value for each pixel of the reprojection by combining each two-dimensional sample associated with the pixel according to the weighted sample coverage amount; And
d) indicating the reprojection. The method of claim 1, wherein identifying each sample in step a) is accomplished using a horizontal scan. The method of claim 1, wherein step a) involves identifying up to two samples for each pixel in the three-dimensional reprojection. The method of claim 1, wherein step b) further comprises determining whether a particular sample associated with the three-dimensional pixel has a leading sample in an adjacent pixel. 5. The method of claim 4, wherein step b) further comprises determining whether a particular sample associated with the three-dimensional pixel has a trailing sample in the adjacent pixel. The method of claim 1, wherein steps a), b), c) and d) are performed on corresponding left and right eye images for stereoscopic display. The method of claim 1, wherein the two-dimensional image is a first eye view for stereoscopic display, and steps a), b) and c) are performed for the corresponding second eye view. . The method of claim 1, wherein steps a), b), c) and d) are performed on at least two images corresponding to at least two different views for stereoscopic display. The method of claim 1, wherein b) comprises determining an overlap between the two or more samples, and c) wherein the c) includes a coverage amount for one of the two or more samples having a depth further from the observer. Reducing the amount of coverage overlap with one or more other samples of the two or more samples. 2. The method of claim 1, further comprising adjusting the final value for any pixel where the total weighted coverage amount for all samples contributing to the final value is less than 100%. As a multiple sampling device,
A processor;
Memory; And
Computer coded instructions implemented in the memory and executable by the processor,
The computer coded instructions are configured to implement a multi-sample analysis method of reprojection of a two-dimensional image,
The method comprises:
a) identifying one or more samples of a two-dimensional image for each pixel in the reprojection;
b) determining one or more sample coverage amounts for each pixel of the reprojection, wherein each sample coverage amount identifies an area of a pixel covered by a corresponding two-dimensional sample; And
c) analyzing the final value for each pixel of the reprojection by combining each two-dimensional sample associated with the pixel according to the weighted sample coverage amount. 12. The multisampling device of claim 11, further comprising a three-dimensional visual display configured to display reprojection after multisample analysis. 13. The method of claim 12,
d) displaying the reprojection on the three-dimensional visual display. 12. The multiple sampling apparatus of claim 11, wherein identifying each sample in step a) is accomplished using a horizontal scan. 12. The apparatus of claim 11, wherein step a) involves identifying at most two samples for each pixel in the three-dimensional reprojection. 12. The multiple sampling apparatus of claim 11, wherein step b) further comprises determining whether a particular sample associated with the pixel under reprojection has a leading sample at an adjacent pixel. 17. The multisampling apparatus of claim 16, wherein step b) further comprises determining whether a particular sample associated with the pixel under reprojection has a trailing sample at an adjacent pixel. 12. The multiple sampling apparatus of claim 11, wherein steps a), b), and c) are performed on corresponding left and right eye images for stereoscopic display. 12. The multiple sampling apparatus of claim 11, wherein steps a), b), c), and d) are performed on at least two images corresponding to at least two different views for stereoscopic display. 12. The multiple sampling apparatus of claim 11, wherein the two-dimensional image is a first eye view for stereoscopic display, and steps a), b) and c) are performed for a corresponding second eye view. 12. The method of claim 11, wherein step b) comprises determining an overlap between two or more samples, and step c) recalling an amount of coverage for one of the two or more samples having a depth further from the observer. And reducing the amount of coverage overlap with one or more other samples of the two or more samples. 12. The apparatus of claim 11, wherein the method further comprises adjusting the final value for any pixel where the total weighted coverage amount for all samples contributing to the final value is less than 100%. . As a computer program product,
A non-transitory computer readable storage medium having program code embodied in the computer readable storage medium for multiple sample analysis of reprojection of a two-dimensional image,
The computer program product,
a) computer readable program code means for identifying one or more samples of a two-dimensional image corresponding to each pixel of the reprojection;
b) computer readable program code means for determining, for each pixel of three-dimensional reprojection, one or more sample coverage amounts, each identifying an area of the pixel covered by the corresponding two-dimensional sample;
c) computer readable program code means for analyzing a final value for each pixel of said three-dimensional reprojection by combining each two-dimensional sample associated with the pixel according to the weighted sample coverage amount; And
d) computer readable program code means for displaying the reprojection. 24. The computer program product of claim 23, wherein identifying each sample in step a) is accomplished using a horizontal scan. 24. The computer program product of claim 23, wherein step a) involves identifying at most two samples for each pixel of the three-dimensional reprojection. 24. The computer program product of claim 23, wherein step b) further comprises determining whether a particular sample associated with the three-dimensional pixel has a leading sample in an adjacent pixel. 27. The computer program product of claim 26, wherein step b) further comprises determining whether a particular sample associated with the three-dimensional pixel has a trailing sample at an adjacent pixel. 24. The computer program product of claim 23, wherein steps a), b), c) and d) are performed on corresponding left and right eye images for stereoscopic display. 24. The computer program product of claim 23, wherein steps a), b), c), and d) are performed on two or more images corresponding to two or more different views for stereoscopic display. 24. The computer program product of claim 23, wherein the two-dimensional image is a first eye view for stereoscopic display, and steps a), b) and c) are performed on the corresponding second eye view. 24. The method of claim 23, wherein step b) comprises determining an overlap between the two or more samples, and step c) determining the amount of coverage for one of the two or more samples having a depth further from the observer. Reducing the amount of coverage overlap with one or more other samples of the two or more samples. 24. The computer program product of claim 23, wherein the method further comprises adjusting the final value for any pixel where the total weighted coverage amount for all samples that contributed to the final value is less than 100%.

Images