TWM529333U - Embedded three-dimensional image system - Google Patents

Description

Stereo embedded image system

This creation is about an imaging system, especially a stereoscopic embedded image system.

When a person views the stereoscopic (3D) image displayed on the display, since the position of the eyes of the person is left-right, the eyes of the same display are seen from different angles, so that the images seen by both eyes are made. It will be slightly different. The human brain combines the two images viewed by both eyes into one image, and through the mechanism of the depth perception of the objects through the eyes, the image depth is judged according to the similarity and difference of the binocular images. To create a stereoscopic visual effect.

The conventional method of generating stereoscopic images is to combine two-dimensional (2D) images with corresponding depth information to synthesize images of binoculars to be separately provided, wherein the depth information is grayscale images. However, after the composite image is synthesized by the two-dimensional image and the depth information, the synthesized image has some pixels without color information, that is, a hole, especially in a place where the depth changes strongly, a huge cavity is generated, so In the hollow area of the synthetic image, the human brain cannot be combined into a complete stereoscopic image. A common solution is to smooth the depth map. Although the size of the hole can be reduced, the original depth information is destroyed, and the stereoscopic effect of the synthesized image is greatly reduced.

Therefore, the main purpose of this creation is to provide a stereoscopic embedded image system to reduce the impact of voids in stereoscopic images.

The creation of the stereoscopic embedded image system comprises: a high-quality multimedia interface receiving end, receiving an original color image data; and a functional programmable logic gate array device electrically connected to the high-quality multimedia interface receiving end to receive the original The color image data, the field effect programmable logic gate array device comprises: a stereo matching unit, converting the original color image data into a color image data and a depth image data; and a depth image plotter comprising: a depth a pre-processing unit, wherein the input end is connected to the stereo matching unit to receive the color image data and the depth image data, and generate a pre-processing result according to the color image data and the depth image data, the pre-processing result includes two Dimensional image and depth map; two image warping units, wherein the input end is connected to the stereo matching unit and the depth map pre-processing unit, respectively generating a first image according to the pre-processing result and the color image data distortion Data and a second image data; two patching sequence estimation units, which lose The input end is respectively connected to the two image distorting units to respectively receive the first image data and the second image data, and respectively generate an estimate of the repair order of the first image data and the second image data respectively. a first estimated image repairing data and a second estimated image repairing data; and two optimal patch matching units, wherein the input ends are respectively connected to the two patching sequence estimating units to respectively receive the first estimated image repairing data and The second estimated image is repaired, and the optimal patch is matched according to the first estimated image repair data and the second estimated image repair data, respectively, to generate a first simulated image data and a second simulation respectively. The image data; and a high-quality multimedia interface transmitting end electrically connected to the field-effect programmable logic gate array device to output the first analog image and the second analog image.

According to the present invention, as a whole, the first image and the second image are generated by sequentially matching the color image data and the depth image data through image distortion, patching order estimation and patch matching, wherein This creation uses the fill order estimation and the best patch search to fill the holes, so that the size of the synthesized holes is reduced and the distortion of the image is reduced without greatly affecting the stereoscopic effect of the composite image.

Referring to FIG. 1, the stereoscopic embedded image system includes a high definition multimedia interface (HDMI) receiving end 10, and a Field Programmable Gate Array (FPGA) device 11 and A high-quality multimedia interface sender 12. The input end of the field effect programmable logic gate array device 11 is electrically connected to the high quality multimedia interface receiving end 10, and the high quality multimedia interface receiving end 10 is electrically connected to the field effect programmable logic gate array device 11 The output.

The high-definition multimedia interface receiving end 10 and the high-definition multimedia interface transmitting end 12 are widely used digital audio and video transmission media, which can receive or transmit uncompressed audio signals and video signals. In the present embodiment, the high-definition multimedia interface receiving end 10 can be connected to an image generating device via a transmission line, and the image generating device can be a digital camera, a mobile device or a computer product to receive a pending image from the image generating device. An original color image data RXdata and a synchronization signal SYNC. The high-definition multimedia interface transmitting end 12 can be connected to a terminal display device via another transmission line, and the terminal display device can be a display or a computer product.

The field effect programmable logic gate array device 11 is a programmable logic device commonly used in the field of image processing technology. Generally, the field effect programmable logic gate array device 11 includes a stereo matching unit 110 and a depth-of-image-based rendering (DIBR) device 111, the stereo matching unit 110 converts the original color image data RXdata into a color image data IMcolor and a depth image data IMdepth, which is the stereo matching unit 110 The existing function of the depth image plotter 111 operates according to the synchronization signal SYNC and will not be described in detail herein.

Referring to FIG. 2 , the depth image plotter 111 includes a depth map pre-processing unit 112 , two image warping units 113 , two patch sequence estimating units 114 and an optimal patch matching unit 115 .

The input end of the depth map pre-processing unit 112 is connected to the stereo matching unit 110 to receive the color image data IMcolor and the depth image data IMdepth processed by the stereo matching unit 110, and according to the color image data IMcolor and the depth The image data IMdepth performs a depth map pre-processing step to generate a pre-processing result R, which includes a two-dimensional (2D) image map and a depth map.

The input ends of the two image warping units 113 are connected to the stereo matching unit 110 and the depth map pre-processing unit 112, respectively performing an image warping step according to the pre-processing result R to respectively generate an image warping step. For example, if the first image data IM1 is to be viewed by the user's left eye, the second image data IM2 is for the user's right eye to view.

The input ends of the two patching sequence estimating units 114 are respectively connected to the two image warping units 113 to respectively receive the first image data IM1 and the second image data IM2 to respectively respectively the first image data IM1 and the second image. The data IM2 performs a patching sequence estimation step, and correspondingly generates a first estimated image patching material IME1 and a second estimated image patching material IME2.

The input ends of the two optimal patch matching units 115 are respectively connected to the two patching sequence estimating units 114, and the outputs of the two optimal patch matching units 115 are respectively connected to the high-quality multimedia interface transmitting end 12. The two optimal patch matching units 115 respectively receive the first estimated image repairing material IME1 and the second estimated image repairing material IME2 of the two patching sequence estimating units 114, and respectively repairing the data according to the first estimated image IME1 and The second estimated image repairing material IME2 performs an optimal patching step to generate a first analog image data IMS1 and a second analog image data IMS2, respectively, and then transmitted to the high-quality multimedia interface transmitting end 12 .

The first analog image data IMS1 and the second analog image data IMS2 are transmitted to the terminal display device through the high-quality multimedia interface transmitting end 12, so that the synthesized first simulated image data IMS1 is displayed by the terminal display device. And the second analog image data IMS2. For example, the first analog image data IMS1 can be viewed by the left eye corresponding to the person, and the second analog image data IMS2 can be viewed by the right eye of the person, so that the user views the terminal display device. In the picture, the brain combines the first analog image data IMS1 and the second analog image data IMS2 into an image having a stereoscopic (3D) visual effect.

Through the setting of the depth map pre-processing unit 112, the generated pre-processing result R is performed more smoothly for the subsequent image warping and hole filling steps.

The operation of the depth map pre-processing unit 112, the image warping unit 113, the patching order estimating unit 114, and the optimal patch matching unit 115 will be separately described below.

The depth map pre-processing unit 112 determines, according to the color image data IMcolor and the depth image data IMdepth, that the block edge of an object belongs to a depth edge or an image edge.

For the judgment of the depth edge, a difference degree D is first described, and the degree of difference D is expressed as follows: (1)

In the formula (1), d _max represents the maximum difference value of the horizontal displacement of the two corresponding points in the left and right images, and d _min represents the minimum difference in the horizontal displacement of the two corresponding points in the left and right images. The value, d, represents the horizontal displacement of the two corresponding points in the left and right images. The horizontal displacement d is calculated by the focal length f of the camera as follows: (2) b is the reference distance between the two cameras, z is the vertical distance between the object and the camera, and f is the focal length of the camera.

The depth difference value D _{diff is} defined as follows: (3)

In the formula (3), D _{i , j} denotes a depth block centered on a pixel located at coordinates ( i , j ).

Therefore, the depth map pre-processing unit 112 determines the depth edge according to the following formula: (4)

In the formula (4), λ is a first judgment threshold value, and the first judgment threshold value λ is expressed as follows: (5)

b is the reference distance of the two cameras, H is the area of the so-called hole, and f is the focal length of the camera. If DepthEdge=1, it is judged as the depth edge.

For the determination of the edge of the image, the depth map pre-processing unit 112 first calculates an image intensity value Var _{i , j} of the block of the object _, and the image intensity value Var _{i , j is} expressed as follows: (6)

In equation (6), Var _{i , j} represents the expected value of the block centered on the ( i , j ) pixel of the object, μ is the center of all image blocks and centered on the ( i , j ) pixel The average of the block differences, Y _{i , j} represents the pixel intensity at coordinates ( i , j ), which is expressed as follows: (7)

Therefore, the depth map pre-processing unit 112 determines the image edge according to the following formula: (8)

In the formula (8), γ is a second judgment threshold value, and the second judgment threshold value γ is expressed as follows: (9)

In the formula (9), Ng represents the number of different gray scale values, and h _i represents the gray scale value of the histogram.

On the other hand, if the block of the object is determined to be both the depth edge and the image edge, the depth map pre-processing unit 112 performs a depth alignment step and an extension step, the depth alignment step is to set the depth value of the block. d _{p, q is} set to a minimum depth, as follows: (10)

In the formula (10), 1≦p≦m, 1≦q≦n. (p, q) is the coordinate value of the pixel, and m and n are the threshold values.

Furthermore, the edge direction of this creative judgment depth expansion is as follows: (11)

In the aforementioned extended step, k neighbor pixels are copied at the edge of the depth as follows: (12)

In the formula (12), 1≦ p ≦ k . k is the threshold value.

The depth map pre-processing unit 112 will output an image map and a depth map to the image warping unit 113, making the image warping unit 113 easier to process and execute.

Each of the image warping units 113 mainly projects the pixel points originally on the plane of the color image data IMcolor to the corresponding real three-dimensional points in the three-dimensional coordinate system, and then projects onto the virtual image plane of the terminal display device. The focal length of the camera is denoted by f , and the reference distance of the camera is denoted by t _x . A point with depth in the X , Y, and Z dimensions is expected to be projected to the pixel location of the camera image plane , with Above, the following two expressions indicate: (13) (14)

In the equations (13) and (14), xc and xr are obtained from the information of the color image data IMcolor and the depth image data IMdepth, respectively, and Zc is a zero-parallax setting plane. Z is the vertical distance between the object and the camera. Equations (13) and (14) assume that the object is in front of the terminal display device (i.e., between the user and the terminal display device). When presenting a 3D video, it is necessary to define an appropriate range of objects for the user to feel different depths when viewing the same object from different angles, and the perceived depth is proportional to the screen's perceived depth, as shown in the following equation: (15)

In equation (15), p is the screen parallax, Z is the vertical distance between the object and the camera, e is the distance between the two eyes of the user, and d is the horizontal displacement of the two corresponding points in the left and right images. Refer to the difference value. From the equation (15), the viewing distance Z symbolizes the true distance of the camera in the real world, but the depth map represents the distance of the normalized 8 gray scale. Therefore, each of the image warping units 113 can convert the depth value using linear quantization, as shown in the following formula (16), g is defined as the gray scale in the depth image data IMdepth, and D represents the object and the camera in the three-dimensional space. The true distance, when the object is in the position of Z _c , the images of the left and right eyes will coincide, and the stereo depth will be exactly equal to the position of the screen. That is to say: objects after zero parallax (Zero Parallax) will look like they are inside the screen. Objects before Zero Parallax will have the feeling of jumping out of the screen. Therefore, this creation sets the zero parallax to Z _c equal to the viewing distance. (16)

Among the formulas (13) and (14) Can be brought in by (16), and the following formula is derived: (17)

In the formula (17), It can be expressed as follows, wherein the multiplication can be implemented using a shift register. (18)

In the formula (18), without the complicated division of the equation (17), unlike the conventional technique using a large number of complicated mathematical expressions, this creation will make the step of image distortion easier to use hardware (ie: Each of the image warping units 13 can be implemented by including a displacement register executing the operation of (18). In the foregoing, the formulas (1) to (17) are prior art in the field of image processing technology, and the process thereof is not deduced in detail herein.

The image warping unit 113 supplies a 3D image to which the patch needs to be repaired to the patching order estimating unit 114.

It should be noted that due to the difference of the user's viewpoint, some of the virtual left and right eye images will become visible in the area where the original image is shaded, but there is no information in the left and right eye images to form a cavity. . These newly exposed areas are called disocclusions in computer graphics. The information of unfilled areas cannot be found in the original color image data IMcolor, nor in the associated depth image data IMdepth. This creation uses the methods of average textures of neighboring pixels to fill these newly exposed areas. This method is called hole-filling.

Regarding each of the patching order estimating units 114, according to Criminisi et al. (Criminisi, A., Perez, P., Toyama, K.. "Region filling and object removal by exemplar-based image inpainting," IEEE Transactions on Image Processing, 13(9), pp.1200-1212, 2004), should be transmitted first along the linear structure before "cavity filling", so that the structure extension and the void boundary shape are connected. Criminisi et al. designed a priority function. Determine the order of filling, as briefly described below. Given a patch Ψ _p whose center point is p and p is in the hole region Ω , the weight P ( p ) is calculated as follows: (19)

In the formula (19), the trust item C(p) and the data item D(p) are respectively defined as: (20) (twenty one)

In equations (20) and (21), Is the illumination at p point (isophote), Φ is the non-void area, Is an orthogonal operation symbol, np is a unit vector orthogonal to ∂Ω, and a is a normalization factor. In short, C(p) is the proportion of non-void in Ψp, and then D(p) is the inner product of the gradient term ÑIp and the general term np. Criminisi et al. define the weight P(p) as the product of the trust item C(p) and the data item D(p), and the data item D(p) gives a larger weight block to the independent block, and the trust item C (p) Give a higher weight to the block surrounded by the non-voided area. According to the filling procedure inside the hole, since the trust item C(p) is between the range [0, 1], the trust item C(p) becomes smaller every time it is updated, indicating that the more uncertain the hole is filled inside the hole. The image value, because the weight P(p) may decay to zero, P(p)=C(p)D(p) will cause the illumination to stop transmitting. In view of this, this creation is different from Criminisi et al., which defines the weight P(p) as follows: (twenty two)

The depth term D _p ( p ) gives the background area a high weight, which is defined as follows: (twenty three)

Therefore, in this creation, the existing surrounding points can be used to repair the lost area.

The patching sequence estimating unit 114 outputs the pre-patched hole sequence to the optimal patch matching unit 115, and then the best patch matching unit 115 searches for the most matching patch for patching.

With respect to each of the optimum patch matching units 115, in order to search for the most suitable patch, the present calculation calculates the similarity for each pixel between the patches. In DIBR, there is not only image information, but also depth information. The similarity measure method of each of the best patch matching units 115 is called a depth-based gray-level distance (DBGLD) method. Depth information to gray scale distance method (refer to L. Zhang and WJ Tam, "Stereoscopic image generation based on depth images for 3D TV," IEEE Transactions on Broadcasting, vol. 51, No. 2, pp. 1191-199, June Produced by 2005).

[Psi] so that there is a cavity to be filled patches, Ψ vertical neighbor patches and [Psi] is the vertical distance VD (ΨD) is defined as: (twenty four)

In the formula (24), v denotes a vertical neighbor patch of Ψ.

Furthermore, Ψ horizontal neighbor patches with smooth gray scale level from [Psi] HD (Ψ) is defined as: (25)

In the equation (25), h denotes a horizontal neighbor patch that is Ψ.

The depth-based gray-level distance (DBGLD) is defined as: (26)

Among them, Ψ 's most suitable patch Ψ ¢ can be obtained as follows: (27)

Therefore, the first analog image data IMS1 and the second analog image data IMS2 are generated through the operation processing of the two optimal patch matching units 115.

This creation uses a public depth image sequence provided by Microsoft Research that provides a resolution of 1024. A continuous image of 768 and its corresponding depth map. This creation uses two sets of stereoscopic films, "ballet" and "break-dancers". The "ballet" film has a large number of depth discontinuities in two different depth ranges, and a large number of depth discontinuities results in a large number of unfilled areas at different depth levels. Furthermore, "break-dancers" movies have a large number of objects at almost the same depth level, and progressive depth discontinuities can result in small unfilled areas in the movie.

Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are the reference for evaluating imaging quality. The PSNR and SSIM performances of the "ballet" films were 30.8883 and 0.6834 respectively, while the PSNR and SSIM performances of the "break-dancers" films were 31.0556 and 0.6767, respectively, which showed excellent image quality.

10‧‧‧High-quality multimedia interface receiver
11‧‧‧ Field effect programmable logic gate array device
110‧‧‧ Stereo matching unit
111‧‧‧Deep image plotter
112‧‧‧Depth map pre-processing unit
113‧‧‧Image Distortion Unit
114‧‧‧ Patching order estimation unit
115‧‧‧Best patch matching unit
12‧‧‧High-quality multimedia interface sender
RXdata‧‧‧ original color image data
SYNC‧‧‧Synchronized signal
IMcolor‧‧‧ color image data
IMdepth‧‧‧Deep image data
R‧‧‧ pre-processing results
IM1‧‧‧ first image data
IM2‧‧‧Second image data
IME1‧‧‧First Estimated Image Repair Information
IME2‧‧‧ second estimated image repair data
IMS1‧‧‧ first simulated image data
IMS2‧‧‧Second analog image data

Figure 1: Block diagram of an embodiment of the present stereoscopic embedded image system. FIG. 2 is a schematic flow chart of an embodiment of the present stereoscopic embedded image system.

10‧‧‧High-quality multimedia interface receiver

11‧‧‧ Field effect programmable logic gate array device

110‧‧‧ Stereo matching unit

111‧‧‧Deep image plotter

12‧‧‧High-quality multimedia interface sender

RXdata‧‧‧ original color image data

SYNC‧‧‧Synchronized signal

IMcolor‧‧‧ color image data

IMdepth‧‧‧Deep image data

IMS1‧‧‧ first simulated image data

IMS2‧‧‧Second analog image data

Claims (2)

A stereoscopic embedded image system, comprising: a high-quality multimedia interface receiving end, receiving an original color image data; and an effect programmable logic gate array device electrically connected to the high-quality multimedia interface receiving end to receive the original The color image data, the field effect programmable logic gate array device comprises: a stereo matching unit, converting the original color image data into a color image data and a depth image data; and a depth image plotter comprising: a depth a pre-processing unit, wherein the input end is connected to the stereo matching unit to receive the color image data and the depth image data, and generate a pre-processing result according to the color image data and the depth image data, the pre-processing result includes two Dimensional image and depth map; two image warping units, wherein the input end is connected to the stereo matching unit and the depth map pre-processing unit, respectively generating a first image according to the pre-processing result and the color image data distortion Data and a second image data; two patching sequence estimation units, which lose The input end is respectively connected to the two image distorting units to respectively receive the first image data and the second image data, and respectively generate an estimate of the repair order of the first image data and the second image data respectively. a first estimated image repairing data and a second estimated image repairing data; and two optimal patch matching units, wherein the input ends are respectively connected to the two patching sequence estimating units to respectively receive the first estimated image repairing data and The second estimated image is repaired, and the optimal patch is matched according to the first estimated image repair data and the second estimated image repair data, respectively, to generate a first simulated image data and a second simulation respectively. The image data; and a high-quality multimedia interface transmitting end electrically connected to the field-effect programmable logic gate array device to output the first analog image and the second analog image. The stereoscopic embedded image system of claim 1, wherein each of the image warping units includes a displacement register.