KR101960852B1 - Apparatus and method for multi-view rendering using background pixel expansion and background-first patch matching - Google Patents

Abstract

An apparatus and method for reconstructing a hole that occurs in multi-view rendering are provided. Holes in the output view are restored by using images temporally adjacent to the image associated with the image watermark. Holes are restored by scaling the pixels adjacent to the holes. Further, the hole is restored by searching for the patch most similar to the area including the hole, and using the searched patch to fill the area including the hole.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a multi-view rendering apparatus and method using background pixel expansion and background priority patch matching,

The following embodiments relate to a multi-view rendering apparatus and method.

In order to generate a 3D image, multi-view 3D images having a wide viewing angle must be continuously displayed.

However, it is not easy to capture the images of the multi-viewpoints and to transmit the captured multi-viewpoint images in real time. This is because it is not easy to store and transmit photographed data, including physical limitations of the photographing system.

Thus, the 3D image generation apparatus can generate a 3D image using only a small number of (for example, two to three) input views (or reference views)) and reproduce the generated 3D image A 3D display device may generate multiple output views by interpolating or extrapolating the input views.

One embodiment provides an apparatus and method for recovering a hole in an output view image generated by image watermarking.

According to one aspect, there is provided a processor for controlling one or more processor-executable units, an image generating unit that generates an image of an output view by image warping using an image of a reference view and binocular disparity information of the image of the reference view And an adjacent image-based hall reconstructing unit for reconstructing the hall generated by the image overwriting using the wiping unit and the images temporally adjacent to the image related to the image warping.

The temporally adjacent images may be images temporally adjacent to the image of the reference view.

The temporally adjacent images may be temporally adjacent to the image of the output view.

The adjacent image-based hall reconstruction unit may use a color value of pixels corresponding to the first pixel in the temporally adjacent images to reconstruct a first pixel in the hole.

Wherein the neighboring-image-based hall reconstructing unit selects the pixels corresponding to the first pixel in the temporally adjacent images based on the movement if the reference view image and the temporally adjacent images are entirely moved with time .

The adjacent image-based hole restoration unit may use only pixels other than the hole pixels among the pixels corresponding to the first pixel to restore the hole.

The adjacent image-based hall reconstruction unit may use only a background pixel among the pixels corresponding to the first pixel to reconstruct the hall.

The image processing apparatus may further include a buffer interval setting unit for enlarging the hole generated by the image watermarking.

Wherein the buffer section setting section enlarges the hole by considering the buffer region adjacent to the hole as the hole, and the adjacent image-based hole restoration section restores the color value of the first pixel when the first pixel is a pixel in the buffer region The first pixel may be restored.

The image processing apparatus may further include a binocular parallax crack detecting unit for setting a crack in an image of the output view as an image.

The binocular parallax crack detecting unit may detect cracks of pixels in the output view in which the sum of differences of binocular parallax values with adjacent pixels is larger than a predefined threshold value.

According to another aspect, there is provided a processor for controlling one or more processor-executable units, the processor comprising: a processor for generating processor-executable units for generating an image of an output view by image warping using an image of a reference view and binocular disparity information of the image of the reference view And a neighboring pixel scaling-based hole reconstruction unit for reconstructing the hole generated by the image waving by scaling one or more pixels adjacent to the hole.

The adjacent pixel scaling-based hole restoration unit may scale only the background pixel among the one or more pixels.

One or more of the hole pixels and the one or more pixels constituting the hole may be on the same horizontal line.

The adjacent pixel scaling based hole restoration unit may scale the one or more pixels in a direction perpendicular to a gradient of a background pixel adjacent to the hole, and the background pixel may be one of the one or more pixels.

According to another aspect, an image of an output view is generated by a processor controlling one or more processor-executable units, an image of a reference view, and image warping using binocular disparity information of the image of the reference view And an optimal patch search-based hole restoration unit for searching the background most similar to the area including the image including the image, the ping unit and the hole generated by the image overwriting, and restoring the hole by using the searched patch. A processing device is provided.

The area including the hole may be composed of a hole area and a background area, and the optimal patch search-based hole restoration part may restore the hole area using a part corresponding to the hole area in the searched patches.

Wherein the optimal patch search-based hole restoration unit searches for a first patch for a first area, searches for a second patch for a second area, and uses the average value of the first patch and the second patch to search for the first patch And a hole portion overlapping in the second region.

According to another aspect of the present invention, there is provided an image processing method comprising: an image overwriting step of generating an output view image by image-waving an image of the reference view based on binocular disparity information of a reference view image by a processor; There is provided an image processing method including an adjacent image-based hall reconstruction step of reconstructing a hall created in the output view using the reconstructed image.

The image processing method may further include a buffer interval setting step of enlarging the hole.

The image processing method may further include a binocular parallax crack detecting step of setting a crack among the images of the output view as the blank.

The image processing method may further include an adjacent pixel scaling-based hole restoration step of restoring the hole by scaling one or more pixels adjacent to the hole.

The image processing method may further include searching for a patch most similar to an area including the hole from the background and restoring the hole by using the retrieved patch.

According to another aspect, there is provided a processor for controlling one or more processor-executable units, an image generating an output view image based on an image of a reference view, and binocular disparity information of the image of the reference view, An image reconstruction unit for reconstructing a reconstructed image of a current frame of an output image using the background information of images temporally adjacent to the image related to the generation of the output view, And generating a multi-view generated by the multi-view generator.

According to another aspect, there is provided a processor for controlling one or more processor-executable units, an image generator for generating an image of an output view based on at least one reference view image, A binocular parallax crack detector for detecting a crack in a predefined object of an image, said predefined object having different binocular disparity values assigned to different parts of said predefined object, said crack having at least one reference view Wherein the generated image is generated in the predefined object due to the generation of an image of the output view based on the image of the output view and the background information of frames temporally adjacent to the image associated with the generation is used A multi-view generation unit for restoring the hall existing in the current frame of the generated output view image, , Including multi-view is generated is provided.

The temporally adjacent frames may be frames temporally adjacent to the image of the reference view.

The temporally adjacent frames may be frames temporally adjacent to the image of the output view.

According to another aspect, there is provided a method comprising: generating, by a processor, an image of an output view based on at least one reference view image; detecting a crack in a predefined object of the image of the generated output view, Wherein the defined object has different binocular disparity values assigned to different parts of the predefined object and the crack is caused by the presence of the preview image due to the generation of the image of the output view based on the image of the at least one reference view Restoring the holes present in the current frame of the generated output view image by using the background information of temporally adjacent frames in the image related to the generation, A multi-view generation method is provided, comprising the steps of:

According to another aspect of the present invention, there is provided a display device including an image processing apparatus, comprising: an image and a ping section for generating an image of an output view based on binocular disparity information of an image of a reference view and an image of the reference view; An adjacent image-based hall reconstruction unit for reconstructing a hall existing in a current frame of an image of the generated output view using background information of images temporally adjacent to an image related to generation of the output image, And a control unit for generating a signal to be displayed by the display device based on an image of the generated output view having the hole restored by the adjacent image-based hole restoration unit A display device is provided.

An apparatus and method are provided for restoring a hole in an output view image generated by image watermarking.

Figure 1 illustrates a view generation method based on three input views according to an example.
Figure 2 illustrates a frame generation method of an extrapolated view according to one embodiment
3 is a structural diagram of an image processing apparatus according to an embodiment.
FIG. 4 illustrates restoration of a hole using images temporally adjacent to an image related to image warping according to an example.
FIG. 5 illustrates hole enlargement by setting a buffer interval according to an example.
Fig. 6 illustrates generation of cracks and hole setting by detecting binocular parallax cracks according to an example.
Figure 7 illustrates adjacent pixel scaling according to an example.
Figure 8 illustrates adjacent pixel scaling using a background pixel according to an example.
Figure 9 illustrates scaling in a direction perpendicular to the gradient of the background according to an example.
FIG. 10 illustrates restoration of holes based on optimal patch search for work.
Fig. 11 illustrates restoration of holes using overlapping patches according to an example.
12 is a flowchart of an image processing method according to an embodiment.
13 shows a display device including an image processing apparatus according to an embodiment.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

Figure 1 illustrates a view generation method based on three input views according to an example.

The subject 110 to be photographed is composed of a foreground and a background 112. [ The foreground includes a first object (114) and a second object (116).

Depending on the viewpoint of the observer, the relative position of the first object 114 and the second object 116 with respect to the background 112 moves to the left or right.

The first input device 120, such as a camera, captures the subject 110 at a first point in time and the second input device 130 captures the subject 110 at a second point in time.

The input devices 120, 130, and 140 capture the subject 110 at each point in time and generate input views 122, 132, and 142, respectively.

For example, the first input view 122 provides an image when a viewer views the subject 110 at a first viewpoint, and the second input view 132 provides an image when the viewer views the subject 110 at a second viewpoint Provide images when viewed.

Each of the input views 122, 132 and 142 consists of a sequence of frames. That is, the input view 122, 132, or 142 is composed of frames output in a specific number of frames per unit time, for example, 30 frames per second (FPS). A frame at a specific time point is data for generating an image at the specific time point. Thus, each of the input views 122, 132, and 142 provides a sequence of images. Each of the images corresponds to a specific moment.

A frame (or image) is composed of pixels. A pixel in a frame (or image) has a coordinate value composed of an x coordinate and a y coordinate.

Each of the pixels has a color value. The color value of a pixel can be expressed using any type (e.g., RGB or YCbCr) to represent the color.

Also, each of the pixels has a depth value. The depth value of a pixel represents the distance between the object (or background) represented by the pixel and the shooting time (i.e., the viewpoint of the view). The depth value of the pixel may be expressed using any binary format for representing the distance, e.g. integer or floating point.

The depth value of the pixels may be included in the frame. That is, a frame may have a color value and a depth value of a pixel.

In addition, the depth value of the pixels may be provided separately from the frame. The depth information of the frame (or image) represents depth values of pixels constituting the frame (or image). The depth information of the frame may be provided separately from the frame or the input view. Depth information about the image in the input view may be provided separately from the input view.

The frame 124 of the first input view 122 at the specific time t, the frame 134 of the second input view 132 and the frame 144 of the third input view 142 are shown as squares.

The 3D imaging device needs to provide the user with an image at a different point in time than the viewpoints of the input views 122, 132, Thus, the 3D imaging device is based on the input views 122, 132, and 142, and the input views 122 and 132 and 142 are output views (or target views) Lt; / RTI > The points corresponding to the above input views 122, 132 and 142 may be named reference view points.

In order for an image to be provided when the viewer looks at the subject 110 at a different point in time than the viewpoints of the input devices 120, 130 and 140 (i.e., reference viewpoints), the input devices 120, An output view at an intermediate viewpoint should be generated by view extrapolation or view interpolation using the frames provided by the input views 122, 132, and 142 generated by the input views 130 and 140.

Creating an output view means creating frames in the output view. Creation of the output view may mean providing images at the viewpoint of the output view.

The output view is an interpolated view generated by interpolation or an extrapolated view generated by extrapolation.

View interpolation means creating an output view at any virtual time point between (i.e., inside) the views of the input views 122, 132, The view interpolation can generate the output view frame (or image) by referring to the left and right input view frames (or images) adjacent to the virtual view point to be generated. An interpolated view is an output view generated by view interpolation.

View extrapolation refers to creating an output view at any point in time that is outside the viewpoints of the input views 122, 132, and 142. That is, the view extrapolation means to generate an output view at the right side of the left input view 122 or an output view at the right side view than the right input view 142. An extrapolated view is an output view generated by view extrapolation.

The view extrapolation generates an output view frame (or image) by referring to a frame (or image) of one input view 122 or 142 at the outermost position. Thus, information that can be used for view extrapolation is relatively limited relative to information that can be used for view interpolation. Therefore, since there is a relatively small amount of information that can be used, in the image generated by the view extrapolation, much image deterioration occurs as compared with the image generated by the view interpolation.

The triangles 150, 152, 160, 162, 170, 172, 180 and 182 shown represent the frames of the interpolated or extrapolated view at the time the triangle is located, respectively.

The view at a point that is further to the left than the viewpoint of the leftmost input view 122 is an extrapolated view. The view at the right side of the right input view 142 is an extrapolation view. The frames 150, 152, 180 and 182 of the extrapolated view at time t are shown.

The view at the virtual viewpoint generated between the views of the input views 122, 132 and 142 is an interpolation view. The frames 160, 162, 170 and 172 of the interpolation view at time t are shown.

An interpolated view and an extrapolated view also consist of a sequence of frames that occur over a predefined time period.

Frames in interpolation views or extrapolated views may not have depth information. That is, the frame of the interpolation view or extrapolation view may be a two-dimensional frame.

As described above, based on the N input views, M output views at the time of the N input views and at different times can be generated.

If the 3D imaging device provides the viewer with specific output views of the M output views, depending on the position of the viewer, the viewer can perceive a continuous real 3D image through the output views of the 3D imaging device. For example, when the 3D image device outputs the first output view to the left eye of the viewer and the second output view to the viewer's right eye, the user can recognize the 3D image.

Figure 2 illustrates a frame generation method of an extrapolated view according to one embodiment

The input view 210 provides a sequence of frames. The frame 212 of the input view 210 at a particular time t is shown.

Based on the input view 210, a first extrapolated view and a second extrapolated view are generated through view extrapolation at the right viewpoint than the input view 210.

The frame 220 of the first extrapolated view and the frame 230 of the second extrapolated view at the specific time t are generated using the frame 212 of the input view 210.

1 and the frame 212 of the input view 210 is composed of a background 214, a first object 216 and a second object 218 .

The frame 220 of the first extrapolated view also comprises a background 224, a first object 226, and a second object 228. In addition, the frame 230 of the second extrapolated view may include a background 234, a first object 236, and a second object 238.

The viewpoint of the first extrapolated view is located to the right of the viewpoint of the input view 210. Thus, the background 224, first object 226, and second object 228 in the frame 220 of the first extrapolated view are located further to the left than in the frame 212 of the input view 210.

How far the background 224 is to the left is dependent on the distance from the viewpoint of the input view 210 to the background 224 and the distance between the viewpoint of the input view 210 and the viewpoint of the first extrapolated view Lt; / RTI >

The frame 220 of the first extrapolated view is moved to the frame boundary hole 244 which can not be properly filled by the frame 212 of the input view 210, as the background 224 moves all the way to the left, Respectively. In some cases, the background 224 may not move. If the background 224 does not move, a frame boundary hole 244 is not created.

The objects 226 and 228 constituting the foreground move together as the background 224 moves. Objects 226 and 328 also move to the left than background 224. [

How far those objects 226 and 228 are to the left relative to the background 224 depends on the distance from the viewpoint of the input view 210 to each of the objects 226 and 228, Lt; RTI ID = 0.0 > 210 < / RTI >

Because the objects 226 and 228 move further to the left relative to the background 224, the frame 220 of the first extrapolated view may not be properly filled by the frame 212 of the input view 210, (246 and 248).

To create an extrapolated view, appropriate pixels must be extrapolated in the frame boundary hole 244 and the object boundary holes 246 and 248.

The frame 230 of the second extrapolated view also has a frame boundary hole 254 and object boundary holes 256 and 258.

The viewpoint of the second extrapolated view is further away from the viewpoint of the input view 210 as compared to the viewpoint of the first extrapolated view. The background 234, the first object 236 and the second object 238 in the frame 230 of the second extrapolated view are respectively the background 224 in the frame 220 of the first extrapolated view, And the second object 228. The second object 228 and the second object 228 are located on the left side.

The frame boundary hole 254 and the object boundary holes 256 and 258 in the frame 230 of the second extrapolation view are respectively the frame boundary hole 244 and the object boundary holes 246 and 248 in the frame 220 of the first extrapolated view, 248).

Thus, more pixels need to be extrapolated in the frame boundary hole 254 and object boundary holes 256 and 258 in the frame 230 of the second extrapolated view.

That is, the farther the viewpoint of the extrapolated view is from the outermost view, the wider the range of pixels to be extrapolated.

The frame boundary holes 244 and 254 and the object boundary holes 246, 248, 256, and 258 are collectively named.

There is a need to provide a method for recovering hole areas.

3 is a structural diagram of an image processing apparatus according to an embodiment.

The image processing apparatus 300 generates an output view through view interpolation and view extrapolation using the binocular disparity information of the reference view image and the reference view image, respectively. The image processing apparatus 300 restores the holes generated by view interpolation and view extrapolation in the output view image.

The output view image is the image at the view point of the output view. As described above, when an image at a new viewpoint is generated, a point to be newly observed at the new viewpoint appears alone in the image.

The image processing apparatus 300 may recover the holes generated in the view interpolation and the view extrapolation process.

The image processing apparatus 300 includes an image and ping unit 310, a buffer interval setting unit 320, a binocular parallax crack detecting unit 330, an adjacent image-based hall restoring unit 340, an adjacent pixel scaling-based hole restoring unit 350, And an optimal patch search-based hole reconstructing unit 360.

First, the image and the ping unit 310 will be described.

The image and ping unit 310 generates an image of the output view by a method of image warping using the binocular disparity information of the reference view image and the reference view image. (I.e., the image and the ping block 310 generate a frame of the output view by image warping using the frame of the reference view and the binocular disparity information of the frame of the reference view). The image and the ping block 310 generate the reference view image The image of the output view can be generated by image-waving. For example, when the binocular disparity information of the reference view image is not provided, the image and the ping unit 310 can generate binocular disparity information of the reference view image.

N input images (i.e., images of N reference views) are aligned with respect to the epipolar line.

A view at a virtual viewpoint (i.e., an output view) can be generated by using a weight that is proportional to the distance from the reference view to the view of the virtual viewpoint. That is, the view at the virtual viewpoint can be generated by using a weight proportional to the distance between the viewpoint of the reference view and the virtual viewpoint.

The y coordinate value of the first pixel in the output view image may be equal to the y coordinate value of the second pixel in the reference view image corresponding to the first pixel. That is, the y coordinate of the pixel may not be changed by the warping.

The x-coordinate of the first pixel in the output view (or a rendered view) image can be calculated by the following equation (1).

Here, I _{reference view} represents the image (or frame) of the reference view (or input view). , And I _{rendered view} represents the image (or frame) of the output view (or the _{rendered view} ).

x represents the coordinate value ( x coordinate value) of the second pixel in the reference view.

x ' represents the coordinate value ( x coordinate value) of the first pixel in the output view.

d is the binocular disparity value of the second pixel derived from the depth information of the image or the depth information of the pixel. In general, the binocular disparity value of a pixel is inversely proportional to the depth value of the pixel. Therefore, the description of the depth and depth information described above can be applied to binocular disparity and binocular disparity information.

α is a weight value proportional to the distance between the viewpoint of the reference view and the viewpoint of the output view.

Therefore, the above-mentioned equation (1) means the following 1) to 5).

1) The second pixel in the reference view is shifted by? X by warping. That is, when the first pixel in the second pixel and the output views in the reference view image correspond to each other, the x-coordinate value of the first pixel is a value obtained by adding an α x in the x-coordinate of the second pixel.

2) Pixels with large binocular parallax values move a lot by warping. The binocular disparity value is inversely proportional to the depth value. Therefore, a pixel having a small depth value moves more than a pixel having a large depth value.

The object is close to the viewpoint of the reference view, and the background is farther away from the viewpoint of the reference view. Therefore, among the pixels of the reference view image, the pixel representing the object moves more than the pixel representing the background.

A pixel with a depth value of infinity (or a pixel with a binocular parallax value of 0) may not be moved by warping.

3) The farther the view of the reference view and the view of the output view are, the more the pixels in the reference view move.

In addition to the warping based on Equation (1) described above, the image and ping unit 310 can generate images of the output view using various warping methods.

4) The binocular parallax value (or depth value) of the first pixel is the binocular parallax value (or depth value) of the second pixel.

5) One or more pixels in the reference view image may be moved to the same coordinates in the output view. In this case, a pixel closest to the viewpoint of the output view among the one or more pixels may be preferentially displayed.

Next, the buffer section setting section 320 will be described.

The buffer period setting unit 320 enlarges the hole generated by the image overwriting.

The hole enlargement by the buffer period setting unit 320 will be described in detail below with reference to FIG.

Next, the binocular parallax crack detector 330 will be described.

The binocular parallax crack detector 330 sets a crack in the generated output view image alone.

The hole setting by the binocular parallax crack detector 320 is described in detail below with reference to Fig.

Next, the adjacent image-based hall restoration unit 340 will be described.

The adjacent image-based hall restoration unit 340 restores the hall generated by the image watermarking. The restoration of holes by the adjacent image-based hall restoration unit 340 will be described in detail below with reference to FIG.

Next, the adjacent pixel scaling-based hole restoration unit 350 will be described.

The adjacent pixel scaling-based hole reconstructing unit 350 reconstructs holes generated by image overwriting by scaling one or more pixels adjacent to the holes.

Adjacent background scaling by the adjacent pixel scaling-based hole restoration unit 350 will be described in detail below with reference to Figs. 7 to 9.

Next, the optimal patch search-based hole reconstruction unit 360 will be described.

The optimal patch search-based hole restoration unit 360 searches for a patch most similar to the area including holes generated by the image watermarking from the background, and restores the hole by using the detected patch.

The optimal patch search and hole restoration by the optimal patch search-based hole restoration unit 360 will be described in detail below with reference to FIG.

After the hole is restored by the adjacent image-based hole restoration unit 340, a part or a part of the hole is restored by the adjacent pixel scaling-based hole restoration unit 350 and the optimal patch search-based hole restoration unit 360 .

The adjacent pixel scaling-based hole restoration unit 350 and the optimal patch search-based hole restoration unit 360 can restore the remaining holes by using the background pixels of the output view image.

Depending on the characteristics of the area adjacent to the hole, it can be determined how the hole will be restored.

For example, if the area adjacent to the hole is a texture area, the optimal patch search-based hole reconstruction unit 360 can reconstruct the hole. If the area adjacent to the hole is not a texture area, the adjacent pixel scaling-based hole restoration unit 350 can restore the hole. Since the adjacent pixel scaling-based hole restoration unit 350 scales the background pixels, the texture may be damaged by the scaling.

That is, if the area adjacent to the hole is uniform or strong edge appears in the area adjacent to the hole, the hole area can be restored by scaling the background pixel. Due to this restoration, the characteristics of the adjacent area (i.e., background) can be maintained even in the reconstructed hole area.

Also, when the area adjacent to the hole is a texture area, a region closest to the adjacent area is detected in the background area in the reference view image. The hole area is recovered using the detected area. Due to this restoration, texture components can be well maintained even in the restored hole area.

FIG. 4 illustrates restoration of a hole using images temporally adjacent to an image related to image warping according to an example.

The reference view image (or frame) used in image watermarking is an image (or frame) at a specific time t in the series of images (or frames) in the reference view image. The image at a specific time t is referred to as " image (t) " Here, the image may be a reference view image or an output view image. The frame at a specific time t is named " frame (t) " or the current frame. Further, the frame at the specific time t-1 is referred to as " frame (t-1) " The frame at a specific time t + 1 is referred to as " frame (t + 1) " or the next frame.

In Figure 4, the output view frame (t) 440 includes a hole 442. Output view frame (t) 440 is generated by warping using a reference view frame (t) 430.

In general, images temporally adjacent to the image t (e.g., image t-1, image t + 1, image t-2, and image t + 2) (Or an object) indicated by the foreground and the foreground and background that are the same as or similar to the background. Therefore, by using information of images temporally adjacent to the reference view image or the output view image, holes generated by image watermarking can be restored.

In Fig. 4, the object corresponding to the foreground moves from top to bottom. Thus, the portion 412 corresponding to the background obscured by the object in the reference view frame t 430 may be revealed in the reference view frame t-1. Also, the portion 452 of the reference view frame t + 1 450, the portion 422 of the output view frame t-1 420 and the portion 422 of the output view frame t + 462 also correspond to the background obscured by the object in the reference view frame (t) Accordingly, portions 412, 422, 452, and 462 may be used to restore hole 442. [ The adjacent image-based hall reconstruction unit 340 may reconstruct a hole of the image t of the output view based on Equation (2). Equation (2) describes a method of restoring a hole in the frame using frames temporally adjacent to the frame of the output view.

Here, f _t represents the output view frame (t) 440. That is, f _t represents a frame generated by warping at time t.

f _t ( i , j ) represents the color value of the pixel whose coordinate value is ( i , j ) among the pixels of the output view frame t 440. Here, in the output view frame (t) 440, a pixel whose coordinate value is ( i , j ) is a pixel in the hole 442. A pixel whose coordinate value is ( i , j ) is hereinafter referred to as " pixel ( i , j ) ".

f _{t- 1} represents the previous frame of the output view frame (t) 440 (i.e., the output view frame (t-1) 420). f _{t +1} represents the next frame of the output view frame (t) 440 (i.e., the output view frame (t + 1) 460).

f _{t - 1} represents the output view frame (t-1) 420. In addition, f _{t +1} indicates the output view frame (t + 1) 460.

That is, the adjacent images based hole recovery unit 340 may use a temporally adjacent in the output frame in the output view of the view frame (t) (440) restores the hole created by the image warping .α _{t- 1} is output Is a coefficient that determines whether pixel ( i , j ) of view frame (t-1) 420 is to be used to recover pixel ( i , j ) of output view frame (t) 440. alpha _{t- 1} may have a value of 0 or 1. _t- α ₁ is 0, the pixel (i, j) of an output view, frame (t-1) (420) is not used to restore the pixel (i, j) of an output view, frame (t) (440) Do not.

α _{t +1} is used to determine whether to restore the pixel (i, j) of an output view, frame (t + 1) (460) a pixel (i, j) is an output view frame (t) (440) of Coefficient. ? _{t +1} may have a value of 0 or 1. If α _{+1 t} is 0, the pixel (i, j) of an output view, frame (t + 1) (460) is not used to restore the pixel (i, j) of an output view, frame (t) (440) Do not.

If _t- α ₁ and α _{t +1} are both 1, the pixels in the output view frame (t-1) color value of the pixel (i, j) of the section 420 and outputs the view frame (t + 1) (460) the average value of the color values of ( i , j ) becomes the color value of the pixel ( i , j ) of the output view frame (t) 440. α _t- when ₁ is 0, and the α _{t +1} is 1, the output frame (t) color value of the pixel (i, j) of 440 pixels in the output view of frame (t + 1) (460) ( i , j ). _t-, and α ₁ is 1, α _{t +1} if 0, the pixels in the output view frame (t) color value of the pixel (i, j) of the 440 output the view frame (t-1) (420) ( i , j ).

Here, the pixels used for restoring the holes may be regarded as pixels included in the background. In general, the background is not changed in position by warping. For example, the portion 412 of the reference view frame t-1 410 and the portion 422 of the output view frame t-1 420 may each have the same position and color in the frame. That is, the coordinates of the first pixel included in the background in the reference view frame and the coordinates of the second pixel corresponding to the first pixel in the output view frame are the same.

Accordingly, the adjacent image-based hall reconstructing unit 340 may reconstruct the hall generated by image overwriting using temporally adjacent reference view frames in the output view frame (t) 440. [ That is, f _{t- 1} in Equation 2 may be replaced by f ' _{t- 1} representing the reference view frame t-1 410 and f _{t +1} may be replaced with reference view frame t + 1 450 ) represents a can be replaced by f _{'t +1,}

As described above, other frames within the output view or frames within the reference view that are temporally adjacent to the output view frame may be used to recover the holes in the output view frame generated by the image watermarking. In addition, it is possible to reconstruct a hole using an image represented by a frame other than the entire information included in the frame.

That is, the adjacent image-based hall reconstruction unit 340 may reconstruct a hole in an image of an output view generated by image warping using an image temporally adjacent to the image related to the image warping. Here, the image related to the image watermarking may mean one or more of the reference view image and the output view image. The temporally adjacent images may refer to temporally adjacent images in the reference view image. In addition, the temporally adjacent images may refer to images temporally adjacent to the output view image.

An output view image is generated by image overwriting. Therefore, the adjacent image-based hall reconstructing unit 340 can reconstruct the hall generated by the image watermarking using the image temporally adjacent to the image related to the generation of the output view image. Here, the image related to generation of the output view image may mean one or more of the reference view image and the output view image.

According to Equation (2), the adjacent image-based hall reconstruction unit 340 reconstructs the previous output view frame (i.e., the output view frame t-1) of the output view frame t 440 420) and the immediately following output view frame (i.e., the output view frame (t + 1) 460). However, a predetermined number of output view frames (or reference view frames) that are temporally adjacent to the output view frame t 420 (or the reference view frame t 410) May be used to restore the holes of the first electrode 420. Here, the frame can be replaced with an image.

When the output view image t is generated, the output view image (or the reference view image) after the time t (e.g., the output view image t + 1) may not be available. In this case, the adjacent image-based hall reconstruction unit 340 reconstructs the neighbor image based on the image (t-3), the image (t-2), and the image Holes can be restored.

The number of temporally adjacent images can be dynamically changed based on the storage capacity of the image processing apparatus 300, the complexity of the reference view image, and the complexity of the output view image.

In Equation (2), the color value of the pixel of the frame (t) is restored by using pixels having the same coordinate value as the pixel of the frame (t). However, the above restoration presupposes that a series of images have not moved as a whole over time. (Ie, the background of the image does not move at all or very little).

In order to recover the color values of the pixels of the output view frame (t) 440 when the images have moved entirely over time, this shift must be considered.

For example, if the output view frame (t) 440 has moved by one pixel to the left as a whole relative to the output view frame (t-1) 420 (or output view frame (t) a frame (t) of pixels in order to set the color value of the pixel (i, j) of 440, an output view, frame (t-1) 440 (or, the reference view frame (t) (410)) (i +1, j ) must be used.

That is, the adjacent image-based hall reconstructing unit 340 reconstructs the adjacent pixels of the output view image t in order to reconstruct the first pixel in the hole of the output view image t, Can be used. Alternatively, the adjacent image-based hall reconstructing unit 340 may reconstruct the original image of the pixel corresponding to the first pixel in the reference view images adjacent to the reference view image t to reconstruct the first pixel in the hole of the output view image t, Can be used.

If the temporally adjacent images of the reference view image t and the reference view image t are moved in time as a whole, the adjacent image-based hall reconstructing unit 340 reconstructs the reference view image t (Or the output view image t) and neighboring reference view images (or output view images) corresponding to the first pixel.

The adjacent image-based hall restoration unit 340 may determine the coefficient alpha based on Equation (3) below.

D (f _t (i, j)) is a binocular disparity value of the pixel (i, j) of an output view, frame (t) (420). Th is a predefined threshold. Thus, the value of α _t is 1 if f _t ( i , j ) is not a hole pixel and the binocular parallax value of f _t ( i , j ) is a predefined threshold value, otherwise it is zero.

First, it can be determined whether or not f _t ( i , j ) is a hole pixel representing a hole.

If f _t ( i , j ) is an odd pixel, the color value of f _t ( i , j ) does not exist or is an invalid value. Thus, the color value of f _t ( i , j ) can not be used to reconstruct the hole in the output view image. Therefore, the value of? _T becomes zero.

That is, the adjacent image-based hall reconstructing unit 340 reconstructs only the non-hall pixel among the pixels corresponding to the first pixel in the reference view image t adjacent to the reference view image t, Can be used to restore. In addition, the adjacent image-based hall reconstructing unit 340 reconstructs only the pixels in the output view image t that are not the hole pixels among the pixels corresponding to the first pixel in the output view image t, Can be used to restore.

Next, when f _t ( i , j ) is not an odd pixel, it can be determined whether f _t ( i , j ) is a pixel representing a foreground or a pixel representing a background.

The holes generated by the multi-view rendering appear in the output view image generated by warping a part of the background that was hidden in the foreground in the reference view image. Alternatively, the hole may comprise a part of the above background.

Therefore, f _t ( i , j ) can be used to win holes in the output view image only if f _t ( i , j ) is a background pixel representing the background.

That is, the adjacent image-based hall reconstructing unit 340 reconstructs only the background pixels among the pixels corresponding to the first pixels in the reference view images temporally adjacent to the reference view image t, Can be used to. In addition, the adjacent image-based hall restoration unit 340 restores only the background pixels among the pixels corresponding to the first pixels in the output view images temporally adjacent to the output view image t, Can be used to.

Whether f _t ( i , j ) is a foreground pixel representing the foreground or a background pixel representing the background can be determined based on the binocular disparity value of f _t ( i , j ).

In general, a pixel representing a foreground has a binocular parallax value larger than a pixel representing a background.

Thus, f _t If the binocular parallax value of the (i, j) is smaller than the reference value Th, the adjacent images based hole recovery unit 340 may be considered a (i, j) f _t a background pixel, f _t ( i , j ) can be used to reconstruct the hole.

FIG. 5 illustrates hole enlargement by setting a buffer interval according to an example.

An output view image 510 in which an output view image 510 and buffer regions 522 and 524 are displayed in FIG. 5 is shown.

The binocular disparity value of a pixel used in multi-view rendering can be obtained by converting the physical depth value of the pixel. Further, the binocular disparity value of the pixel can be determined by an estimation using a reference image.

The binocular disparity value of the pixel may have a wrong value due to a matching error or the like (especially if it is obtained by estimation).

If the binocular disparity value of the pixel is incorrect, the boundary between the object and the background of the output view image 510 may be inconsistent with the boundary between the object and the background of the binocular parallax image.

The pixels on the left side of the hole area 512 of the output view image 510 have a color value indicating the background, despite the pixels representing the foreground due to the above-mentioned inconsistency.

In consideration of such a problem, in particular, when the estimated binocular disparity value is used, the area adjacent to the hole needs to be set as a buffer area.

Therefore, the buffer section setting section 310 enlarges the holes by considering the buffer areas 522 and 524 adjacent to the holes as holes.

The buffer section setting section 310 may set, as the buffer area, pixels having a distance from the hole (or the outermost point of the hole) smaller than a predefined threshold value.

When the color value of the pixel in the buffer area is restored, the following equation (4) can be used.

( I , j ) of the frame t (i.e., the pixel to which the color value is to be restored), as well as the color value of the pixel of the previous frame of the frame t and the color of the pixel of the subsequent frame, Is also used for restoration. Pixels ( i , j ) in the buffer area already have color values, unlike pixels that were originally holes. Therefore, the color value of the pixel ( i , j ) itself in the buffer area can be used for restoration of the pixel ( i , j ) in the buffer area considered as the hole.

That is, when restoring the first pixel in the buffer region, the adjacent image-based hole restoration unit 340 may restore the first pixel based on the color value of the first pixel.

Fig. 6 illustrates generation of cracks and hole setting by detecting binocular parallax cracks according to an example.

6, a binocular parallax image 620 of a reference view image 610 and a reference view image 610 is shown.

As shown, the first portion 622 and the second portion 624 of the binocular parallax image 620 have different binocular disparity values from each other. In general, since the first portion 622 and the second portion 624 represent the same object, they must have the same or similar binocular disparity value. However, particularly when the binocular disparity value is estimated, the first part 622 and the second part 624 may have different binocular disparity values from each other.

Since the first portion 622 and the second portion 624 have different binocular disparity values, distances moved by warping are different from each other.

Because the first portion 622 and the second portion 624 move a different distance from each other, within the output view image 630 generated by the warping, the portion corresponding to the first portion 622 and the portion corresponding to the second portion 622, A crack 632 may occur between the portions corresponding to the protrusion 624.

In the portion where cracks 632 are generated, a background rather than a foreground can be displayed. That is, the background may be displayed instead of the first portion 622 and the second portion 624.

That is, the cracks 632 refer to the portions of the objects (or the separated portions of the object) in which the background is displayed because of the different binocular parallax values. That is, when the binocular disparity value is allocated to parts of a specific object, a crack occurs inside the object when the object is warped.

Therefore, when the reference view image 610 is warped by the image and ping unit 310, the output view image 630 generated by the warping has a crack.

In a portion where cracks 632 are generated, a background rather than a foreground is displayed. That is, in the portion where the crack 632 is generated, the background color value is warped.

Therefore, image quality deterioration occurs in the output view image due to the crack 632.

If the crack 632 is set to be the hole, the hole restoring method can be applied to the portion where the crack 632 appears. Therefore, deterioration in image quality due to the crack 632 can be reduced.

The binocular parallax crack detector 330 detects a crack in the output view image. The binocular parallax crack detector 330 sets a portion where a crack is generated to be a hole.

The crack can be detected based on Equation (5) below.

Here, D _{i , j} is the binocular parallax value of the pixel ( i , j ) 642 in the output view image 630.

Pixel ( i , j ) 642 is a pixel for which it is checked whether it is cracked.

D _{i + m , j + n} are the binocular parallax values of the pixel ( i , j ) and the neighboring pixel ( i + m , j + n ).

Th is a predefined threshold.

That is, the binocular parallax crack detector 330 can detect cracks in a pixel in the output view in which the sum of the differences of the binocular parallax values with adjacent pixels 644 is larger than a predefined threshold value.

Cracks occur because the background pixel is warped to the area where the foreground appears. Therefore, the difference between the binocular disparity value of the background pixel located at the crack point and the binocular disparity value of the foreground pixel adjacent to the background pixel is large.

Thus, as in Equation 5, a crack can be detected based on the difference in binocular disparity values with adjacent pixels 644. In addition, by allocating the detected cracks alone, the image deterioration can be compensated.

The adjacent pixels 544 shown in Fig. 6 are exemplary. Pixels at a certain distance from the first pixel in the output view may be adjacent pixels of the first pixel.

Figure 7 illustrates adjacent pixel scaling according to an example.

The adjacent pixel scaling-based hole restoration unit 350 uses one or more pixels adjacent to the hole to restore the hole of the image 710. [

The adjacent pixel scaling-based hole restoration unit 350 may scale pixels in a horizontal direction.

A horizontal line 712 that is the object of the hole restoration is shown.

The adjacent pixel scaling-based hole restoration unit 350 detects holes in the horizontal line 712 and detects the number of consecutive hole pixels 730.

The adjacent pixel scaling-based hole restoration unit 350 may perform a scan in the horizontal direction to detect holes.

The adjacent pixel scaling-based hole restoration unit 350 selects as many pixels 740 as the number of consecutive hole pixels 730. [ The selected pixels are generally non-hole pixels.

Selected pixels 740 are pixels adjacent to consecutive hall pixels 730.

The selected pixels 740 may be pixels on the same horizontal line as the consecutive hall pixels 730.

In FIG. 7, the selected pixels 740 are the pixels to the right of the consecutive hall pixels 730. However, the pixels to the left of the consecutive hall pixels 730 may also be the selected pixels 740. Also, the selected pixels 740 may be pixels on the right side of successive hole pixels 730 and pixels on the left side.

The adjacent pixel scaling-based hole restoration unit 350 can restore the hole by scaling the selected pixels 740 in the direction toward the hole pixels 730. [

For example, the color values of the first and second hall pixels are generated using the color values of the first selected pixel. Thus, the first hall pixel and the second hall pixel are restored by the first selected pixel.

The scaling of the selected pixels 740 may be such that the area represented by the selected pixels 740 is doubled and the stretched area replaces the area represented by the hole pixels 730 and the selected pixels 740 have.

At this time, the number of the hole pixels 730 and the number of the selected pixels 740 are equal to each other. Thus, all of the holes can be reconstructed by scaling the selected pixels 740 with two pixels each. Also, the selected pixels 740 can all be scaled uniformly.

Figure 8 illustrates adjacent pixel scaling using a background pixel according to an example.

The adjacent pixel scaling-based hole restoration unit 350 selects pixels 820 that are not the same number of holes as the number of consecutive hall pixels 810.

Each of the non-hole pixels 820 has a binocular parallax value.

The neighboring pixel scaling-based hole reconstruction unit 350 classifies each non-hole pixel 820 into foreground pixels or background pixels according to binocular parallax values.

For example, the neighboring pixel scaling-based hole restoration unit 350 may classify pixels of the non-hole pixels 820 whose foreground parallax values are larger than a predefined threshold value into foreground pixels, Pixels below the threshold can be classified as background pixels.

8, the previous three pixels 830 have been classified as background pixels, and the last pixel 840 has been classified as foreground pixels.

The adjacent pixel scaling-based hole restoration unit 350 can restore the hole by scaling the background pixels 830 in the direction toward the hole pixels 730. [

At this time, the number of background pixels 830 may be smaller than the number of the hole pixels 730. Thus, all or a portion of the background pixels 830 may be scaled to two or more pixels. Also, background pixels 840 may be non-uniformly scaled.

Foreground pixel 840 is excluded from scaling. Thus, the distortion of the foreground image due to scaling can be prevented.

The scaling methods described above extend pixels 740 or 830 only in the horizontal direction, which is the scan direction. Thus, such scaling methods can be readily implemented. Further, when such a scaling method is used, deterioration of image quality is not largely recognized even if a small-sized hole is restored.

Figure 9 illustrates scaling in a direction perpendicular to the gradient of the background according to an example.

The output view image 910 includes a hole 912.

If there is a particular shape 916 of the background 914, the shape 916 may be deformed if the horizontal scaling described above is used. Shape 916 is a collection of one or more background pixels.

This shape 916 thus needs to be scaled in a direction 926 perpendicular to the direction 924 of the edge 918 of the shape 916 contained in the hole 912. [

The adjacent pixel scaling-based hole restoration unit 350 selects the background pixel 922 adjacent to the hole 912.

The neighboring pixel scaling-based hole restoration unit 350 calculates a gradient of the edge 918 including the background pixel 922. [ The gradient of the edge 918, which includes the background pixel 922, is a gradient of the background pixel 922.

The adjacent pixel scaling-based hole restoration unit 350 detects the number of consecutive hole pixels in the vertical direction of the gradient of the edge 918.

The adjacent pixel scaling-based hole restoration unit 350 scales the background pixels in the vertical direction of the gradient of the edge 918 so as to recover the detected consecutive hall pixels.

In other words, the adjacent pixel scaling-based hole restoration unit 350 is configured to detect the background pixels (or background pixels) including the adjacent background pixels 922 in a direction perpendicular to the gradients of the background pixels 922 adjacent to the holes 912 The hole can be restored by scaling it. The background pixels may represent a shape 916 that includes a background pixel 922.

At this time, the adjacent pixel scaling-based hole restoration unit 350 can detect the number of the hole pixels that are continuous in the direction perpendicular to the gradient of the background pixel 922, and can detect the background to be used for scaling based on the detected number of hole pixels Determine the pixels, and determine how far the background pixels are to be scaled.

By using the above-described scaling in the direction perpendicular to the gradient of the background pixel 922, the adjacent pixel scaling-based hole restoration unit 350 can restore the hole area while maintaining the orientation of the background pixels 916 .

The scaling method using the gradient of the background results in a more natural result than simply scaling in the horizontal direction since it scales the background pixels in the strong corners direction in the background. However, the scaling method using the background gradient performs relatively complicated operations.

Figure 10 illustrates the reconstruction of a hole based on optimal patch search with respect to work.

The output view image 1010 includes a hole.

If the background adjacent to the hole is a texture area, simply scaling the background pixels does not allow the hole to be reconstructed correctly.

When the background adjacent to the hole is a texture, in the entire background area, a patch most similar to the background adjacent to the hole can be detected. The hole can be recovered using the detected patch. That is, if a texture that is the same as or very similar to the texture of the background adjacent to the hole is found in another background area, a portion adjacent to the other background area can be used to recover the hole.

First, consider how to determine the point to restore using the patch.

A method of preferentially processing a hole adjacent to the background may be used. By using the method of sequentially restoring the area of the hole from the part adjacent to the background, mixing of the foreground pixels in the hole restoration process can be structurally restricted.

The optimal patch search-based hole restoration unit 360 detects the hole pixels in the raster scan direction.

The detected initial hole point 1012 is adjacent to the foreground. Therefore, it is a point unsuitable for priority restoration.

The optimal patch search-based hole reconstruction unit 360 detects the consecutive hole pixels from the initial hole point 1012 in the scan direction.

The optimal patch search-based hole reconstruction unit 360 sets the end of successive hole pixels as the outermost hole point 1014. The outermost hole point 1014 was adjacent to the background. Therefore, a patch that can recover the hole area around the outermost hole point 1014 can be retrieved by using the color value and binocular disparity value of the pixel that is adjacent to the outermost hole point 1014 and is not a hole .

The optimal patch search-based hole restoration unit 360 performs optimal patch search based hole restoration on the set outermost hole point 1014. In the following, optimal patch search based hole reconstruction is described.

The optimal patch search-based hole restoration unit 360 sets the area adjacent to the outermost hole point 1014 as the window area 1016. [ Window region 1016 may be N x N pixels.

The window region 1016 includes a hole region 1020 and a background region 1018. Background area 1018 is used to search for patches. The hole area 1020 is the area to be recovered by using the detected patch.

The optimal patch search-based hole restoration unit 360 detects an optimal patch corresponding to the window area 1016 in the entire background area, detects the outermost hole point 1014 and the hole area 1020 set using the detected patch, .

The optimal patch means an area having the highest degree of similarity to the window area 1016 of the entire background area.

The patch has the same size as the window area 1016. The patch also comprises a portion corresponding to the hole region 1020 and a portion corresponding to the background region 1018. [

The similarity between a particular portion of the background region and the window region 1016 is calculated to select the patch. The specific portion has the same size as the window region 1016 and is composed of a portion corresponding to the hole region 1020 and a portion corresponding to the background region 1018. [

When the degree of similarity between the specific portion and the window region 1016 is calculated, the hole region 1020 is excluded from the calculation of the degree of similarity, and only the background region 1018 is used for calculating the similarity.

For example, when a part corresponding to the background area 1018 of the first specific part has the same color value and binocular parallax value as the background area 1018 of the window area 1016, A portion corresponding to the background region 1018 and the background region 1018) are considered to be the same. Thus, the first specific portion can be selected as the optimum patch.

The optimal patch search-based hole restoration unit 360 may use a mean of absolute difference (MAD) method for calculating the similarity. The MAD method can use color values and binocular disparity values.

That is, when the MAD between the second specific portion and the window region 1016 among the plurality of specific portions is the minimum value, the optimal patch search-based hole reconstruction unit 360 can select the second specific portion as the optimal patch.

When the optimal patch is determined, the optimal patch search-based hole reconstruction unit 360 reconstructs the hole region 1020 using a portion corresponding to the hole region 1020 of the determined patch.

Fig. 11 illustrates restoration of holes using overlapping patches according to an example.

 The optimal patch search-based hole restoration unit 360 may use two or more patches to restore a specific hole pixel (or hole area).

That is, the optimal patch search-based hole reconstruction unit 360 can select two or more window regions 1112 and 1114 based on two or more outermost hole points different from each other.

The optimal patch search-based hole restoration unit 360 can search for optimal patches for each of two or more window areas 1112 and 1114, and can restore holes using two or more detected patches.

When two or more window regions 1112 and 1114 overlap each other, the overlapped hole region can be restored by two or more patches.

In this case, the optimal patch search-based hole restoration unit 360 may restore the overlapped hole area using the average value of the color values of the two or more patches and the average value of the binocular parallax values.

That is, the optimal patch search-based hole restoration unit 360 may restore the hole using the average value of the color values of the two or more patches and the average value of the binocular disparity values.

12 is a flowchart of an image processing method according to an embodiment.

In the image warping step S1210, an image of the output view is generated by image warping using the image of the reference view and the binocular disparity information of the image of the reference view.

In the buffer interval setting step (S1220), by setting the buffer interval, the hole generated in the output view is enlarged.

In the binocular parallax crack detection step (S1230), a crack in the image of the output view is set alone.

In the adjacent image-based hall restoring step (S1240), holes generated in the output view are restored by using images temporally adjacent to the reference view image.

In the adjacent pixel scaling-based hole reconstruction step (S1250), the hole is reconstructed by scaling one or more pixels adjacent to the hole.

In the optimal patch search-based hole restoration step (S1260), the patch most similar to the area including the hole is searched from the background, and the hole is restored by using the searched patch.

The technical contents according to the embodiment of the present invention described above with reference to Figs. 1 to 11 can be applied to the embodiment of the image processing method of Fig. Therefore, a more detailed description will be omitted below.

The method according to an embodiment of the present invention can be implemented in the form of a program command which can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic recording media such as magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic recording media such as floppy disks, Optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa. The one or more software modules described herein may be executed by a processor unique to the unit or by a processor common to one or more modules. The described methods may be executed by a particular machine, such as a general purpose computer, a general purpose processor, or the image processing apparatus described herein.

13 shows a display device including an image processing apparatus according to an example.

13, the multi-view display device 1300 may include a control unit 1310 and an image processing device 1305. [

The multi-view display device 1300 may take the form of a 3D display for displaying 3D images and may employ a multi-view scheme to three or more different viewpoints. Optionally, the multi-view display device 1300 may take the form of a stereoscopic display outputting the left and right images.

The control unit 1301 can generate one or more control signals for controlling the multi-view display device 1300 and generate one or more signals to be displayed by the multi-view display device 1300. [ The control unit 1301 may include one or more processors.

The image processing apparatus 1305 may be used to generate a multi-view image for the multi-view display device 1300 and may include, for example, an image and a ping section, a buffer section setting section, a binocular parallax crack detecting section, An adjacent pixel scaling-based hole reconstruction unit, and an optimal patch search-based hole reconstruction unit. The above units are not shown in Fig. However, each of these units corresponds to, for example, similarly named units described herein, with reference to FIG. Therefore, it is not further described in this example.

The image processing device 1305 may be installed internally in the multi-view display device 1300 and may be attached to the multi-view display device 1300 and may be implemented (e.g., embody. Regardless of the physical configuration of the image processing apparatus 1305, the image processing apparatus 1305 may have the capabilities described above with reference to FIGS. The image processing apparatus 1305 may include one or more internal processors. Alternatively, the one or more processors may be included within the multi-view display device 1300, such as one or more processors of the controller 1301.

The 3D imaging apparatus and method described above can utilitize various video formats. The above video format may include formats such as H.264 / MPEG-4 AVC, High Efficiency Video Coding (HEVC), Dirac video compression format, and VC-1, Lt; / RTI >

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

300: image processing device
310: image and ping
340: adjacent image-based hole reconstruction unit
350: adjacent pixel scaling-based hole reconstruction unit
360: optimal patch search-based hole reconstruction unit

Claims (41)

A processor for controlling one or more processor-executable units;
An image and a ping section for generating an image of an output view by image warping using the image of the reference view and the binocular disparity information of the image of the reference view;
A buffer section setting section for enlarging a hole in the output view by considering a buffer area adjacent to a hole in the output view generated by the image watermark as a hole; And
An adjacent image-based hole restoration unit for restoring the enlarged hole using images temporally adjacent to the image related to the image warping;
Lt; / RTI >
Wherein the adjacent image-based hall restoration unit restores the first pixel based on a color value of the first pixel when the first pixel is a pixel in the buffer area. The method according to claim 1,
Wherein the temporally adjacent images are temporally adjacent to an image of the reference view. The method according to claim 1,
Wherein the temporally adjacent images are temporally adjacent to the image of the output view. The method according to claim 1,
Wherein the adjacent image-based hall reconstruction unit uses a color value of pixels corresponding to the first pixel in the temporally adjacent images to recover a first pixel in the hole. 5. The method of claim 4,
Wherein the neighboring-image-based hall reconstructing unit selects the pixels corresponding to the first pixel in the temporally adjacent images based on the movement, when the reference view and temporally adjacent images are entirely moved in time Image processing apparatus. 5. The method of claim 4,
Wherein the adjacent image-based hall reconstruction unit uses only pixels of the pixels corresponding to the first pixel that are not a hall pixel to reconstruct the hall. 5. The method of claim 4,
Wherein the adjacent image-based hall restoration unit uses only a background pixel among the pixels corresponding to the first pixel to restore the hall. delete delete The method according to claim 1,
A binocular parallax crack detector for setting a crack in an image of the output view as an image,
Further comprising: 11. The method of claim 10,
Wherein the binocular parallax crack detecting unit detects, as a crack, a pixel having a sum total of differences of binocular parallax values with adjacent pixels of pixels in the output view larger than a predefined threshold value. delete delete delete delete delete delete delete An image watermarking step of generating an image of an output view by image-waving an image of the reference view based on binocular disparity information of the image of the reference view by the processor;
A buffer area setting step of enlarging a hole in the output view by considering a buffer area adjacent to a hole in the output view generated by the image watermark as a hole; And
An adjacent image-based hall restoring step of restoring the enlarged hall using images temporally adjacent to the image related to the image warping;
Lt; / RTI >
Wherein the neighboring-image-
And restoring the first pixel based on the color value of the first pixel when the first pixel is a pixel in the buffer area
And an image processing method. 20. The method of claim 19,
Wherein the temporally adjacent images are temporally adjacent to an image of the reference view. 20. The method of claim 19,
Wherein the temporally adjacent images are temporally adjacent to an image of the output view. delete 20. The method of claim 19,
An adjacent pixel scaling-based hole restoration step of restoring the hole by scaling one or more pixels adjacent to the hole; And
Searching for a patch most similar to an area including the hole from the background and restoring the hole by using the retrieved patch,
Further comprising the steps of: A computer-readable recording medium embodying a program for carrying out the method according to any one of claims 19 to 21 and 23. A processor for controlling one or more processor-executable units;
An image generating unit generating an image of an output view based on binocular disparity information of a reference view and an image of the reference view;
A buffer zone setting unit for enlarging a hole in the output view by considering a buffer zone adjacent to the hole in the output view as a hole, the hole in the output view being generated as a result of generation of the output view; And
Based holographic reconstruction unit for reconstructing the enlarged hole in the output view using background information of images temporally adjacent to the image related to the generation,
Lt; / RTI >
Wherein the adjacent image-based hall reconstruction unit restores the first pixel based on the color value of the first pixel when the first pixel is a pixel in the buffer area. 26. The method of claim 25,
Wherein the temporally adjacent images are images temporally adjacent to the image of the reference view. 26. The method of claim 25,
Wherein the temporally adjacent images are temporally adjacent to the image of the output view. 26. The method of claim 25,
Wherein the image generating unit generates an image of the output view by interpolating or extrapolating data from the reference view. delete delete delete delete delete delete delete delete delete delete A display device including an image processing apparatus,
An image and a ping block for generating an image of an output view based on a binarized parallax information of an image of a reference view and an image of the reference view;
A buffer region setting unit that enlarges a hole in the output view by considering a buffer region adjacent to a hole in the output view generated by the image watermark as hole, the hole being generated as a result of generation of the output view;
An adjacent image-based hall reconstruction unit for reconstructing the enlarged hole in the generated output view using background information of images temporally adjacent to the generation-related image; And
And a control unit for generating a signal to be displayed by the display device based on the image of the generated output view having the hole reconstructed by the adjacent image-
Lt; / RTI >
Wherein the adjacent image-based hall reconstruction unit restores the first pixel based on the color value of the first pixel when the first pixel is a pixel in the buffer area. 40. The method of claim 39,
Wherein the temporally adjacent images are images temporally adjacent to the image of the reference view. 40. The method of claim 39,
Wherein the temporally adjacent images are images temporally adjacent to the image of the output view.

Images