JP2014515197A - Multi-view rendering apparatus and method using background pixel expansion and background-first patch matching - Google Patents

Abstract

  An apparatus and method is provided for restoring holes that occur in multi-view rendering. Holes in the output view are restored by using temporally adjacent images.

Description

  The following embodiments relate to a multi-view rendering apparatus and method.

  In order to generate a 3D (3-Dimensional) video, a multi-view 3D video having a wide viewing angle must be continuously expressed.

  However, it is not easy to shoot multi-view videos and transmit the taken multi-view videos in real time. This is because it is not easy to store and transmit captured data, including the physical limitations of the imaging system.

Accordingly, the 3D image generation apparatus may generate the 3D image using only a small number (for example, 2 to 3) input views (or reference views), and the 3D display apparatus that reproduces the generated 3D image is input. Multiple output views can be generated by interpolating or extrapolating views.
Japanese Patent No. 2009-533778 Korean Registered Patent No.0731979 Korean registered patent No. 0960294 Korean Registered Patent No. 0924716

  One embodiment of the present invention provides an apparatus and method for restoring holes in an output view video generated by video warping.

  According to one embodiment, an output view video is generated by video warping using a processor that controls executable units of one or more processors and binocular disparity information of the reference view video and the reference view video. An image processing apparatus comprising: an image warping unit; and an adjacent image base hole restoration unit that restores a hole generated by the image warping using one or more images temporally adjacent to the image related to the image warping. Provided.

  The one or more videos that are temporally adjacent may be videos that are temporally adjacent to the video of the reference view. The one or more videos that are temporally adjacent may be video that is temporally adjacent to the video of the output view.

  The adjacent image base hole restoration unit may use a color value of a pixel corresponding to the pixel in the hole in the temporally adjacent image to restore the pixel in the hole. When the reference view image and the temporally adjacent image move as a whole according to time, the adjacent image base hole restoration unit may include the hole in the temporally adjacent image based on the movement. A pixel corresponding to the pixel within may be selected.

  The adjacent image base hole restoration unit may restore the hole using a pixel obtained by removing one or more hole pixels corresponding to the pixel in the hole. The adjacent image base hole restoration unit may use a background pixel among pixels corresponding to the pixel in the hole to restore the hole.

  The video processing apparatus may further include a buffer interval setting unit that expands a hole generated by video warping.

  The buffer section setting unit enlarges the hole by regarding a buffer region adjacent to the hole as the hole, and the adjacent video base hole restoration unit is configured such that the pixel in the hole is a pixel in the buffer region. In some cases, the pixel in the hole may be restored based on the color value of the pixel in the hole.

  The video processing apparatus may further include a binocular parallax crack detection unit that sets a crack in the hole in the video of the output view. The binocular parallax crack detection unit may detect a pixel in which a sum of differences between binocular parallax values from adjacent pixels is larger than a predetermined threshold as a crack.

  According to another embodiment, an output view video is generated by video warping using a processor that controls one or more processor-executable units and a reference view video and binocular disparity information of the reference view video. An image processing apparatus is provided that includes an image warping unit and an adjacent pixel scaling base hole restoration unit that restores a hole generated by the image warping by scaling one or more pixels adjacent to the hole.

  The neighboring pixel scaling base hole restoration unit may scale one or more background pixels of the one or more pixels.

  The hole and the one or more pixels may be on the same horizontal line.

  The adjacent pixel scaling base hole restoration unit scales the one or more pixels in a direction perpendicular to a gradient of a background pixel adjacent to the hole, and the background pixel is one of the one or more pixels. May be.

  According to another embodiment, an output view image is generated by a processor controlling an executable unit of one or more processors and image warping using a reference view image and binocular disparity information of the reference view image. An image warping unit, and an optimal patch search base hole restoring unit that searches a background for a patch that is most similar to a region including a hole generated by the image warping, and restores the hole by using the searched patch; Is provided.

  The region including the hole is configured as a hole region and a background region, and the optimum patch search base hole restoration unit restores the hole region using a portion corresponding to the hole region in the searched patches. Also good. The optimal patch search base hole restoration unit searches for a first patch for a first region, searches for a second patch for a second region, and uses an average value of the first patch and the second patch. A hole portion overlapping in the first region and the second region may be restored.

  Further, according to a further embodiment, a video warping step of generating an output view video by image warping the reference view video based on binocular disparity information of the reference view video by a processor, and the video warping There is provided a video processing method including a neighboring video-based hole restoration step of restoring holes generated in the output view using one or more videos temporally adjacent to related videos.

  The image processing method may further include a setting step of a buffer section for enlarging the hole. The image processing method may further include a binocular parallax crack detection step of setting a crack in the hole in the video of the output view. The image processing method may further include an optimum patch search base hole restoration step of searching for a patch most similar to an area including the hole from the background and restoring the hole by using the searched patch.

  According to a further embodiment, a processor that controls executable units of one or more processors, a video generator that generates video of an output view based on video of at least one reference view, the generated A binocular parallax crack detection unit for detecting a crack in a predefined object of an output view image, wherein the predefined object is different from each other assigned to different parts of the predefined object. A binocular parallax crack detection unit that occurs in the predefined object due to the generation of the output view video based on the at least one reference view video, with an eye parallax value; , Reassign the crack as a hole and one or more temporally adjacent to the image associated with the generation Multi-view generation apparatus and a multi-view generation unit for restoring the holes that exist in the current frame of image of the generated output views by using the background information of the frame is provided.

  The one or more frames that are temporally adjacent may be frames that are temporally adjacent to the video of the reference view. The one or more frames that are temporally adjacent may be frames that are temporally adjacent to the video of the output view.

  According to a further embodiment, the processor generates an output view image based on the at least one reference view image, and includes a crack in a predefined object of the generated output view image. Detecting, wherein the predefined object has different binocular disparity values assigned to different parts of the predefined object, and the crack is based on the video of the at least one reference view. Occurring in the predefined object due to generating the video of the output view; and reallocating the crack as a hole and one or more adjacent in time to the video associated with the generation Image of the generated output view by using background information of frames Multiview generation method comprising the steps of restoring the holes that exist in the current frame is provided.

  According to a further embodiment of the present invention, in a display device comprising a video processing device, a video warping unit that generates an output view video based on a reference view video and binocular disparity information of the reference view video; An adjacent video-based hole restoration unit that restores a hole using one or more videos temporally adjacent to the video related to the generation, wherein the holes are generated by generating the output view. A video comprising a video base hole restoration unit and a control unit that generates a signal to be displayed by the display device based on the generated video of the output view having the hole restored by the adjacent video base hole restoration unit. A display device comprising a processing apparatus is provided.

  According to the present invention, holes in an output view image generated by image warping can be restored.

A view generation method based on three input views according to an example will be described. 6 illustrates a method of generating a frame for extrapolation views according to an embodiment. 1 is a structural diagram of a video processing apparatus according to an embodiment. The restoration of holes using temporally adjacent images according to an example will be described. The hole expansion by setting the buffer section according to an example will be described. A description will be given of hole setting based on generation of cracks and detection of binocular parallax cracks according to an example. An example of neighboring pixel scaling will be described. An example of neighboring pixel scaling using background pixels according to an example is described. An example of scaling in the direction perpendicular to the background gradient will be described. A hole restoration based on an optimal patch search according to an example will be described. The restoration of holes using overlapping patches according to an example will be described. 5 is a flowchart of a video processing method according to an embodiment. 1 shows a display device including a video processing apparatus according to an embodiment.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. The same reference numerals provided in each drawing denote the same members.

  FIG. 1 illustrates a view generation method based on three input views according to an example.

  A subject 110 to be photographed includes a foreground and a background 112. The foreground includes a first object 114 and a second object 116. The relative positions of the first object 114 and the second object 116 with respect to the background 112 move to the left or right depending on the viewpoint of the observer.

  For example, the first input device 120 such as a camera is the first viewpoint, the second input device 130 is the second viewpoint, and the third input device 140 is the third viewpoint, and each shoots the subject 110. The input devices 120, 130, and 140 capture the subject 110 from their respective viewpoints and generate respective input views 122, 132, and 142.

  For example, the first input view 122 provides a video when the viewer views the subject 110 from the first viewpoint, and the second input view 132 provides a video when the viewer views the subject 110 from the second viewpoint. The third input view 134 provides an image when the viewer views the subject 110 from the third viewpoint.

  Each of the input views 122, 132, and 142 is composed of a sequence of frames. That is, the input view 122, 132, or 142 is composed of frames output from a specific number per unit time, for example, 30 FPS (Frames Per Second). The specific viewpoint frame is data for generating the video of the specific viewpoint. Accordingly, each of the input views 122, 132, and 142 provides a sequence of videos. Each video corresponds to a specific moment.

  A frame (or video) is composed of pixels. Pixels in the frame (or video) have coordinate values composed of x and y coordinates. Each pixel has a color value. Pixel color values may be represented using any format for representing color (eg, RGB or YCbCr). Each pixel also has a depth value. The pixel depth value represents the distance between the object (or background) represented by the pixel and the shooting viewpoint (ie, the viewpoint of the view). Pixel depth values may be represented using any binary format for representing distances, such as integers or floating point numbers, for example.

  Pixel depth values may be included in the frame. That is, a frame can have pixel color and depth values. Also, the pixel depth value may be provided separately from the frame. The depth information of the frame (or video) represents the depth value of the pixels constituting the frame (or video). Frame depth information may be provided separately from the frame or input view. Depth information for the video in the input view may be provided separately from the input view.

  The frame 124 of the first input view 122, the frame 134 of the second input view 132, and the frame 144 of the third input view 142 at a particular vision t are shown as squares.

  The 3D video device needs to provide the user with video from other viewpoints of the input views 122, 132, and 142. Accordingly, the 3D video apparatus generates an output view (or a target view) at a viewpoint different from the input views 122, 132, and 142 based on the input views 122, 132, and 142. The viewpoints corresponding to the input views 122, 132 and 142 are named reference view viewpoints.

  In order to provide an image when the viewer views the subject 110 from a different viewpoint that is not a reference view viewpoint, the image is provided by the input views 122, 132, and 142 generated by the input devices 120, 130, and 140. The output view at the intermediate viewpoint must be generated by view extrapolation using view frames or view interpolation.

  Generating an output view means generating an output view frame. Output view generation means providing video at the viewpoint of the output view. The output view is an interpolation view generated by interpolation or an extrapolation view generated by extrapolation. View interpolation means generating an output view at any virtual viewpoint that is between (ie, internal) the viewpoints of the input views 122, 132, and 142. View interpolation may generate an output view frame (or video) with reference to left and right input view frames (or videos) adjacent to the virtual view viewpoint to be generated. An interpolated view is an output view generated by view interpolation.

  View extrapolation means generating an output view at an arbitrary viewpoint that is in the suburbs of the viewpoints of the input views 122, 132, and 142. View extrapolation means generating an output view at a left viewpoint from the viewpoint of the leftmost input view 122 or an output view at a right viewpoint from the viewpoint of the rightmost input view 142. An extrapolated view is an output view generated by view extrapolation.

  View extrapolation generates an output view frame (or video) with reference to the frame (or video) of one input view 122 or 142 in the outermost contour. Therefore, the information used for view extrapolation is relatively limited compared to the information used for view interpolation. Therefore, since relatively little information is used, video quality generated by view extrapolation is more degraded than video generated by view interpolation.

  The illustrated triangles 150, 152, 160, 162, 170, 172, 180, and 182 each represent an interpolated view frame or an extrapolated view frame.

  The view of the left viewpoint from the viewpoint of the leftmost input view 122 is an extrapolated view. Further, the view of the right viewpoint from the viewpoint of the rightmost input view 142 is an extrapolated view. Frames 150, 152, 180, and 182 are extrapolated view frames of vision t.

  The view at the virtual viewpoint generated between the viewpoints of the input views 122, 132 and 142 is an interpolation view. Frames 160, 162, 170, and 172 of an interpolated view of visual t are shown.

  The interpolated view and extrapolated view are also composed of a sequence of frames generated over a predefined time period. An interpolated view or extrapolated view frame may not have depth information. That is, the frame of the interpolation view or the extrapolation view may be a two-dimensional frame.

  As described above, viewpoints of the N input views and M output views at different viewpoints may be generated.

  If the 3D video device provides a specific output view of the M output views to the viewer according to the position of the viewer, the viewer can continuously perform the actual 3D via the output view of the 3D video device. Can recognize video. For example, if the 3D video device outputs the first output view to the viewer's left eye and the second output view to the viewer's right eye, the user can recognize the 3D video.

FIG. 2 illustrates an extrapolated view frame generation method according to one embodiment. A sequence of frames is provided via an input view 210. A frame 212 of the input view 210 at a particular vision t is illustrated. Based on the input view 210, a first extrapolated view and a second extrapolated view are generated by view extrapolation at the right viewpoint from the input view 210.

  A frame 220 of the first extrapolated view and a frame 230 of the second extrapolated view at the particular vision t are generated using the frame 212 of the input view 210. The input view 210 is a photograph of the subject 210 shown in FIG. 1, and the frame 212 of the input view 210 includes a background 214, a first object 216, and a second object 218.

  The frame 220 of the first extrapolated view is also composed of a background 224, a first object 226, and a second object 228. Further, the frame 230 of the second extrapolation view may include a background 234, a first object 236, and a second object 238. The viewpoint of the first extrapolation view is located on the right side of the viewpoint of the input view 210. Accordingly, the background 224, the first object 226, and the second object 228 in the frame 220 of the first extrapolation view are located further to the left from the frame 212 of the input view 210.

  The left side of the background 224 depends on the distance from the viewpoint of the input view 210 to the background 224, and depends on the distance between the viewpoint of the input view 210 and the viewpoint of the first extrapolation view.

  As described above, because all of the background 224 moves to the left side, the first extrapolated view frame 220 has a frame boundary hole 244 that is not properly filled by the frame 212 of the input view 210. In some cases, the background 224 may not be movable. If the background 224 does not move, the frame boundary hole 244 is not generated.

  Objects 226 and 228 constituting the foreground are moved together when the background 224 moves. Further, the objects 226 and 328 move to the left side of the background 224.

  How far the objects 226 and 228 are located in the background 224 depends on the distance from the viewpoint of the input view 210 to the objects 226 and 228, respectively, and the viewpoint of the input view 210 and the first extrapolated view. Depends on the distance between the viewpoints. Because the objects 226 and 228 move further to the left relative to the background 224, the first extrapolated view frame 220 has object boundary holes 246 and 248 that are not properly filled by the frame 212 of the input view 210.

  Appropriate pixels must be extrapolated to the frame boundary hole 244 and the object boundary holes 246 and 248 to generate an extrapolated view.

  The second extrapolated view frame 230 also has frame boundary holes 254 and object boundary holes 256 and 258. The viewpoint of the second extrapolation view is further away from the viewpoint of the input view 210 than the viewpoint of the first extrapolation view. The background 234, the first object 236, and the second object 238 in the frame 230 of the second extrapolation view are further compared to the background 224, the first object 226, and the second object 228 in the frame 220 of the first extrapolation view, respectively. Located on the left side.

  The frame boundary hole 254 and the object boundary holes 256 and 258 in the frame 230 of the second extrapolation view are wider than the frame boundary hole 244 and the object boundary holes 246 and 248 in the frame 220 of the first extrapolation view, respectively. .

  Therefore, more pixels must be extrapolated to frame boundary hole 254 and object boundary holes 256 and 258 in frame 230 of the second extrapolated view.

  That is, the farther the viewpoint of the extrapolated view is from the outermost input view, the wider the range of pixels that must be extrapolated.

  The frame boundary holes 244 and 254 and the object boundary holes 246, 248, 256, and 258 are collectively referred to as holes.

  FIG. 3 is a structural diagram of a video processing apparatus according to an embodiment. The video processing apparatus illustrated in FIG. 3 generates an output view through view interpolation and view extrapolation using binocular disparity information of the reference view video and the reference view video, respectively. The video processing apparatus restores holes generated by view interpolation and view extrapolation in the video of the output view.

  The output view video is a video at the viewpoint of the output view. As described above, when an image at a new viewpoint is generated, a point that must be newly observed at the new viewpoint is indicated as a hole in the image.

  The video processing apparatus may restore holes generated in the view interpolation and view extrapolation processes. The video processing apparatus 300 includes a video warping unit 310, a buffer interval setting unit 320, a binocular parallax crack detection unit 330, an adjacent video base hole restoration unit 340, an adjacent pixel scaling base hole restoration unit 350, and an optimal patch search base hole restoration unit. 360 may be provided.

  The video warping unit 310 generates an output view video by an image warping method using binocular disparity information of the reference view video and the reference view video. That is, the video warping unit 310 generates an output view frame by image warping using binocular disparity information of the reference view frame and the reference view frame. The video warping unit 310 generates an output view video by warping the reference view video. For example, when the binocular parallax information of the reference view video is not provided, the video warping unit 310 may generate the binocular parallax information of the reference view video.

  N input images (that is, N reference view images) are aligned with respect to an epipolar line.

  The virtual viewpoint view (i.e., the output view) is generated by using a weight proportional to the distance separated from the reference view to the virtual viewpoint view. That is, the view of the virtual viewpoint is generated by using a weight value proportional to the distance between the viewpoint of the reference view and the virtual viewpoint.

  The y coordinate value of the first pixel in the output view image may be the same as the y coordinate value of the second pixel in the reference view image corresponding to the first pixel. That is, the y coordinate of the pixel may not be changed by warping. The x coordinate of the first pixel in the output view (or rendered view) image can be calculated according to Equation (1) below.

Here, Ireference view indicates the video (or frame) of the reference view (or input view). The rendered view shows the video (or frame) of the output view (or rendered view).

  x indicates the coordinate value (x coordinate value) of the second pixel in the reference view. x ′ represents the coordinate value (x coordinate value) of the first pixel in the output view.

  d is a binocular disparity value of the second pixel derived from video depth information or pixel depth information. In general, the binocular disparity value of a pixel is inversely proportional to the pixel depth value. Therefore, the above description regarding depth and depth information may be applied to binocular parallax and binocular parallax information. α is a weight value proportional to the distance between the viewpoint of the reference view and the viewpoint of the output view.

  Therefore, the mathematical formula (1) described above means the following (1) to (5).

  (1) The second pixel in the reference view moves by αd by warping. That is, when the second pixel in the reference view image and the first pixel in the output view correspond to each other, the x coordinate value of the first pixel is a value obtained by adding αd to the x coordinate value of the second pixel.

  (2) Pixels having a large binocular disparity value move a lot due to warping. The binocular disparity value is inversely proportional to the depth value. Therefore, a pixel with a small depth value moves more than a pixel with a large depth value.

  The object is near the reference view viewpoint and the background is far from the reference view viewpoint. Accordingly, the pixels representing the object among the pixels of the reference view image move more than the pixels representing the background.

  Pixels with infinite depth values (or pixels with binocular disparity values of 0) may not be moved by warping.

  (3) The farther the reference view viewpoint and output view viewpoint are, the more pixels in the reference view move.

  The video warping unit 310 may generate the video of the output view using various warping methods other than the warping based on the above-described formula (1).

  (4) The binocular parallax value (or depth value) of the first pixel is the binocular parallax value (or depth value) of the second pixel.

  (5) One or more pixels in the reference view video may move to the same coordinates in the output view. In such a case, the pixel closest to the viewpoint of the output view among the one or more pixels is preferentially displayed.

  The buffer interval setting unit 320 enlarges the hole generated by the video warping. The hole expansion by the buffer section setting unit 320 will be described in detail below with reference to FIG.

  The binocular parallax crack detection unit 330 sets a crack in a hole in the generated video of the output view. The hole setting by the binocular parallax crack detection unit 320 will be described in detail below with reference to FIG.

  The adjacent video base hole restoration unit 340 restores the holes generated by the video warping. The hole restoration by the adjacent image base hole restoration unit 340 will be described in detail below with reference to FIG.

  The adjacent pixel scaling base hole restoration unit 350 restores a hole generated by image warping by scaling one or more pixels adjacent to the hole. The adjacent background scaling by the adjacent pixel scaling base hole restoration unit 350 will be described in detail below with reference to FIGS.

  The optimal patch search base hole restoration unit 360 searches the background for a patch that is most similar to the region including the hole generated by the video warping, and restores the hole by using the searched patch. The optimum patch search and hole restoration by the optimum patch search base hole restoration unit 360 will be described in detail below with reference to FIG.

  After a part of the hole is restored by the adjacent video base hole restoration unit 340, the remaining part or part of the hole is restored by the adjacent pixel scaling base hole restoration unit 350 and the optimum patch search base hole restoration unit 360. The

  The adjacent pixel scaling base hole restoration unit 350 and the optimum patch search base hole restoration unit 360 restore the remaining holes by using the background pixels of the output view image. Depending on the characteristics of the area adjacent to the hole, it is determined how the hole is restored.

  For example, when the region adjacent to the hole is a texture region, the optimal patch search base hole restoration unit 360 may restore the hole. If the area adjacent to the hole is not a texture area, the adjacent pixel scaling base hole restoration unit 350 may restore the hole. In order to scale the background pixels, the adjacent pixel scaling base hole restoration unit 350 may damage the texture due to the scaling.

  That is, if the region adjacent to the hole is uniform or a strong edge appears in the region adjacent to the hole, the hole region can be restored by scaling the background pixels. Even in the hole area restored by such restoration, the characteristics of the adjacent area (that is, the background) can be maintained.

  When the area adjacent to the hole is a texture area, the most similar area to the adjacent area is detected in the background area in the reference view video. The hole area is restored using the detected area. The texture component can be maintained even in the hole area restored by such restoration.

  FIG. 4 illustrates hole restoration using temporally adjacent images according to an example.

  Of a series of temporally adjacent videos, a reference view video at visual t may be used in image warping to generate an output view image at visual t. Hereinafter, the video of a specific visual t is named “video (t)” or “current video”. Similarly, a particular visual t frame is named “frame (t)” or “current frame”. The frame of the visual t−1 is named “frame (t−1)” or “previous frame”. The frame of the visual t + 1 is named “frame (t + 1)” or “next frame”.

  In FIG. 4, the output view frame (t) 440 includes a hole 442. The output view frame (t) 440 is generated by warping the reference view frame (t) 430.

  In general, video (t) that is temporally adjacent (for example, video (t-1), video (t + 1), video (t-2), video (t + 2), etc.) is the foreground indicated by the video (t). (Or object) and the same or similar foreground and background as the background. Therefore, the holes generated by the video warping can be restored by using information of the video temporally adjacent to the reference view video or the output view video.

  In FIG. 4, the object corresponding to the foreground moves from top to bottom. Accordingly, a portion 412 corresponding to the background hidden by the object in the reference view frame (t) 430 appears in the reference view frame (t−1) 410. Also, the portion 452 of the reference view frame (t + 1) 450, the portion 422 of the output view frame (t−1) 420, and the portion 462 of the output view frame (t + 1) 460 are also hidden from the object in the reference view frame (t) 430. Corresponding to the background. Accordingly, portions 412, 422, 452, and 462 may be used to restore hole 442. The adjacent video base hole restoration unit 340 may restore the hole 442 of the video (t) of the output view based on the following formula (2). Equation (2) describes a method for restoring holes in the frame using a frame temporally adjacent to the frame of the output view.

Here, f _t indicates an output view frame (t) 440. That, f _t denotes a frame generated by warping in the visual t. f _t (i, j) indicates the color value of the pixel having the coordinate value (i, j) among the pixels of the output view frame (t) 440. Here, in the output view frame (t) 440, the pixel whose coordinate value is (i, j) is the pixel in the hole 442. A pixel having a coordinate value of (i, j) is hereinafter denoted as “pixel (i, j)”.

f _t−1 indicates the previous frame of the output view frame (t) 440, that is, the output view frame (t−1) 420. f _{t + 1} indicates the next frame of the output view frame (t) 440, that is, the output view frame (t + 1) 460. f _t−1 indicates an output view frame (t−1) 420. F _{t + 1} indicates an output view frame (t + 1) 460.

That is, the neighboring video base hole restoration unit 340 may restore a hole generated by video warping using an output view frame temporally adjacent to the output view frame (t) 440. α _t−1 is a coefficient that determines whether pixel (i, j) of output view frame (t−1) 420 is used to restore pixel (i, j) of output view frame (t) 440 It is. α _t-1 may have a value of 0 or 1. If α _t−1 is 0, pixel (i, j) of output view frame (t−1) 420 is not used to recover pixel (i, j) of output view frame (t) 440.

α _{t + 1} is a coefficient that determines whether or not the pixel (i, j) of the output view frame (t + 1) 460 is used to restore the pixel (i, j) of the output view frame (t) 440. α _{t + 1} may have a value of 0 or 1. If α _{t + 1} is 0, pixel (i, j) of output view frame (t + 1) 460 is not used to recover pixel (i, j) of output view frame (t) 440.

If α _t−1 and α _{t + 1} are all 1, then the color value of pixel (i, j) of output view frame (t−1) 420 and the pixel (i, j) of output view frame (t + 1) 460 The average value of the color values becomes the color value of the pixel (i, j) of the output view frame (t) 440. When α _t−1 is 0 and α _{t + 1} is 1, the color value of pixel (i, j) in output frame (t) 440 is the color of pixel (i, j) in output view frame (t + 1) 460. Value. When α _t−1 is 1 and α _{t + 1} is 0, the color value of pixel (i, j) in output view frame (t) 440 is the pixel (i, j) in output view frame (t−1) 420. ) Color value.

  Pixels used to restore holes can be considered as pixels contained in the background. In general, the background is not repositioned by warping. For example, the portion 412 of the reference view frame (t−1) 410 and the portion 422 of the output view frame (t−1) 420 may have the same position and color in each frame. That is, the coordinates of the first pixel included in the background in the reference view frame and the coordinates of the second pixel corresponding to the first pixel in the output view frame are the same.

Therefore, the adjacent video base hole restoration unit 340 may restore holes generated by video warping using temporally adjacent reference view frames. That is, f _t−1 in Equation (2) may be replaced with f ′ _t−1 indicating the reference view frame (t−1) 410, and f _{t + 1} indicates f ′ _{t + 1} indicating the reference view frame (t + 1) 450. May be substituted,
Similarly, ft _-1 and _{ft + 1} in equation (2) may be replaced by _ft-2 and _{ft + 2} , respectively, or by any different temporally adjacent output view frames. Good. Also, f _t−1 and f _{t + 1} in equation (2) may be replaced by f ′ _t−2 and f ′ _{t + 2} , respectively, or may be replaced by any different temporally adjacent reference view frames. . In other words, the adjacent video base hole restoration unit 340 generates holes in the output view video generated by video warping using one or more temporally adjacent reference videos or one or more temporally adjacent output videos. Restore.

  Based on Equation (2), the neighboring video base hole restoration unit 340 uses one or two reference view frames, in other words, the output view frame (t−1) 420 and the output view frame (t + 1) 460. The hole 442 is restored. However, temporally adjacent output view frames (or reference view frames) may be used to reconstruct the hole 442. For example, the output view frame (or reference view frame) at vision t-2, t-1, t + 1 and t + 2 or any other three or more temporally adjacent images may be used to restore hole 442. May be.

  The temporally adjacent images need not be symmetric with respect to the image at visual t. For example, a hole in the video at visual t may be restored using the video at visual t-2 and t-1, or the video at visual t-3, t-2 and t-1, or any different time May be restored using video that is asymmetric in nature.

  In addition, the number of temporally adjacent videos may be dynamically changed based on the storage capacity of the video processing device 300, the complexity of the reference view video, the complexity of the output view video, and the like.

  In the formula (2), the color value of the pixel of the frame (t) is restored using the pixel having the same coordinate value as the pixel of the frame (t). In other words, the above-described restoration is based on the premise that a series of images are not moved as a whole according to time. That is, the background of the image does not move at all or hardly in some situations.

  If the video moves globally as a function of time, such movement should be taken into account to restore the color values of the pixels of the output view frame (t) 440.

  For example, when the output view frame (t) 440 is moved to the left by one pixel as a whole compared to the output view frame (t−1) 420 (or the output view frame (t) 410), the output view frame (t) ) To set the color value of 440 pixels (i, j), the color value of pixel (i + 1, j) of output view frame (t−1) 440 (or reference view frame (t) 410) is Must be used.

  That is, the neighboring image base hole restoration unit 340 corresponds to the first pixel in the output view image adjacent to the output view image (t) in order to restore the first pixel in the hole of the output view image (t). Pixel color values may be used. Alternatively, the neighboring video base hole restoration unit 340 corresponds to the first pixel in the reference view video adjacent to the reference view video (t) in order to restore the first pixel in the hole of the output view video (t). Pixel color values may be used.

  In addition, when the video adjacent in time to the reference view video (t) and the reference view video (t) moves as a whole according to time, the adjacent video base hole restoration unit 340 may perform a reference view based on the movement. A pixel corresponding to the first pixel may be selected in the reference view video (or output view video) adjacent to the video (t) (or output view video (t)). The adjacent video base hole restoration unit 340 can determine the coefficient α based on the following formula (3).

D ( _ft (i, j)) is the binocular parallax value of the pixel (i, j) of the output view frame (t) 440. Th is a predefined threshold value. Therefore, the value of α _t is 1 when f _t (i, j) is not a hole pixel, and the binocular parallax value of f _t (i, j) is a predefined threshold value, otherwise Is 0.

First, it is determined whether or not f _t (i, j) is a hole pixel indicating a hole. If f _t (i, j) is a hole pixel, the color value of f _t (i, j) does not exist or is an incorrect value. Therefore, the color values of f t _(i, j) can not be used to restore the holes in the output view video. Therefore, the value of α _t is 0.

  That is, the adjacent video base hole restoration unit 340 may restore holes in the output view video t by removing corresponding hole pixels in the temporally adjacent reference view video. Further, the adjacent video base hole restoration unit 340 may restore holes in the output view video by removing corresponding hole pixels included in the temporally adjacent output view video.

Next, if f _t (i, j) is not a hole pixel, it is determined whether f _t (i, j) is a pixel representing the foreground or a pixel representing the background.

  A hole generated by the multi-view rendering is a part of the background that is hidden in the foreground in the reference view video and appears in the output view video generated by warping. Alternatively, the hole may include a part of the background.

Therefore, f t _(i, j) may be used to restore a hole in the f _{t (i,} j) is output view image only if a background pixels representing background.

  That is, the adjacent video base hole restoration unit 340 restores a hole in the output view video t using only the corresponding background pixels in the temporally adjacent reference view video. Further, the adjacent video base hole restoration unit 340 restores a hole in the output view video t by using only the corresponding background pixel in the temporally adjacent output view video.

Whether f _t (i, j) is a foreground pixel representing the foreground or a background pixel representing the background is determined based on the binocular disparity value of f _t (i, j).

  In general, pixels representing the foreground have a larger binocular disparity value than pixels representing the background.

_Therefore, f t _(i, j) when the binocular parallax value is smaller than the reference value Th, considers the adjacent image-based hole recovery unit 340 _f t (i, j) and the background _pixels, f t _(i, j ) May be used to restore the hole.

  FIG. 5 illustrates hole expansion by setting a buffer section according to an example. FIG. 5 shows an output view image 520 on which an output view image 510 and buffer areas 522 and 524 are displayed.

  The binocular disparity value of the pixel used in multi-view rendering can be obtained by converting the physical depth value of the pixel. Also, the binocular disparity value of the pixel may be determined by estimation using a reference image.

  The binocular disparity value of a pixel may have an incorrect value due to a matching error (especially when obtained by estimation). If the binocular disparity value of the pixel is incorrect, the boundary between the object of the output view image 510 and the background may not match the boundary between the object of the binocular disparity image and the background. The pixel on the left side of the hole area 512 of the output view image 510 has a color value representing the background even though it is a pixel representing the foreground due to the above-described mismatch.

  That is, when the estimated binocular disparity value is used, an area adjacent to the hole needs to be set as a buffer area in order to prevent this problem.

  Accordingly, the buffer section setting unit 320 expands the hole by regarding the buffer areas 522 and 524 adjacent to the hole as holes.

  The buffer section setting unit 320 may set a pixel whose distance from the hole (or the outermost point of the hole) is smaller than a predetermined threshold as the buffer area. When the color value of the pixel in the buffer area is restored, the following formula (4) may be used.

In the equation (4), not only the color value of the pixel of the previous frame of the frame (t) and the color value of the pixel of the subsequent frame, but also the pixel (i, j) (that is, the color value of the frame (t) is restored. The color value of the pixel itself is also used for restoration. The pixel (i, j) in the buffer region already has a color value, unlike the pixel that was originally a hole. Thus, the color value of the pixel (i, j) in the buffer area itself may be used to recover the pixel (i, j) in the buffer area that was considered a hole.

  That is, when restoring the first pixel in the buffer area, the neighboring video base hole restoration unit 340 restores the first pixel based on the color value of the first pixel.

  FIG. 6 illustrates hole setting based on the occurrence of cracks and detection of binocular parallax cracks according to an example. In FIG. 6, the binocular parallax image 620 of the reference view image 610 and the reference view image 610 is shown.

  As illustrated, the first portion 622 and the second portion 624 of the binocular parallax image 620 have different binocular parallax values. In general, the first portion 622 and the second portion 624 should have the same or similar binocular disparity value to each other to represent the same object. However, particularly when the binocular disparity value is estimated, the first portion 622 and the second portion 624 may have different binocular disparity values.

  Since the first portion 622 and the second portion 624 have different binocular parallax values, the distance moved by warping is different. Since the first portion 622 and the second portion 624 move at different distances, the portion corresponding to the first portion 622 and the portion corresponding to the second portion 624 in the output view image 630 generated by warping. Cracks 632 may occur.

  A background other than the foreground may be displayed at the portion where such a crack 632 occurs. That is, the background may be displayed instead of the first part 622 and the second part 624. As described above, the crack 632 means a portion where a background is displayed between objects (or separated portions of the object) due to different binocular parallax values. That is, when the binocular disparity value is assigned to an object portion and the object is warped, a crack is generated inside the object.

  In the portion where the crack 632 occurs, the background color value is warped. Therefore, the image quality of the output view image may be deteriorated by the crack 632. If the crack 632 is set to a hole, the hole restoration method can be applied to a portion where the crack 632 appears. Therefore, image quality deterioration due to the crack 632 can be reduced.

  The binocular parallax crack detection unit 330 detects a crack in the output view image. The binocular parallax crack detection unit 330 sets a hole where the crack has occurred. Cracks can be detected based on Equation (5) below.

Here, D _{i, j} is the binocular parallax value of the pixel (i, j) 642 in the output view video 630. Pixel (i, j) 642 is a pixel that checks whether or not it is a crack. D _{i + m, j + n} is the binocular parallax value of the pixel (i + m, j + n) adjacent to the pixel (i, j). Th is a predefined threshold value.

  If the sum of differences in binocular disparity values between a pixel in the output view image 630 and an adjacent pixel 644 is greater than a predefined threshold, the binocular disparity crack detection unit 330 detects the pixel as a crack. .

  Cracks occur when background pixels are warped in the area where the foreground appears. Therefore, the difference between the binocular disparity value of the background pixel located at the crack point and the binocular disparity value of the foreground pixels adjacent to the background pixel is large.

  Therefore, as in Equation (5), a crack can be detected based on the difference in binocular parallax value with the adjacent pixel 644. Further, image quality deterioration can be compensated by assigning the detected crack to a hole.

  The adjacent pixel 544 illustrated in FIG. 6 is exemplary. Any combination of pixels spaced from the first pixel in the output view may be used as an adjacent pixel of the first pixel.

  FIG. 7 illustrates adjacent pixel scaling according to an example. The adjacent pixel scaling base hole restoration unit 350 uses one or more pixels adjacent to the hole to restore the hole of the image 710.

  The adjacent pixel scaling base hole restoration unit 350 may scale the pixels in the horizontal direction.

  A horizontal line 712 is shown that is subject to hole restoration. The adjacent pixel scaling base hole restoration unit 350 detects holes in the horizontal line 712 and detects the number of consecutive hole pixels 730.

  The adjacent pixel scaling base hole restoration unit 350 scans in the horizontal direction to detect holes. The adjacent pixel scaling base hole restoration unit 350 selects only the same number of pixels 740 as the number of consecutive hole pixels 730. The selected pixel is generally a pixel that is not a hole. The selected pixel 740 is a pixel adjacent to a continuous hole pixel 730. The selected pixel 740 may be a pixel on the same horizontal line as consecutive hole pixels 730.

  In FIG. 7, the selected pixel 740 is the pixel to the right of successive hole pixels 730. However, the pixel to the left of successive hole pixels 730 may also be the selected pixel 740. Also, the selected pixel 740 may be a pixel on the right side and a pixel on the left side of the continuous hole pixel 730.

  The adjacent pixel scaling base hole restoration unit 350 restores holes by scaling the selected pixel 740 in a direction toward the hole pixel 730.

  For example, the color values of the first hole pixel and the second hole pixel are generated using the color value of the first selected pixel. Thus, the first hole pixel and the second hole pixel are restored by the first selected pixel.

  The scaling of the selected pixel 740 may double the area indicated by the selected pixel 740, and the increased area may replace the area indicated by the hole pixel 730 and the selected pixel 740.

  Here, the number of hole pixels 730 and the number of selected pixels 740 are the same. Thus, all holes can be restored by scaling each selected pixel 740 to two pixels. Also, all selected pixels 740 can be scaled equally.

  FIG. 8 illustrates adjacent pixel scaling using background pixels according to an example.

  The adjacent pixel scaling base hole restoration unit 350 selects pixels 820 that are not the same number of holes as the number of consecutive hole pixels 810. Each pixel 820 that is not a hole has a binocular disparity value. The adjacent pixel scaling base hole restoration unit 350 classifies each pixel 820 that is not a hole as a foreground pixel or a background pixel according to the binocular disparity value.

  For example, the adjacent pixel scaling-based hole restoration unit 350 classifies pixels that are larger than a threshold value that is a binocular disparity value among pixels 820 that are not holes as foreground pixels, and the binocular disparity value is defined in advance. Pixels below the threshold are classified as background pixels.

  In FIG. 8, the previous three pixels 830 are classified as background pixels, and the last pixel 840 is classified as a foreground pixel. The adjacent pixel scaling base hole restoration unit 350 may restore a hole by scaling the background pixel 830 in a direction toward the hole pixel 810.

  Here, the number of background pixels 830 may be smaller than the number of hole pixels 810. Thus, all or part of the background pixel 830 is scaled to more than one pixel. Also, the background pixels 840 may be scaled non-uniformly. Foreground pixels 840 are excluded from scaling. Therefore, the foreground image can be prevented from being deformed by scaling.

  The above-described scaling method expands the pixel 740 or 830 only in the horizontal direction of the scan direction in FIG. Therefore, it can be easily realized by such a scaling method. In addition, when such a scaling method is used, even if a small-sized hole is restored, deterioration in image quality is not recognized so much.

  FIG. 9 illustrates scaling in a direction perpendicular to a background gradient according to an example.

  The output view image 910 includes a hole 912. If there is a specific shape 916 in the background 914 and the horizontal scaling described above is used, the shape 916 may not be maintained smoothly.

  Accordingly, such a shape 916 needs to be scaled in the direction 924 and the vertical direction 926 of the edge 918 of the shape 916 contained within the hole 912.

  The adjacent pixel scaling base hole restoration unit 350 selects a background pixel 922 adjacent to the hole 912. The adjacent pixel scaling base hole restoration unit 350 calculates the gradient of the edge 918 including the background pixel 922. The slope of edge 918 including background pixel 922 is the slope of background pixel 922. The adjacent pixel scaling base hole restoration unit 350 detects the number of hole pixels continuous in the vertical direction of the gradient of the edge 918. The adjacent pixel scaling base hole restoration unit 350 scales the background pixels in the vertical direction of the gradient of the edge 918 so that the detected consecutive hole pixels can be restored.

  That is, the adjacent pixel scaling base hole restoration unit 350 scales the background pixel (or part of the background) including the adjacent background pixel 922 in a direction perpendicular to the gradient of the background pixel 922 adjacent to the hole 912. May be restored. The background pixel shows a shape 916 that includes a background pixel 922.

  Here, the adjacent pixel scaling base hole restoration unit 350 detects the number of hole pixels that are continuous in the vertical direction with respect to the gradient of the background pixel 922, and determines the background pixel used for scaling based on the detected number of hole pixels. And how much the background pixels are scaled.

  By using the scaling of the background pixel 922 and the vertical scaling described above, the adjacent pixel scaling base hole restoration unit 350 may restore the hole region while maintaining the directionality of the background pixel (eg, the shape 916). it can.

  Background pixels can be scaled in a direction perpendicular to the edge by a scaling method using a background gradient. Therefore, the scaling method using the background gradient can achieve a more natural result than the scaling method performed in the horizontal direction. Scaling methods using background gradients must perform relatively complex operations.

  FIG. 10 illustrates hole restoration based on an optimal patch search according to one embodiment.

  The output view image 1010 includes a hole. If the background adjacent to the hole is a texture region, the hole cannot be accurately restored by simply scaling the background pixels.

  When the background adjacent to the hole is a texture, a patch that is most similar to the background adjacent to the hole is detected in the entire background region. The hole is restored using the detected patch. That is, when a texture that is the same as or very similar to the texture of the background adjacent to the hole is found in another background region, the portion adjacent to the other background region may be used to restore the hole.

  First, consider how to determine a point to restore using a patch.

  A method of preferential processing from holes adjacent to the background is used. By using a method in which the region of the hole is restored in order from the portion adjacent to the background, the foreground pixels may be structurally limited in the hole restoration process.

  The optimal patch search base hole restoration unit 360 detects a hole pixel in a raster scan direction. The detected initial hall point 1012 is adjacent to the foreground. Therefore, the point is not suitable for preferential restoration. The optimal patch search base hole restoration unit 360 detects hole pixels continuous in the scanning direction from the initial hole point 1012. The optimum patch search base hole restoration unit 360 sets the tips of consecutive hole pixels as the outermost hole point 1014. The outermost hall point 1014 is adjacent to the background. Therefore, a patch that is adjacent to the outermost hole point 1014 and can recover the hole area around the outermost hole point 1014 by using the color value and binocular parallax value of a pixel that is not a hole is searched.

  The optimal patch search base hole restoration unit 360 restores the optimal patch search base hole to the set outermost hole point 1014. The following explains how to restore the optimal patch search base hole. The optimal patch search base hole restoration unit 360 sets an area adjacent to the outermost hole point 1014 as the window area 1016. The window area 1016 may be N × N pixels.

  The window area 1016 includes a hole area 1020 and a background area 1018. The background area 1018 is used to search for patches. The hole area 1020 is an area that is restored by using the searched patch.

  The optimum patch search base hole restoration unit 360 detects an optimum patch corresponding to the window area 1016 in the entire background area, and restores the outermost hole point 1014 and the hole area 1020 set using the detected patch. The optimum patch means an area having the highest similarity with the window area 1016 in the entire background area.

  The patch has the same size as the window area 1016. The patch also includes a part corresponding to the hole area 1020 and a part corresponding to the background area 1018. The similarity between the patch and the window area 1016 may be calculated. The patch has the same size as the window area 1016 and is composed of a part corresponding to the hole area 1020 and a part corresponding to the background area 1018.

  When the similarity between the specific part and the window area 1016 is calculated, the hole area 1020 is excluded from the calculation of the similarity, and only the background area 1018 is used for calculating the similarity.

  For example, if a part of the patch corresponding to the background area 1018 has the same color value and binocular parallax value as the background area 1018, the patch is considered to be the same as the background area 1018. Therefore, the patch may be selected as the optimum patch.

  The optimal patch search base hole restoration unit 360 may use a method such as MAD (Mean of Absolute Difference) in order to calculate the similarity. The MAD method may use color values and binocular parallax values.

  That is, when the MAD between the specific patch and the window area 1016 is the minimum value among the plurality of patches, the optimal patch search base hole restoration unit 360 may select the specific patch as the optimal patch. .

  If the optimal patch is determined, the optimal patch search base hole restoration unit 360 restores the hole area 1020 using a portion corresponding to the hole area 1020 of the decided patch.

  FIG. 11 illustrates hole restoration using patch duplication according to an example.

  The optimal patch search base hole restoration unit 360 may use two or more patches to restore a specific hole pixel (or hole area). That is, the optimal patch search base hole restoration unit 360 may select two or more window regions 1112 and 1114 based on two or more outermost hole points different from each other. The optimum patch search base hole restoration unit 360 may search for the optimum patch for each of the two window regions 1112 and 1114, or may restore the hole using the two searched patches.

  If the two window areas 1112 and 1114 overlap each other, the overlapped hole area may be restored by two patches.

  In such a case, the optimum patch search base hole restoration unit 360 may restore the overlapped hole region using the average value of the color values of the two patches and the average value of the binocular parallax values.

  FIG. 12 is a flowchart of a video processing method according to an embodiment.

  In an image warping step S1210, an output view image is generated by image warping using the reference view image and the binocular disparity information of the reference view image. In the buffer section setting step S1220, the holes generated in the output view are enlarged by setting the buffer section. In the binocular parallax crack detection step S1230, a crack is set in the hole in the video of the output view.

  In adjacent video-based hole restoration step S1240, holes generated in the output view are restored by using one or more videos temporally adjacent to the reference view video or the output view video. In adjacent pixel scaling base hole restoration step S1250, holes are restored by scaling one or more pixels adjacent to the holes.

  In the optimum patch search base hole restoration step S1260, a patch that is most similar to the region including the hole is searched from the background, and the hole is restored by using the searched patch.

  The technical contents according to the embodiment of the present invention described above with reference to FIGS. 1 to 11 can be directly applied to the embodiment of the video processing method shown in FIG. Therefore, this detailed description will be omitted below.

  The method according to an embodiment of the present invention may be realized in the form of program instructions capable of executing various processes via various computer means and recorded on a computer-readable recording medium. The computer readable medium may include one or a combination of program instructions, data files, data structures, and the like. The program instructions recorded on the medium may be specially designed and configured for the purposes of the present invention, and are well known and usable by those skilled in the computer software art. May be. Examples of computer-readable recording media include magnetic media such as hard disks, floppy (registered trademark) disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as optical disks, and ROMs. A hardware device specially configured to store and execute program instructions, such as RAM, flash memory, etc., may be included. Examples of the program instructions include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate at one or more software layers to perform the operations of the present invention.

  FIG. 13 shows a display device including a video processing apparatus according to an example. Referring to FIG. 13, the multi-view display device 1300 may include a control unit 1310 and a video processing device 1305.

  The multi-view display device 1300 may have a form of a 3D display for displaying a 3D image, and may employ a multi-view scheme in three or more different viewpoints. Alternatively, the multi-view display device 1300 may have a stereoscopic display form that outputs a left image and a right image.

  The control unit 1301 may generate one or more control signals for controlling the multi-view display device 1300, and may generate one or more signals displayed by the multi-view display device 1300. The control unit 1301 may include one or more processors.

  The video processing apparatus 1305 may be used to generate a multi-view video for the multi-view display device 1300. For example, a video warping unit, a buffer interval setting unit, a binocular parallax crack detection unit, an adjacent video base hall One or more of a restoration unit, an adjacent pixel scaling base hole restoration unit, and an optimum patch search foundation hole restoration unit may be optionally provided. In FIG. 13, the unit is not shown. However, each such unit corresponds to the similarly named unit described herein, for example with reference to FIG. Therefore, it will not be further described in this embodiment.

  The video processing apparatus 1305 may be provided internally in the multi-view display device 1300, may be attached to the multi-view display device 1300, or may be realized separately from the multi-view display device 1300. Regardless of the physical configuration of the video processing device 1305, the video processing device 1305 may have the capabilities described above with reference to FIGS. Video processing device 1305 may include one or more internal processors. Alternatively, the one or more processors may be included in the multi-view display device 1300 together with the one or more processors of the controller 1301.

  The 3D image apparatus and method described above can apply various video formats. The video format is H.264. May include formats such as H.264 / MPEG-4AVC, High Efficiency Video Coding (HEVC), Dirac Video Compression Format, and VC-1, etc. Absent.

  As described above, the present invention has been described with reference to the limited embodiments and drawings. However, the present invention is not limited to the above-described embodiments, and any person having ordinary knowledge in the field to which the present invention belongs can be used. Various modifications and variations are possible from such an embodiment.

  Accordingly, the scope of the present invention is not limited to the disclosed embodiments, but is defined not only by the claims but also by the equivalents of the claims.

300 Video processing device 310 Video warping unit 340 Adjacent video base hole restoration unit 350 Adjacent pixel scaling base hole restoration unit 360 Optimal patch search base hole restoration unit

Claims (41)

A processor that controls executable units of one or more processors;
A video warping unit that generates a video of an output view by video warping using binocular disparity information of a video of a reference view and the video of the reference view;
An adjacent video base hole restoration unit that restores a hole generated by the video warping using one or more videos temporally adjacent to the video related to the video warping;
A video processing apparatus comprising:   The video processing apparatus according to claim 1, wherein the one or more videos that are temporally adjacent are videos that are temporally adjacent to the video of the reference view.   The video processing apparatus according to claim 1, wherein the one or more videos that are temporally adjacent are videos that are temporally adjacent to the video of the output view.   The adjacent image base hole restoration unit uses a color value of a pixel corresponding to the pixel in the hole in the temporally adjacent image to restore a pixel in the hole. The video processing apparatus according to 1.   When the reference view image and the temporally adjacent image move as a whole according to time, the adjacent image base hole restoration unit may include the hole in the temporally adjacent image based on the movement. The video processing apparatus according to claim 4, wherein a pixel corresponding to the pixel is selected.   5. The adjacent video base hole restoration unit restores the hole using a pixel obtained by removing one or more hole pixels corresponding to the pixel in the hole. Video processing equipment.   The video processing apparatus according to claim 4, wherein the adjacent video base hole restoration unit uses only a background pixel among pixels corresponding to the pixels in the hole to restore the hole.   The video processing apparatus according to claim 1, further comprising a buffer section setting unit that enlarges a hole generated by the video warping. The buffer section setting unit enlarges the hole by regarding the buffer area adjacent to the hole as the hole,
The adjacent image base hole restoration unit restores the pixel in the hole based on a color value of the pixel in the hole when the pixel in the hole is a pixel in the buffer area. The video processing apparatus according to claim 8.   The video processing apparatus according to claim 1, further comprising a binocular parallax crack detection unit configured to set a crack in the hole in the video of the output view.   The video processing according to claim 10, wherein the binocular parallax crack detection unit detects, as a crack, a pixel in which a sum of differences between binocular parallax values from adjacent pixels is larger than a predetermined threshold. apparatus. A processor that controls one or more processor-executable units;
A video warping unit that generates a video of an output view by video warping using binocular disparity information of a video of a reference view and the video of the reference view;
An adjacent pixel scaling base hole restoration unit that restores the holes generated by the image warping by scaling one or more pixels adjacent to the holes;
A video processing apparatus comprising:   The image processing apparatus of claim 12, wherein the adjacent pixel scaling base hole restoration unit scales one or more background pixels of the one or more pixels.   The video processing apparatus of claim 12, wherein the hole and the one or more pixels are on the same horizontal line.   The adjacent pixel scaling base hole restoration unit scales the one or more pixels in a direction perpendicular to a gradient of a background pixel adjacent to the hole, and the background pixel is one pixel of the one or more pixels. The video processing apparatus according to claim 12. A processor that controls executable units of one or more processors;
A video warping unit that generates a video of an output view by video warping using binocular disparity information of a video of a reference view and the video of the reference view;
Search for a patch that is most similar to a region including a hole generated by the video warping from a background, and use the searched patch to restore the hole by using an optimum patch search base hole restoration unit;
A video processing apparatus comprising: The region including the hole is composed of a hole region and a background region,
The video processing apparatus according to claim 16, wherein the optimum patch search base hole restoration unit restores the hole area using a portion corresponding to the hole area in the searched patches.   The optimal patch search base hole restoration unit searches for a first patch for a first region, searches for a second patch for a second region, and uses an average value of the first patch and the second patch. The video processing apparatus according to claim 16, wherein a hole portion overlapping in the first region and the second region is restored. A video warping step of generating an output view video by image warping the reference view video based on binocular disparity information of the reference view video by a processor;
An adjacent video-based hole restoration step of restoring holes generated in the output view using one or more videos temporally adjacent to the video related to the video warping;
A video processing method comprising:   The video processing method according to claim 19, wherein the temporally adjacent video is one or more videos temporally adjacent to the reference view video.   The video processing method according to claim 19, wherein the temporally adjacent video is one or more videos temporally adjacent to the video of the output view. A step of setting a buffer section for enlarging the hole;
A binocular parallax crack detection step of setting a crack in the hole in the video of the output view;
The video processing method according to claim 19, further comprising: An adjacent pixel scaling base hole restoration step for restoring the hole by scaling one or more pixels adjacent to the hole;
Search for a patch that is most similar to the region including the hole from the background, and use the searched patch to restore the hole by using an optimum patch search base hole restoration step;
The video processing method according to claim 19, further comprising:   24. A recording medium readable by a computer that stores a program for performing the method according to any one of claims 19 to 23. A processor that controls executable units of one or more processors;
A video generation unit that generates a video of an output view based on video of a reference view and binocular disparity information of the video of the reference view;
An adjacent video base hole restoration unit that restores a hole in the video of the generated output view using background information of one or more videos temporally adjacent to the video related to the generation, An adjacent video base hole restoration unit generated as a result of generating the output view;
A multi-view generation apparatus comprising:   The multi-view generation apparatus according to claim 25, wherein the one or more videos that are temporally adjacent are videos that are temporally adjacent to the video of the reference view.   26. The multi-view generation apparatus according to claim 25, wherein the one or more videos that are temporally adjacent are videos that are temporally adjacent to the video of the output view.   The multi-view generation apparatus according to claim 25, wherein the video generation unit generates the video of the output view by interpolating or extrapolating data from the reference view. A processor that controls executable units of one or more processors;
A video generation unit that generates an output view video based on at least one reference view video;
A binocular parallax crack detection unit for detecting a crack in a predefined object of the generated video of the output view, wherein the predefined object is assigned to a different part of the predefined object Binocular parallax that occurs in the predefined object due to the crack generating the video of the output view based on the video of the at least one reference view with different binocular parallax values determined A crack detection unit;
Reassigning the crack as a hole and using the background information of one or more frames temporally adjacent to the video associated with the generation, the hole present in the current frame of the generated output view video A multi-view generator that restores
A multi-view generation apparatus comprising:   30. The multi-view generation apparatus according to claim 29, wherein the one or more frames that are temporally adjacent are frames that are temporally adjacent to the video of the reference view.   The multi-view generation apparatus according to claim 29, wherein the one or more frames that are temporally adjacent are frames that are temporally adjacent to the video of the output view.   30. The multi-view according to claim 29, wherein the binocular parallax crack detection unit detects, as a crack, a pixel in which a sum of differences in binocular parallax values from adjacent pixels is larger than a predetermined threshold. Generator.   30. The multi of claim 29, wherein the temporally adjacent frames are at least one temporally generated frame before the current frame and a frame generated after the current frame. View generator. Generating an output view video based on the at least one reference view video by a processor;
Detecting cracks in a predefined object of the generated video of the output view, wherein the predefined object is a different binocular assigned to a different part of the predefined object With the disparity value, the crack occurs in the predefined object due to generating the output view video based on the at least one reference view video; and
Reassigning the crack as a hole and using the background information of one or more frames temporally adjacent to the video associated with the generation, the hole present in the current frame of the generated output view video Step to restore,
A multi-view generation method comprising:   35. The multi-view generation method according to claim 34, wherein the one or more frames that are temporally adjacent are frames that are temporally adjacent to the video of the reference view.   35. The multi-view generation method according to claim 34, wherein the one or more frames that are temporally adjacent are frames that are temporally adjacent to the video of the output view.   35. The multi-view generation method according to claim 34, wherein pixels having a sum of differences in binocular parallax values from adjacent pixels larger than a predetermined threshold are detected as cracks.   The multi-frame according to claim 34, wherein the temporally adjacent frames are at least one temporally generated frame before the current frame and a frame generated after the current frame. View generation method. In a display device comprising a video processing device,
A video warping unit that generates an output view video based on a reference view video and binocular disparity information of the reference view video;
An adjacent video-based hole restoration unit that restores a hole using one or more videos temporally adjacent to the video related to the generation, wherein the holes are generated by generating the output view. Image base hall restoration department,
A control unit for generating a signal to be displayed by the display device based on the generated video of the output view having the hole restored by the adjacent video base hole restoration unit;
A display device comprising: a video processing apparatus comprising:   40. The display device of claim 39, wherein the one or more videos that are temporally adjacent are videos that are temporally adjacent to the video of the reference view.   40. The display device of claim 39, wherein the one or more temporally adjacent videos are temporally adjacent videos of the output view.