KR100590025B1 - Method and device for synthesizing intermediate images in a multi-view square camera based display system - Google Patents

Abstract

The present invention provides a method for synthesizing an intermediate image from a part of a multiview image based on a multiview forward camera arrangement, and a distance matching step of identifying an intermediate image to be synthesized and an area where three neighboring reference images overlap with each other; ; A block matching step of performing block matching on a block in which the difference image between the reference images exists in the overlapping region; An image correction step of correcting an image of the overlapping area according to the block matching result; And synthesizing the intermediate image by integrating the corrected overlapping regions. In the image synthesizing step, the intermediate image may be synthesized by integrating a corresponding region of the reference image with the corrected overlapping region with respect to an area not included in the intermediate image determined to correspond to only one reference image in the distance matching step. have. In addition, the image synthesis may include performing image blending on the boundary line of the overlapped region. The block matching step includes calculating a parallax vector for a block in which the difference image between the reference images exceeds a predetermined threshold value in the overlapping region, and the image correction step uses the calculated parallax vector. To correct the image of the corresponding block.

Multiview, Intermediate, Composite, 3D, Reference, Distance Matching, Block Matching, Image Blending

Description

METHOD AND DEVICE FOR SYNTHESIZING INTERMEDIATE IMAGES IN A MULTI-VIEW SQUARE CAMERA BASED DISPLAY SYSTEM}

1 is an illustration of an intermediate image synthesis apparatus according to the prior art.

2 is an exemplary diagram of a multi-resolution multi-baseline stereo (MR-MBS) matching method for synthesizing intermediate images according to the prior art;

3 is a conceptual diagram illustrating an example of applying an intermediate image synthesis method according to a preferred embodiment of the present invention to a 4x4 square camera array.

4A is an exemplary diagram of region segmentation when the intermediate image synthesis method according to the preferred embodiment of the present invention is applied.

4B is an enlarged view of a target image in FIG. 4A.

5 is a step-by-step flowchart of the intermediate image synthesis method according to a preferred embodiment of the present invention.

The present invention relates to stereoscopic image synthesis, and more particularly, to an intermediate image synthesis method and apparatus for a multiview stereo camera array based display.

After high definition television (HDTV), 3D stereoscopic television has been widely studied as a next generation display. Although the method of using binocular disparity has been suggested as an early method of stereoscopic television, the viewer can feel a certain three-dimensional feeling, but it is not enough to feel the liveliness and reality as when a person actually sees an object. In order to overcome this disadvantage, many studies have been conducted to express more vibrant three-dimensional images by displaying using multi-view images obtained from two or more cameras.

A video stream used for multiview image processing is generally obtained by using one camera per viewpoint, and in a multiview camera arrangement method, an in-line parallel camera arrangement and a radial camera arrangement are used. Etc. However, in the case of a circular camera array, view distortion may occur when displaying a multiview image obtained from different cameras on one screen. This problem can be solved by a parallel camera arrangement, but in the case of a parallel arrangement, it is difficult to display a three-dimensional image up and down. In recent years, multi-view signal processing and display devices using square camera arrays have been studied.

In order to implement a multi-view video system for 3D stereoscopic television, a transmitter transmits an image obtained from a camera having various arrangements, and a receiver performs a display based on the transmitted data. However, in order to transmit a multiview video through one broadcasting channel, a compression rate much higher than that of a conventional single view video transmission is required for video compression. However, it is virtually impossible to transmit images of all viewpoints through existing broadcasting channels with the compression algorithms presented so far. To this end, instead of transmitting all viewpoint images corresponding to the camera array, an intermediate image synthesis which transmits only the number of viewpoints allowed by the channel and makes an unsent viewpoint image based on the transmitted image at the receiver. Many techniques are being studied.

The main algorithms related to intermediate image synthesis include feature based method, block based method, pixel-recursive method, and likelihood model for adaptive point extraction. The mesh method was mainly used. In addition, a method using a depth map is also used. Such an algorithm is described in Korean Patent Application No. 10-2000-0063753 "Multi-view Video System and Image Manufacturing Method for 3D Stereoscopic Image" (hereinafter referred to as "quotation document 1"), Korean Patent Application No. 10-1998- 0005297 "Dense Depth Information Extraction Method Using Multiple Camera Input Images and Intermediate Image Synthesis Method Using the Same" (hereinafter, referred to as "quotation document 2"), and the like, for example, Korean Patent Application No. 10-1995-0042335 It is applied to application systems such as "three-dimensional catalog shopping system and shopping method".

1 shows the decompression means 34 and the intermediate image generating means 35 disclosed in the cited reference 1 of the prior art.

According to Cited Reference 1, a method of finally obtaining N-view images by generating new (NM) images one by one between M-view images obtained in a stereoscopic system using N parallel multiview images is proposed. have. To this end, the adaptive point extractor 63 for extracting the adaptive point between the reference image, which is a reference of the inter-image, and the occluded region maker 64 for compensating the occlusion area between the reference image based on the adaptive point extraction data, It is used.

More specifically, as shown, the video decompression means 34 reconstructs an image of nine views and inputs the intermediate image generation means 35. The intermediate image generating unit 35 extracts an adaptive point for the reference image Ov1, Ov3, Ov4, Ov6, Ov7, and Ov9 from each of the underlying reference images Ov2, Ov4, Ov5, Ov6, and Ov8. In operation 63, the occlusion area manufacturer 64 extracts and generates the occlusion area based on the adaptive point extraction data. Next, by synthesizing the data corresponding to the adaptation point and the interpolated occlusion area to generate a new intermediate image, it is possible to obtain 17 views.

FIG. 2 illustrates a multi-resolution multiple baseline stereo (MR-MBS) matching method disclosed in Citation Document 2, which is a prior art.

The MR-MBS method of FIG. 2 is a method of densely extracting depth information of each pixel from images obtained from finite cameras at various positions, and creating an intermediate image using the multi-linear stereo matching (MBS). It's a different implementation.

Referring to FIG. 2, an input image obtained from multiple cameras for one object is decomposed to multi-resolution, and then one pixel (sub) of the object is subjected to multi-linear stereo matching (MSB) and occlusion area processing on the most low-resolution image. The depth information for the node) is obtained (step 1), and the information is transferred to step 2 for high resolution processing. In the second step, multi-linear stereo matching and occlusion area processing is performed on multiple pixels (child nodes) at high resolutions corresponding to one pixel of low resolution based on the transmitted depth information to obtain depth information of the child nodes. By repeating this process up to the highest resolution, depth information for each pixel of the object is precisely obtained, and an intermediate image is obtained based on the depth information.

However, in the conventional method of synthesizing intermediate images, when the base-line distance is far from each other or the angle and distance between the camera positions are far from each other, a large number of occlusion areas are generated, and thus it is not easy to synthesize the intermediate images. . In addition, since the adaptation point extractor is applied to the entire object, unnecessary depth map information and excessive adaptation point data may be generated.

For example, in Citation 1, when synthesizing the images between the left and right images based on the viewpoint images transmitted from the transmitting unit in the parallel camera arrangement, the distance between the referenced left and right images becomes considerably farther when the baseline distance is a little further away. do. And, due to this, in the synthesis of the intermediate image, the occlusion generation area increases a lot, and the adaptive point extractor alone cannot completely compensate the occlusion area. In addition, since the adaptation point extractor is applied to the entire object, excessive simulation time is required.

In the above-mentioned Reference 2, although the problem of extending the processing time and the boundary line in the depth map of the multi-line stereo matching is solved by a multi-resolution method using multiple cameras existing at arbitrary positions, the position and arrangement between each camera If the angle gets larger, the resolution of each pixel of the object through multiple cameras cannot be obtained accurately, and the exact depth map cannot be obtained. In addition, since this method obtains the entire depth map through all the cameras for one object and generates the required intermediate image, simulation time is consumed because of unnecessary depth map information.

In order to solve the above problems, the present invention proposes an effective method for synthesizing untransmitted view images based on a view image transmitted from a transmitter in an imaging system based on a square camera array, thereby providing a multi-view video in a limited channel situation. The purpose is to facilitate the implementation of the system (stereoscopic display).

In order to achieve the above object, according to the first aspect of the present invention, there is provided a method for synthesizing an intermediate image from a part of a multiview image based on a multiview forward camera arrangement, A distance matching step of identifying an area where the two reference images overlap; A block matching step of performing block matching on a block in which the difference image between the reference images exists in the overlapping region; An image correction step of correcting an image of the overlapping area according to the block matching result; And synthesizing the intermediate image by integrating the corrected overlapping regions.

In this case, the image synthesizing step may be performed by integrating a corresponding region of a reference image with the corrected overlapping region with respect to an area not included in the intermediate image determined to correspond to only one reference image in the distance matching step. Can be synthesized. In addition, the image synthesis may include performing image blending on the boundary line of the overlapped region.

The distance matching step may include calculating a distance HD of two parallel reference images among the three reference images by parallel movement and a minimum mean difference error; The parallel reference image is moved by half of the calculated distance to determine the position of the intermediate image, and the distance (VD) between the intermediate image and the other reference image is perpendicular to the parallel movement and minimum average Calculating by difference error; The parallel reference image is moved by half of the calculated distance HD and the other reference image is moved by the calculated distance VD, so that the intermediate image and three neighboring reference images overlap with each other. It may include the step of identifying the area.

The block matching step may be performed only on blocks in which the difference image between the reference images in the overlapping region exceeds a predetermined threshold value. The block matching may include calculating a parallax vector for a block in which the difference image between the reference images exceeds a predetermined threshold value in the overlapping region, and the image correction step includes calculating the disparity vector. And correcting an image of a corresponding block by using the same.

The block matching step may be to search for a block in a diagonal direction with respect to one of the three reference pictures toward the positions of the other two reference pictures.

According to a second aspect of the present invention, there may be provided an intermediate image synthesizing apparatus composed of means for performing each step of the above-described intermediate image synthesis method.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

Figure 3 shows an example of applying the intermediate image synthesis method according to a preferred embodiment of the present invention to the square camera array of 4x4. In FIG. 3, among 4 × 4 multi-view images obtained from each camera, ten images indicated by gray circles represent images transmitted from the transmitter to the receiver, and the remaining six images should be synthesized at the receiver without being transmitted to the receiver. An intermediate image is shown.

The present invention uses a matching distance, block matching, and image blending technique between reference images in order to efficiently implement a multiview stereoscopic system having a square camera arrangement. The undelivered intermediate image is synthesized by the receiver.

To this end, according to the present invention, for one intermediate image synthesis, Left-Right-Bottom, Top-Bottom-Right, or Top -Bottom-Left Three reference images are used, which is different from the prior art of performing interpolation using only two left and right images.

For example, when the images are displayed in the left and right index and the up and down index order shown in FIG. 3, the middle image (2,1) located second from the left at the top is the left (1,2) -right image (the surrounding image). 1,3) -the lower image (2,2) is synthesized as a reference image. However, the intermediate images 4 and 1 positioned at the upper right corner and the intermediate images 1 and 4 positioned at the lower left corner may refer to only two reference images, respectively.

As described above, according to the present invention, since the intermediate image is interpolated using three neighboring viewpoint images, the occlusion area can be minimized even if the distance between the cameras such as the base line is somewhat longer.

 In the case of a square camera arrangement, when a distance and an arrangement between cameras are determined, an area to be shared between neighboring viewpoint images is determined. In consideration of this, the simulation time and complexity are significantly reduced by finding a shared area at regular intervals using a distance matching method, and compensating for the occluded area using a block matching method only for the areas where many occlusions occur in the shared area. You can. In addition, an image blending technique may be used for the region where the boundary surface is finally generated to synthesize an accurate and natural intermediate image.

In FIG. 3, for convenience of description, an example of synthesizing six intermediate images from ten viewpoint images in a 4x4 square camera array structure has been described. However, the intermediate image synthesis method according to the present invention is not limited thereto. It can also be applied to the square camera array structure. For example, in the case of using the intermediate image synthesis algorithm according to the present invention, four intermediate images may be generated from five viewpoint images in a system having a 3x3 camera structure, and 12 intermediate images may be generated from 13 viewpoint images in a 5x5 camera structure. Can be.

4A illustrates region segmentation for intermediate image synthesis according to a preferred embodiment of the present invention. In FIG. 4A, a target image represents an intermediate image synthesized according to the present invention, and a left image (Reference Image 1), a right image (Reference Image 2), and a base image are represented by the target image. Three reference images used to synthesize an image are illustrated.

5 illustrates a stepwise intermediate image synthesis method according to a preferred embodiment of the present invention. In general, a matching distance step for acquiring an overlapping area and a block matching step for an occlusion area are shown. Block Matching) and image correction, and finally, Image Blending to resolve the boundary. On the other hand, when implementing the intermediate image synthesis method according to a preferred embodiment of the present invention in hardware, the means for performing each of these steps may be implemented individually, or some or all of them may be integrated with each other.

Hereinafter, an intermediate image synthesis method according to a preferred embodiment of the present invention will be described in detail with reference to FIGS. 4A and 5.

(1) Matching Distance Step

The distance matching step is to identify an overlapped region between the predicted intermediate image and the three reference images, which corresponds to steps S100 to S110 of FIG. 5.

More specifically, in step S100, by using a left image (Reference Image 1) and a right image (Reference Image 2) of FIG. 4A, one pixel is horizontally arranged on the other image based on one of the images. A minimum distance error (MDE), for example, a minimum average distance error (MADE) is applied while moving in parallel by (pixel). Accordingly, it is possible to obtain a horizontal distance HD (hereinafter, referred to as "HD") in which the average error sum is minimum.

Subsequently, in step S105, half-HD corresponding to the half is obtained from the HD obtained in step S100, and the position of the desired intermediate image (target image of FIG. 4A) is predicted. That is, by moving the base image by 1 pixel vertically based on the image obtained by parallelly moving one of the left image (Reference Image 1) and the right image (Reference Image 2) by half-HD, The vertical distance (VD; hereinafter referred to as "VD") at which the average error sum is minimized is calculated by MDE.

,

,

영역에 각각 중첩된다. 그리고, ④ 영역은 좌영상 및 우영상 간의 중첩 영역이며, 중간 영상의

영역에 중첩된다. 중간 영상의 ⑤, ⑥ 영역은 참조 영상 중에서 좌영상, 우영상에만 각각 존재하여, 차분이 발생하지 않는 영역이다. Next, in step S110, an overlapping region between the reference images is determined using the above-described VD and HD, and accordingly, a difference image (Diff. Image) in each overlapping region may be obtained. That is, when the lower image is vertically moved by VD based on the image obtained by parallelly moving one of the left and right images by half-HD, the region is divided according to mutual overlap between the reference images. For example, in the case of FIG. 4A, the regions ①, ②, and ③ are overlapping (shared) regions in which a difference image is generated between at least one of the left image and the right image and the lower image.

,

,

Each overlaps an area. The area ④ is an overlapping area between the left image and the right image,

Overlaid on the area Areas ⑤ and ⑥ of the intermediate image exist only in the left image and the right image of the reference image, so that no difference occurs.

(2) block matching step

Block matching steps S115 to S130 are for the occlusion area, and the value of the difference image between the reference image (that is, the difference between the lower image and the left image, or the difference between the lower image and the right image) is determined using the aforementioned matching distance. Block matching may be performed on existing blocks, and block matching may be performed on blocks having a difference value greater than the threshold value TH. This block matching step is repeated until the search is completed for all the difference images (S115).

In more detail, step S120 may calculate the block energy of the difference image determined in step S110 described above, and may selectively adjust the block size to, for example, 16x16, 8x8, 4x4, and the like. Subsequently, the block energy obtained in the above-described step S115 is compared with a specified threshold value TH (step S125), and if there is a block whose block energy exceeds the threshold value, step S130 is performed. Proceed to). If the block energy of the specific block does not exceed the threshold in step S125, the process proceeds to step S115 described above.

영역을 구하기 위하여, 하영상(Base Image)을 수직으로 VD 만큼, 수평으로 half_HD 만큼 이동시킨 후, 하영상(Base Image)의 ① 영역 내의 블록과 가장 유사한 블록을 좌영상(Reference Image 1) 또는 우영상(Reference Image 2)의 ① 영역에서 찾는다. In operation S130, block matching between reference images is performed on blocks whose block energy exceeds a threshold. For example, the target image

In order to obtain the area, move the base image vertically by VD and horizontally by half_HD, and then move the block most similar to the block in the ① area of the base image to the left image (Reference Image 1) or right. Search in the 1 area of the image (Reference Image 2).

When the left image, the right image, and the lower image are matched by the above-described distance matching method, the protruding parts of the object (for example, a raised arm, a leg, a holding object, etc.) of the object are synthesized due to the arrangement of the images. Based on the target image, an area where a difference greater than or equal to a threshold value appears in the upper image, is displayed on the left in the right image, and on the right in the left image. Using this property, when block matching is performed using the right image (Reference Image 2), that is, when the lower image is vertically raised by VD and the horizontal image is shifted by half_HD, the difference between the left and right images is obtained. The search time can be reduced by setting the search path diagonally leftward and downward. In the same way, when using the left image (Reference Image 1), the search path is determined diagonally rightward and downward. That is, in searching for a block that satisfies the above-described threshold condition based on one reference image, the search path is a diagonal direction toward the positions of the other two reference images.

영역은 하영상(Base Image)을 기준으로, 좌영상(Reference Image 1)의 ② 영역에서 우측, 아래 방향으로 블록 매칭을 수행하고,

영역은 하영상을 기준으로 우영상의 ② 영역에서 좌측, 아래 방향으로 블록 매칭을 행한다.In a similar manner as described above, the intermediate image (Target Image)

The area is based on the base image, and performs block matching in the right and down directions in the ② area of the left image (Reference Image 1),

The region performs block matching in the left and the down directions in the region ② of the right image based on the lower image.

영역은 좌영상(Ref. Image 1)과 우영상(Ref. Image 2)의 ④에만 해당하는 영역으로서, 하나의 영상(즉, 좌영상 또는 우영상)을 기준으로 좌영상과 우영상 간의 블록 매칭을 수행하고, 매칭되는 두 블록(좌영상의 블록과 우영상의 블록)간의 평균 값을 가지는 평균 블록을 구한다. 이어서, 전술한 기준 영상(좌영상 또는 우영상)을 기준으로 평균 블록을 half-HD 만큼 평행이동 시 킨다.Of the intermediate image

The area corresponds only to ④ of the left image (Ref. Image 1) and the right image (Ref. Image 2), and block matching between the left image and the right image based on one image (that is, the left image or the right image). Next, an average block having an average value between two matching blocks (block of left image and block of right image) is obtained. Subsequently, the average block is moved by half-HD based on the above-described reference image (left image or right image).

The area ⑤ of the intermediate image is an area existing only in the left image (Reference Image 1), and in this case, the area is taken from the left image. In the same way, the area ⑥ of the intermediate image exists only in the right image, so it is taken from the right image. Considering the entire image of the target image, this end portion does not have a significant visual impact when the multi-view image is actually viewed through the display device.

(3) image correction

,

,

영역의 합성에 있어서, 좌영상, 우영상, 그리고 하영상을 이용하여 전술한 단계(S130)의 블록 매칭을 실행하면, 시차 벡터(disparity vector) (dx, dy)를 얻을 수 있다. 거리 매칭 방법을 통해 얻은 거리(distance), 즉, VD, half_HD를 이용하여 참조 영상을 평행 이동시켜서 차분 영상을 구하면, 폐색 영역에 대해서만 차분이 발생한다. 이에 따라, 단계(S135)에서는, 일정한 임계값 이상인 차분 영역에 대하여, 시차 벡터(disparity vector)를 이용하여 영상 보정을 함으로써, 원하는 중간 영상(Target Image)을 합성할 수 있다. Of FIG. 4A

,

,

In synthesizing the region, when the block matching of the above-described step S130 is performed using the left image, the right image, and the lower image, a disparity vector (dx, dy) can be obtained. When a difference image is obtained by parallelly moving a reference image using distance obtained through a distance matching method, that is, VD and half_HD, a difference occurs only in the occlusion region. Accordingly, in step S135, a desired intermediate image may be synthesized by correcting an image using a disparity vector with respect to a difference region having a predetermined threshold value or more.

For example, it is assumed that the intermediate image to be actually obtained is located below + dy in the vertical direction with respect to the lower image. In this case, when block matching is performed using the right image (that is, when block matching is performed using the right image based on the lower image shifted by VD vertically and half_HD vertically), the parallax vector (- dx, + dy), and on the contrary, when block matching is performed using the left image, a parallax vector (+ dx, + dy) can be obtained in the diagonal right and lower directions.

,

및

영역을 합성할 수 있다. 한편, 수학식 1은 우영상(Right)을 이용하여 블록 매칭을 하는 경우이고, 수학식 2는 좌영상(Left)을 이용하여 블록 매칭을 하는 경우이다. Therefore, in step S135, the bottom image (Bottom) is corrected with respect to the difference region generated by using distance matching, as shown in Equations 1 and 2, based on the lower image, so that the desired intermediate image is corrected.

,

And

Regions can be synthesized. On the other hand, Equation 1 is a case of block matching using a right image (Right), Equation 2 is a case of block matching using a left image (Left).

Bottom [y + dy, x] = Bottom [y, x]

Bottom [y, x] = Right [y, x-dx]

Bottom [y + dy, x] = Bottom [y, x]

Bottom [y, x] = Left [y, x + dx]

When using the right image, as described above, a parallax vector of (-dx, + dy) is obtained. In Equation 1, a block at an arbitrary position in the lower image Bottom is shifted by + dy by any (y, x) coordinate value, that is, in case of block matching. Then, the continuous pattern is overlapped down and up in the bottom image. Subsequently, when a block of (y, x-dx) of the right image (Right) is substituted into (y, x), which is the original position (block) of the lower image, the bottom image (Bottom) is an intermediate image to be obtained. (Target Image) In this manner, using Equation 2, a desired target image (intermediate image) can be obtained using a left image.

영역의 경우, 좌영상의 ① 영역과 우영상의 ① 영역을 선택 적으로 참조하고,

영역은 좌영상의 ② 영역을 참조하며,

영역의 경우에는 우영상의 ③ 영역을 참조한다.Intermediate video

In case of area, refer to ① area of left image and ① area of right image selectively.

The area refers to area ② of the left image.

For the area, refer to the area ③ of the right image.

(4) Image Blending

If it is determined in step S115 that the block matching and image correction have been completed for all the difference images, the process proceeds to step S140 and an uncovered area of the target image that cannot be corrected as a lower image is performed. For example, for the region ⑤ and the region ⑥ in FIG. 4, the correction is performed using the left image or the right image. That is, the region not included in the intermediate image determined to correspond to only one reference image in the above-described distance matching step may combine the corresponding region of the reference image with the corrected overlapping region to synthesize the intermediate image.

영역의 경계선, ⑥ 영역과

영역의 경계선(이상, 도 4b의 A), 그리고,

,

,

영역과

, ⑤, ⑥ 영역의 경계선(도 4b의 B)은 알파맵을 이용한 영상 블렌딩 기법에 의하여 부자연스러움을 제거할 수 있다.Subsequently, in step S145, image blending is performed to process the boundary of the piece areas obtained in the above-described step. In this regard, FIG. 4B is an enlarged view of the boundary between the respective regions present in the target image of FIG. 4A. For example, as shown in FIG. 4B, the region ⑤ of the intermediate image and the target image are shown.

The boundary of the region,

The boundary line of the region (above, A of FIG. 4B), and,

,

,

Area and

The boundary lines (B in FIG. 4B) of the regions ⑤ and ⑥ may be removed by an image blending technique using an alpha map.

When image blending is completed in step S145, the intermediate image is finally generated by integrating the regions in step S150.

By using the above-described intermediate image synthesis method, a three-dimensional television (3D TV) can be implemented. That is, in order to realize 3D TV broadcasting through a limited size channel, only a minimum view image should be transmitted. However, the realization of 3D TV is achieved by synthesizing the untransmitted image in real time using the intermediate image synthesis algorithm according to the present invention. It is possible.

In addition, the distance matching and block matching method according to the present invention can be applied to a multi-view compression algorithm. That is, as described above, the overlap region is secured by distance matching, and in consideration of the positional characteristics of the differences appearing according to the reference image (left, right, and bottom), the parallax vector is used only for the region having a large difference. By using a method of obtaining, it can be applied to a multi-view compression algorithm using a parallax vector.

Although the preferred embodiment according to the present invention has been described above, this is merely exemplary and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. For example, in describing a preferred embodiment of the present invention, a left / right / low image is illustrated as a reference image, but according to a position of an intermediate image to be synthesized, an up / down / right image or an up / down / left image May be set differently. Therefore, the protection scope of the present invention should be defined by the following claims.

As described above, according to the present invention, in the case of a camera arrangement having a square structure, when a distance between cameras and an arrangement are determined, a predetermined time interval is determined by a distance matching method in consideration of the fact that an area shared between neighboring viewpoint images is determined. By finding the shared region and compensating for the occluded region by the block matching method only for the region where the occlusion occurs frequently in the shared (overlapped) region, the simulation time and complexity can be significantly reduced. In addition, by using the three reference images, it is possible to reduce the occlusion area and to synthesize an accurate intermediate image.

In addition, the intermediate image synthesis method according to the present invention enables real-time display of a multiview image, and thus can be used for 3D TV implementation, and the distance matching method and the block matching method can be applied to a multiview compression algorithm as an independent module.

Claims (16)

A method of synthesizing an intermediate image from a part of a multiview image based on a multiview forward camera array, A distance matching step of identifying a region where an intermediate image to be synthesized and three neighboring reference images overlap with each other; A block matching step of performing block matching on a block in which the difference image between the reference images exists in the overlapping region; An image correction step of correcting an image of the overlapping area according to the block matching result; An image synthesis step of synthesizing an intermediate image by integrating the corrected overlap regions; Intermediate image synthesis method based on a multi-view forward camera array comprising a. The method of claim 1, wherein the image synthesis step, A multi-view forward camera arrangement for synthesizing the intermediate image by integrating the corresponding region of the reference image with the corrected overlapping region with respect to an area not included in the intermediate image determined to correspond to only one reference image in the distance matching step. Based Image Synthesis Method. The method of claim 1, wherein the image synthesis step, A multi-view forward camera arrangement for synthesizing the intermediate image by integrating the corresponding region of the reference image with the corrected overlapping region with respect to an area not included in the intermediate image determined to correspond to only one reference image in the distance matching step. Based Image Synthesis Method. The method of claim 3, wherein the image synthesis step And performing image blending on the boundary line of the overlapping region. The method of any one of claims 1 to 4, wherein the distance matching step comprises: Calculating a distance (HD) of two parallel reference images among the three reference images by parallel movement and a minimum mean difference error; The parallel reference image is moved by half of the calculated distance to determine the position of the intermediate image, and the distance (VD) between the intermediate image and the other reference image is perpendicular to the parallel movement and minimum average Calculating by difference error, The parallel reference image is moved by half of the calculated distance HD and the other reference image is moved by the calculated distance VD, so that the intermediate image and three neighboring reference images overlap with each other. Steps to identify the area Intermediate image synthesis method based on a multi-view forward camera array comprising a. The method according to any one of claims 1 to 4, wherein the block matching step And a difference image between the reference images in the overlapping region is performed only for blocks exceeding a predetermined threshold value. The method of claim 6, The block matching step may include calculating a parallax vector for a block in which the difference image between the reference images in the overlapping region exceeds a predetermined threshold value. The image correction step includes correcting an image of a corresponding block by using the calculated parallax vector. Intermediate image synthesis method based on multiview forward camera array. The method of claim 6, wherein the block matching step A block is searched in a diagonal direction with respect to one of the three reference pictures toward the positions of the other two reference pictures. Intermediate image synthesis method based on multiview forward camera array. An apparatus for synthesizing an intermediate image from a part of a multiview image based on a multiview forward camera array, Distance matching means for checking a region where an intermediate image to be synthesized and three neighboring reference images overlap with each other; Block matching means for performing block matching on a block in which the difference image between the reference images exists in the overlapped region; Image correction means for correcting an image of the overlapping area according to the block matching result; Image synthesizing means for synthesizing the intermediate image by incorporating the corrected overlap region. Intermediate image synthesis device based on a multi-view forward camera array comprising a. The method of claim 9, wherein the image synthesis means, A multi-view forward camera for synthesizing the intermediate image by integrating the corresponding region of the reference image with the corrected overlapping region with respect to an area not included in the intermediate image determined by the distance matching means only to one reference image. Array-based intermediate image synthesis device. The method of claim 9, wherein the image synthesis means, A multi-view forward camera for synthesizing the intermediate image by integrating the corresponding region of the reference image with the corrected overlapping region with respect to an area not included in the intermediate image determined by the distance matching means only to one reference image. Array-based intermediate image synthesis device. The method of claim 11, wherein the image synthesis means An intermediate image synthesizing apparatus based on a multi-view forward camera array which performs image blending on the boundary line of the overlapped region. The method according to any one of claims 9 to 12, wherein the distance matching means, The distance HD of two parallel reference images among the three reference images is calculated by the parallel movement and the minimum mean difference error, The parallel reference image is moved by half of the calculated distance to determine the position of the intermediate image, and the distance (VD) between the intermediate image and the other reference image is perpendicular to the parallel movement and minimum average Calculated by difference error, The parallel reference image is moved by half of the calculated distance HD and the other reference image is moved by the calculated distance VD, so that the intermediate image and three neighboring reference images overlap with each other. To check the area Intermediate image synthesis device based on multiview forward camera array. 13. The method according to any one of claims 9 to 12, wherein the block matching means And a differential image between the reference images in the overlapping region is performed only for blocks exceeding a predetermined threshold. The method of claim 14, The block matching means calculates a parallax vector for a block in which the difference image between the reference images in the overlap region exceeds a predetermined threshold value, The image correcting means corrects an image of a corresponding block by using the calculated parallax vector. Intermediate image synthesis device based on multiview forward camera array. The method of claim 14, wherein the block matching means A block is searched in a diagonal direction with respect to one of the three reference pictures toward the positions of the other two reference pictures. Intermediate image synthesis device based on multiview forward camera array.

Images