WO2013157779A1

WO2013157779A1 - Image processing apparatus for determining distortion of synthetic image and method therefor

Info

Publication number: WO2013157779A1
Application number: PCT/KR2013/003056
Authority: WO
Inventors: 오병태; 이재준
Original assignee: 삼성전자주식회사
Priority date: 2012-04-16
Filing date: 2013-04-11
Publication date: 2013-10-24

Abstract

An image processing apparatus for defining a synthetic function of a synthetic image and a method therefor are disclosed. The image processing apparatus can define a distortion function of a synthetic image after re-sampling on the basis of the resolution of a depth image and a color image corresponding to the same point in time or different points in time.

Description

Image processing apparatus and method for determining distortion of a composite image

Embodiments are directed to an image processing apparatus and method for determining distortion of a composite image, and more particularly, to an image processing apparatus and method for determining distortion of a composite image in consideration of resolutions of a depth image and a color image.

The depth image refers to an image representing depth information of the color image. In the 3DV system, the depth image is additional information for generating a composite image of two or more color images. The distortion of the depth image used as reference information for compressing the depth image in the 3DV system may be determined as the distortion of the synthesized image corresponding to the intermediate view.

At this time, the composite image is generated using the color image and the depth image. However, when generating the composite image, it is necessary to consider the resolution of the color image and the depth image. In addition, it is also necessary to determine the distortion of the final composite image when the composite image corresponding to the plurality of viewpoints exists.

An image processing apparatus according to an embodiment may include a resampling unit configured to resample a color image and a depth image corresponding to a first view and a second adjacent view; And a distortion determiner configured to determine distortion of the composite image based on the color image and the depth image by using the resampled result.

When the resolution between the color image and the depth image is different from each other at the first or second view, the resampling unit may resample the same to have the same resolution.

The resampling unit may resample the resolution so that the resolution is the same when the resolution of the color image is different from each other between the first view and the second view, or the resolution of the depth image is different between the first view and the second view.

The resampling unit may resample the resolution so that the resolution is the same when the resolution between the composite image and the depth image or the color image of the intermediate view is different.

The distortion determiner may determine the area of the distortion area based on the warped pixels in the composite image as the distortion of the composite image.

According to another exemplary embodiment, an image processing apparatus includes a weight determination unit configured to determine a weight of a composite image corresponding to a plurality of intermediate viewpoints with respect to a target viewpoint of a reference image; And a distortion determiner configured to determine distortion of the final synthesized image related to the reference image by using the determined weight.

The weight determining unit may determine a weight in consideration of the number of the intermediate viewpoints and the distance between the target viewpoint and the intermediate viewpoint.

The weight determiner may determine a weight to be applied to distortion of a composite image corresponding to a plurality of intermediate views in order to determine distortion of a target view of the reference image.

The distortion determiner may determine the distortion of the final synthesized image by applying a weight to an area of the distortion area based on the warped pixels in the synthesized image.

An image processing method according to an embodiment may include resampling in consideration of resolutions of a color image and a depth image corresponding to a first view and a second adjacent view; And determining distortion of the synthesized image based on the color image and the depth image using the resampled result.

The resampling may include resampling such that the resolution is the same when the resolution between the color image and the depth image is different from each other at the first or second viewpoint.

The resampling may include resampling such that the resolution is the same when the resolution of the color image is different between the first view and the second view, or when the resolution of the depth image is different between the first view and the second view.

The resampling may include resampling such that the resolution is the same when the resolution between the composite image of the intermediate view and the input depth image or color image is different.

In the determining of the distortion of the composite image, the area of the distortion area based on the warped pixels in the composite image may be determined as the distortion of the composite image.

According to another exemplary embodiment, an image processing method includes determining a weight of a synthesized image corresponding to a plurality of intermediate viewpoints with respect to a target viewpoint of a reference image; And determining the distortion of the final synthesized image related to the reference image by using the determined weight.

In the determining of the weight, the weight may be determined in consideration of the number of the intermediate viewpoints and the distance between the target viewpoint and the intermediate viewpoint.

The determining of the weight may include determining a weight to be applied to distortion of a composite image corresponding to a plurality of intermediate views in order to determine distortion of a target view of the reference image.

The determining of the distortion of the final synthesized image may be determined as the distortion of the final synthesized image by applying a weight to an area of the distortion area based on the warped pixels in the synthesized image.

1 is a diagram illustrating an image processing apparatus according to an exemplary embodiment.

2 is a diagram illustrating an image processing apparatus according to another exemplary embodiment.

3 is a diagram illustrating a method of determining distortion of a composite image.

4 is a diagram illustrating a process of generating a composite image of an intermediate view.

5 is a diagram illustrating a distortion area of a composite image of an intermediate view.

FIG. 6 is a diagram illustrating a distortion area of a composite image of an intermediate view derived according to linear interpolation.

FIG. 7 is a diagram illustrating a process of determining distortion of a composite image of an intermediate view using a triangle.

8 is a diagram illustrating a process of determining distortion of a synthesized image when an occlusion area occurs.

FIG. 9 is a diagram illustrating a process of determining distortion of a composite image when the resolutions of the color image and the depth image are different at the same time point.

FIG. 10 is a diagram illustrating a process of determining distortion of a composite image when resolutions of a color image and a depth image are different between viewpoints.

FIG. 11 is a diagram illustrating a process of determining a distortion of a final synthesized image from a plurality of intermediate views of composite images.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The image processing apparatus 100 may include a resampling unit 101 and a distortion determiner 102.

The resampling unit 101 may resample by considering the resolutions of the color image and the depth image corresponding to the first viewpoint and the adjacent second viewpoint. In some cases, the color image may be represented by a texture applied to the 3D object. The depth image may be represented as a depth map.

For example, when the resolution between the color image and the depth image is different from each other at the first or second view, the resampling unit 101 resamples any one of the color image and the depth image such that the resolutions of the color image and the depth image are the same. can do. That is, when the color image and the depth image have different resolutions at the same time, the resampling unit 101 may resample the color image and the depth image to have the same resolution.

As another example, when the resolution of the color image is different from each other or the resolution of the depth image is different between the first view and the second view, the resampling unit 101 resamples the same resolution. can do. That is, when the color images have different resolutions at different viewpoints, the resampling unit 101 may resample the color images to have the same resolution. Similarly, when the depth images have different resolutions at different viewpoints, the resampling unit 101 may resample the resolutions of the depth images between viewpoints.

As another example, when the resolution between the composite image and the depth image or the color image of the intermediate view is different, the resampling unit 101 may resample the same resolution. That is, when the resolution between the composite image and the depth image is different or when the resolution between the composite image and the color image is different, the resampling unit 101 may resample the same resolution.

The resampling process will be described in detail with reference to FIGS. 9 and 10.

The distortion determiner 102 may determine the distortion of the synthesized image based on the color image and the depth image by using the resampled result. That is, the image processing apparatus may determine the distortion of the synthesized image after matching the resolutions of the depth image and the color image at the same time point and at different time points. As described above, the distortion of the depth image used to encode the depth image may be determined as the distortion of the composite image based on the depth image and the color image.

For example, the distortion determiner 102 may determine the area of the distortion area based on the warped pixels in the composite image as the distortion of the composite image. In detail, the distortion area may be generated by determining the actual value of the warped pixels to be different from each other due to the encoding / decoding of the color image or the encoding / decoding of the depth image.

A process of determining the distortion of the synthesized image will be described with reference to FIGS. 3 to 8.

Referring to FIG. 2, the image processing apparatus 200 may include a weight determiner 201 and a distortion determiner 202.

The weight determiner 201 may determine weights associated with the composite image corresponding to the plurality of intermediate views. For example, the weight determiner 201 may determine the weight in consideration of the number and positions of the intermediate views. In detail, the weight determiner 201 may determine, as a weight, an influence of the composite image corresponding to the plurality of intermediate views on the reference image. A process of determining the weight will be described in more detail with reference to FIG. 11.

The distortion determiner 202 may determine the distortion of the final composite image related to the reference image by using the weight.

Distortion of the composite image at the midpoint is called ground-truth as the intermediate image (V) synthesized by the warping function (f _w ) based on the original color image (C) and the original depth image (D). The difference between the color image C ′ generated through the encoding / decoding process and the intermediate image V ′ synthesized into the depth image D ′ is calculated.

The distortion of the composite image of the intermediate view may be determined according to Equation 1 below.

In Equation 1, D _syn represents a distortion of a synthesized image of an intermediate view. C denotes an original color image, D denotes an original depth image, C ′ denotes a color image generated through an encoding / decoding process, and D ′ denotes a depth image generated through an encoding / decoding process. Here, the color image C 'means a result in which the noise of the color image generated by encoding or decoding is included in the original color image C, and the depth image D' is generated by encoding or decoding in the original depth image D. It means the result that includes the noise of the depth image. In addition, V means an intermediate image obtained by combining the color image C and the depth image D with the warping function f _w , and V 'means an intermediate image obtained by combining the color image C' and the depth image D 'with the warping function f _w . .

As shown in Equation 1, the disparity error of the actual depth image represents a linear relationship with the depth error of the depth image. The shift error of the depth image is determined by Equation 2 below.

In equation (2)

Denotes a shift error of the depth image, and ΔDepth denotes a depth error of the depth image caused by noise generated during the encoding / decoding process. a denotes a parameter determined by a camera setting value, and δx denotes a distance between a currently encoded view and a view of a composite image.

Accordingly, the distortion of the composite image of the intermediate view is expressed by Equation 3 below.

D _syn denotes a distortion of the synthesized image of an intermediate view, C denotes an original color image, and C ′ denotes a color image including noise of a color image generated by encoding or decoding in the original color image C. C (x, y) means pixels of the original color image. Denotes a shift error of the depth image, and p denotes an arbitrary value.

3 shows Equation 3 in a simpler form. Here, the pixel C (x, y) of the arbitrary color image should be moved to the position of C (x ', y) in the composite image of the intermediate view by the depth value D (x, y). However, the depth value D (x, y) is changed to the depth value D '(x, y) due to an error generated through the encoding / decoding process. Accordingly, the pixel C (x, y) of the color image is actually (x '+ ΔD (x, y), y) rather than (x', y) due to an error of the depth value D (x, y). Move. Here, ΔD (x, y) means the difference between D (x, y) and D '(x, y).

Therefore, Equation 3 represents the pixel C (x, y) of the color image moving to C (x ', y) and the pixel C' of the color image moving to the position (x '+ ΔD (x, y), y). The difference between (x + ΔD (x, y), y) is defined as the distortion of the composite image at the intermediate view.

Referring to FIG. 4, when there are pixels C ₁ ... C _n of a color image of one row and pixels D ₁ ... D _n of a depth image, pixel C of each color image _k is warped by the pixel D _k of the corresponding depth image.

After warping, each pixel may not exactly match the integer grid indicated by the dotted line in the actual midpoint composite image. Referring to FIG. 4, the values reflected in the actual integer grid are interpolated values.

If, due to the encoding / decoding process, it is assumed that pixels C ' ₁ ... C' _n of a color image are changed to pixels D ' ₁ ... D' _n of a depth image. As shown in FIG. 5, values that should be actually warped may be warped to other values due to an error generated by an encoding / decoding process.

In this case, an error, ΔC _k = C _k − C ′ _k , generated during the encoding / decoding process of the color image from the warped values changes the pixel values in the y-axis direction. In addition, an error generated through the encoding / decoding process of the depth image, that is, ΔD _k = D _k −D ′ _k , changes pixel values in the x-axis direction.

Referring to FIG. 5, the curve A (x) interpolated after warping through the original image information is changed to the curve B (x) due to an error generated during the encoding / decoding process. According to an embodiment of the present invention, an area existing between curves A (x) and B (x) may be defined as a distortion area.

In FIG. 5, the distortion of the composite image of the intermediate view is determined according to Equation 4 below.

In Equation 4, D _syn denotes a distortion of a synthesized image of an intermediate view. That is, according to Equation 4, when the interpolated curve A (x) is changed to a curve B (x) reflecting an error based on encoding or decoding after the original image is warped, the distortion of the composite image of the intermediate view is curved A ( The area of the distortion area existing between x) and the curve B (x) is determined.

6 shows the result of simplifying FIG. 5 according to an interpolation method. Then, as illustrated in FIG. 6, the area of the distortion area of the synthesized image of the intermediate view may be more simply calculated by using a straight line that simplifies the curve according to the interpolation method.

In FIG. 6, a linear interpolation scheme is applied to obtain the area of the distortion region. Then, the curve shown in FIG. 5 becomes a straight line connecting two points by linear interpolation as shown in FIG.

In addition, according to an embodiment of the present invention, a distance value per pixel may be calculated, squared and added without processing several pixel values constituting the curve at once.

Referring to FIG. 6, an error generated when the current pixel value (C _k , D _k ) is changed to the value (C ′ _k , D ′ _k ) due to encoding or decoding is considered. That is, the region shown in FIG. 6 occurs due to an error generated in the pixel at the k position. In this case, as shown in FIG. 6, linear interpolation is applied to the curve so that an error of a pixel in which a straight line exists at a specific position affects pixels located around the specific position. That is, if an error occurs in the pixel at the k position, it is influenced by the pixel values existing at the (k-1) position and the (k + 1) position.

In this case, the distortion caused by the pixel at the k position may be determined by the area of the distortion region generated due to the error of the pixel at the k position.

That is, the distortion based on the pixel at the k position in Figure 6 is determined by the following equation (5).

In Equation 5, _{ΔD syn} denotes a distortion generated due to an error of a pixel at a k position, and k, k−1, and k + 1 denote a pixel position. A (K) represents the value of the k position in the linear interpolated curve A (X), and B (K) represents the value of the k position in the linear interpolated curve B (X).

Accordingly, the final distortion of all the pixels present in the linear interpolated curve is determined by Equation 6 below.

According to Equation 6, the final distortion D _syn can be derived by adding all the DELTA D _syn generated due to the error of each pixel present in the linear interpolated curve. p means any number.

FIG. 7 illustrates a simpler process of calculating Equation 5 described above. If only the decoded color image C 'is used without using the original color image C, the error is only in the y-axis direction as shown in FIG.

Is generated. As a result, the area of the distortion area caused by the error of the pixel at the k position may be determined by the area of a triangle composed of pixels at k, k-1, and k + 1 positions constituting the decoded color image C ′.

Accordingly, the area of the distortion region of FIG. 7 may be represented by Equation 7 below.

In Equation 7, D _syn means distortion generated due to an error of a pixel at a k position in FIG. 7. C 'represents a decoded color image. DELTA D can be derived from Equation 2.

In Equation 7, the decoded color image is used when calculating the distortion based on the distortion region. The decoded color image may be replaced with other similar information. As an example. The original color image may be used instead of the decoded color image. As another example, a color image in which a specific filter is applied to the original color image may be used instead of the decoded color image. In this case, the filter and the filter parameter may be more accurately determined based on the relationship between the original color image of the adjacent view and the decoded color image.

As described above, it is assumed that the peripheral pixels exist at positions k-1 and k + 1 with respect to the current pixel existing at position k in the curve B (x). However, in the actual warping process, it is necessary to determine whether the surrounding pixels at the positions k-1 and k + 1 are the actual surrounding pixels of the current pixel at the position k based on the pixel of the color image.

Referring to FIG. 8, the pixel value of the color image which is influenced next after the current pixel C (x, y) through a warping process is not C (x + 1, y) but C (x + 3, y). ) According to an embodiment of the present invention, the distortion region may be expressed according to Equation 8 in consideration of the problems caused by the occlusion region or the hole region after the actual warping.

In Equation 8, D _syn means distortion generated due to an error of a pixel at a k position in FIG. 8. C 'represents a decoded color image. DELTA D can be derived from Equation 2.

Here, k _L and k _R are determined according to the following equation (9).

In Equation 9, k _L represents a position (k-1 to kl) of a pixel existing on the left side of the k position when an error occurs in the pixel at the k position. K _R represents a position (k + 1 to k + r) of the pixel existing on the right side with respect to the k position.

In Equation 9, the condition that the distortion area becomes 0 is when the value of the current pixel is located in the occlusion area after warping. In this case, since the value of k _L is greater than 0, the value of the distortion function of the synthesized image may be set to 0.

The above-described mid-view synthesis distortion (VSD) may be used as a replacement or mixed with conventional distortion of sum of squared distortion (SSD) or sum of absolute distortion (SAD). In one example, the distortion finally used may be determined according to the following equation (10).

In Equation 10, Dist denotes a distortion finally used, VSD denotes a distortion of a synthesis region at an intermediate time point, and SSD denotes a conventional distortion. ω means a weight greater than 0 and less than 1.

Also, λ _{V, which} is a Lagrangian multiplier applied to the rate-distortion cost function based on VSD, is different from λ _D , which is a Lagrangian multiplier applied to the rate-distortion cost function based on SSD. Can be set. In this case, when the final rate-distortion cost function is derived, the VSD-based cost function and the SSD-based cost function can be considered simultaneously.

For example, the final rate-distortion cost function J may be expressed as a weighted sum of the SSD-based cost function J _D and the VSD-based cost function J _V , as shown in Equation 11 below.

In Equation 11, J means a rate-distortion cost function, ω means a weight greater than 0 and less than 1, and may be provided as a configuration of an encoding apparatus. In addition, VSD (View Synthesis Distortion) means distortion of the synthesis region corresponding to the intermediate view, and SSD (Sum of Squared Distortion) means conventional distortion. Here, the SSD may be determined by summing the square of the difference between the original depth block and the encoded depth block. In addition, λ _V denotes a Lagrange multiplier applied to the VSD-based rate-distortion cost function, and λ _D denotes a Lagrange multiplier applied to the SSD-based rate-distortion cost function. Λ _V can be adjusted because VSD takes a different distortion domain. Referring to Equation 11, it can be seen that the depth distortion in the high texture area is generally more harmful to the rate-distortion cost function than the distortion of the depth image in the low texture area.

The rate-distortion cost function obtained in Equation 11 is changed according to the viewpoint of the synthesized image. If there are two or more viewpoints of the composite image, the rate-distortion cost function is finally determined by the weighted sum of the cost functions obtained at each of the viewpoints of the composite image.

Coding distortion in the depth image may not linearly affect the distortion of the synthesized image. The influence of the distortion of the depth image may vary according to the information of the corresponding color image. For example, the same depth distortion in the region where color is present and the region where color is not present may cause distortion of different composite images. In order to efficiently optimize rate-distortion of depth-based 3D image coding, a new type of distortion function for measuring view distortion of a synthesized image by analyzing view warping or view rendering is required.

According to an example, the distortion of the composite image may be determined by Equation 12. Here, the synthesized image is synthesized from images of different viewpoints, and the distortion of the synthesized image may be expressed as view synthesis distortion (VSD).

Here, C means an original color image, and D means an original depth image.

Means a reconstructed color image. For example,

May include a coded or decoded color image. Also,

Denotes a reconstructed depth image. For example,

Denotes a coded or decoded depth image. x, y represents the pixel position in the macro block.

Denotes a proportional coefficient and may be determined by a 3D camera parameter.

May be determined according to Equation 13.

In Equation 13, f denotes a focal length of a camera, and L denotes a baseline between a current view and a rendered view. Z _near means the depth closest to the scene, and Z _far means the depth farthest from the scene.

On the other hand, if the reconstructed color image is not available, the original color image may be used instead of the reconstructed color image. For example, when the depth image is encoded before the color image, it means that the reconstructed color image is not available. Then, Equation 12 may be changed to Equation 14 below.

here,

Denotes the ratio between the original color image and the decoded color image at view I, which is the base view.

May be determined according to Equation 15 below.

Here, C ^I means a color image at view ^{I, which} is a basic view.

Equations 12 to 15 are determined assuming that the resolutions of the depth image and the color image are the same. If the resolutions of the depth image and the color image are different from each other, it is necessary to adjust the resolutions of the depth image and the color image in order to apply the equations (12) to (15). For example, if the resolution of the depth image is 1/2 of the resolution of the color image, it is necessary to adjust the resolution so that the resolution of the depth image is the same as the resolution of the color image.

In other words, if the resolutions of the depth image and the color image are different from each other, the image processing apparatus may perform a resolution mapping process. That is, resolution mapping refers to a process of finding a pixel of a color image corresponding to a pixel of a current depth image. That is, when the resolution of the depth image is W * H and the resolution of the color image is W '* H' = pW * qH, the pixel (x, y) of the depth image is (x ', y') = in the color image. (Round (px), Round (qy)). In this case, Equation 12 becomes Equation 16.

According to an embodiment, the image processing apparatus may make the resolutions of the depth image and the color image the same through resampling. As mentioned above, if the resolution of the depth image is 1/2 of the resolution of the color image, it is necessary to increase the resolution by upsampling the depth image. Then, the image processing apparatus may determine the distortion of the composite image by using Equation 12 instead of the modified Equation 16. In this case, the image processing apparatus may compress the depth image and then resample it in the process of compositing the color image. Resampling may be performed through image interpolation.

For example, the resolution of the depth image may be lower than that of the color image. Therefore, the image processing apparatus may perform resampling for each block in the depth image such that the resolution of the color image and the resolution of the depth image are the same.

Referring to FIG. 9, the color image C1 902 and the depth image D1 903 have different resolutions at a time point 1. Similarly, at time 2, the color image C2 904 and the depth image D2 905 have different resolutions. Thus, the image processing apparatus may resample the depth image D1 903 such that the resolution of the depth image D1 903 is the same as the resolution of the color image C1 902. Similarly, the image processing apparatus may resample the depth image D2 905 such that the resolution of the depth image D2 905 is the same as the resolution of the color image C2 904. The synthesized image 901 of the intermediate view is generated according to the resampled result, and the image processing apparatus may determine the distortion of the synthesized image 901.

Referring to FIG. 10, the color image C1 1002 and the depth image D1 1003 have the same resolution at a time point 1. However, the color image C1 1002 of the viewpoint 1 and the color image C2 1004 of the viewpoint 2 are different from each other in resolution. The depth image D1 1003 of the viewpoint 1 and the depth image D2 1005 of the viewpoint 2 are different from each other in resolution.

Then, in order to generate the composite image 1001 of the intermediate view, the image processing apparatus may resample the color image C1 1002 such that the resolution of the color image C1 1002 and the resolution of the color image C2 1004 are the same. . Also, the image processing apparatus may resample the depth image D1 1003 such that the resolution of the depth image D1 1003 and the resolution of the depth image D2 1005 are the same.

If the resolution of the composite image 1001 is different from the resolution of the color image C1 1002 and the depth image D1 1003, the resolution of the composite image 1001 may be different from that of the color image C2 1004 and the depth image D2 1005. In this case, the image processing apparatus may resample the color image C1 1002, the depth image D1 1003, the color image C2 1004, and the depth image D2 1005 to have the same resolution as that of the composite image 1001. have.

In FIG. 11, it is assumed that synthetic images V1, V2, and V3 1102 to 1104 corresponding to a plurality of intermediate views exist between the input reference image C1 1101 and the reference image C2 1105. In order to derive the final synthesized image based on the reference image C1 1101 and the reference image C2 1105, the image processing apparatus may combine the synthesized images V1, V2, and V3 1102 to 1104 corresponding to a plurality of intermediate viewpoints. It is available.

In Equation 12, VSD means distortion of a composite image corresponding to one viewpoint. If there are composite images corresponding to N intermediate views, the distortion of the final composite image may be determined according to Equation 17 below.

For Equation 17 to be applied, VSD _{i, which} is a distortion of the synthesized image at each intermediate view point i, must be considered. In one example, Equation 17 is an average of the composite distortion VSD _i at each intermediate time i. Here, W _i denotes a weight of the intermediate view i on the final synthesized image. That is, when generating the final synthesized image for the reference image, W _i represents the contribution of the synthesized image corresponding to the intermediate view i to the final synthesized image. N represents the number of synthesized images of intermediate views. For example, W _i may mean a weight indicating how much the current view i contributes to the rendered target view point. In this case, the closer the current time i is to the target time, the larger W _i may be.

In FIG. 11, a viewpoint of the reference image C1 1101 may be defined as a target viewpoint. Then, the distortion at the target viewpoint may be determined as the distortion of the composite image of the plurality of intermediate viewpoints adjacent to the target viewpoint. Here, an intermediate viewpoint adjacent to the target viewpoint corresponding to the reference image C1 1101 means a viewpoint of the synthesized image V1 1102, the synthesized image V2 1103, and the synthesized image V3 1104. As an example, the distortion at the target viewpoint may be determined by applying weights to the plurality of composite images.

In this case, the weight of each of the intermediate images of the intermediate view with respect to the distortion at the target view may increase as the target view is closer. In FIG. 11, the composite image V1 1102 is positioned at 1/4 between the reference image C1 1101 and the reference image C2 1102. The synthesized image V2 1103 is located at 1/2 between the reference image C1 1101 and the reference image C2 1102. In addition, the composite image V3 1104 is positioned at 3/4 between the reference image C1 1101 and the reference image C2 1102. Then, the weight of the composite image V1 1102 is 3/4 with respect to the distortion of the target viewpoint. The weight of the composite image V2 1103 is 2/4 and the weight of the composite image V3 1104 is 1/4 with respect to the distortion of the target viewpoint.

Then, the distortion of the final composite image related to the reference image C1 1101 may be determined as in Equation 18 below.

However, according to equations (17) and (18), the VSD needs to be repeatedly calculated according to the number of intermediate views. Thus, the image processing apparatus may determine the distortion of the final composite image in a simple manner according to Equation 19 below.

That is, according to Equation 19, since the rest except for the α parameter is the same, the final distortion of the final synthesized image may be determined using only one operation without N VSD operations.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims

A resampling unit for resampling in consideration of resolutions of a color image and a depth image corresponding to a first view and a second adjacent view; And

Distortion determiner for determining distortion of the composite image based on the color image and the depth image using the resampled result

Image processing apparatus comprising a.
The method of claim 1,

The resampling unit,

And resampling the resolution so that the resolution is the same when the resolution is different between the color image and the depth image at the first view or the second view.
The method of claim 1,

The resampling unit,

And resampling the resolution so that the resolution is the same when the resolution of the color image is different between the first viewpoint and the second viewpoint or when the depth image has a different resolution between the first viewpoint and the second viewpoint.
The method of claim 1,

The resampling unit,

And resampling the resolution so that the resolution is the same when the resolution between the composite image and the depth image or the color image of the intermediate view is different.
The method of claim 1,

The distortion determining unit,

And an area of a distortion area based on a warped pixel in the synthesized image as a distortion of the synthesized image.
A weight determination unit configured to determine weights of the synthesized images corresponding to the plurality of intermediate views with respect to the target viewpoint of the reference image; And

Distortion determiner for determining the distortion of the final composite image associated with the reference image by using the determined weight

Image processing apparatus comprising a.
The method of claim 6,

The weight determination unit,

And determining a weight in consideration of the number of the intermediate viewpoints and the distance between a target viewpoint and the intermediate viewpoint.
The method of claim 7, wherein

The weight determination unit,

And determining a weight to be applied to distortion of a composite image corresponding to a plurality of intermediate views to determine distortion of a target viewpoint of the reference image.
The method of claim 6,

The distortion determining unit,

And determining a final distortion of the composite image by applying a weight to an area of the distortion region based on the warped pixels in the composite image.
Resampling in consideration of resolutions of the color image and the depth image corresponding to the first viewpoint and the adjacent second viewpoint; And

Determining distortion of the composite image based on the color image and the depth image using the resampled result

Image processing method comprising a.
The method of claim 10,

Resampling is,

And resampling the resolution so that the resolution is the same when the resolution is different between the color image and the depth image at the first or second viewpoint.
The method of claim 10,

Resampling is,

And resampling the resolution so that the resolution is the same when the resolution of the color image is different between the first view and the second view or when the resolution of the depth image is different between the first view and the second view.
The method of claim 10,

Resampling is,

And resampling the resolution so that the resolution is the same when the resolution between the composite image and the input depth image or the color image of the intermediate view is different.
The method of claim 10,

Determining the distortion of the composite image,

And determining an area of a distortion area based on a warped pixel in the synthesized image as a distortion of the synthesized image.
Determining a weight of the synthesized image corresponding to the plurality of intermediate views with respect to the target view of the reference image; And

Determining distortion of the final synthesized image related to the reference image by using the determined weights

Image processing method comprising a.
The method of claim 15,

Determining the weight,

And determining a weight in consideration of the number of the intermediate viewpoints and the distance between a target viewpoint and the intermediate viewpoint.
The method of claim 16,

Determining the weight,

And determining a weight to be applied to distortion of a composite image corresponding to a plurality of intermediate views to determine distortion of a target viewpoint of the reference image.
The method of claim 15,

Determining the distortion of the final composite image,

And determining a final distortion of the composite image by applying a weight to an area of a distortion region based on the warped pixels in the composite image.