JP5940812B2 - Multi-viewpoint image processing apparatus, method, and program - Google Patents

Description

  The present invention relates to a multi-viewpoint image processing apparatus, method, and program for improving the image quality of a low-quality image at a certain viewpoint among a plurality of viewpoint images (multi-viewpoint images).

  2. Description of the Related Art Conventionally, a multi-view image processing system that captures a single object from a plurality of viewpoints and transmits a multi-view image obtained thereby is known. When multi-viewpoint images (including two-viewpoint images) are used in 3D television broadcasting or the like, an image from one viewpoint is transmitted with high image quality, and an image from another viewpoint is transmitted with lower image quality. According to this multi-viewpoint image processing system, the transmission band can be reduced, and efficient multi-viewpoint image transmission can be realized.

  Even if there is a difference in image quality between the multi-viewpoint images transmitted by such a multi-viewpoint image processing system, the viewer can perceive the parallax and thus can feel a stereoscopic effect in the presented image. In addition, since human visual characteristics tend to recognize fine patterns depending on higher-quality images, viewers can perceive images as subjective image quality closer to high-quality images.

  Incidentally, in the technical field of image processing, a technique for improving image quality is known. For example, as a technique for improving the image quality using a plurality of images, there is a plurality of super-resolution (Patent Document 1). This multi-frame super-resolution technique assumes a high-resolution image, estimates a pixel value for each pixel of the low-resolution image using a predetermined function from the assumed high-resolution image, and observes the estimated pixel value. The high-resolution image is searched so that the difference between the pixel value is minimized.

  For example, a texture super-resolution technique using a three-dimensional model of an object has been proposed (Patent Document 2). This super-resolution technique generates a subject image under a pseudo light source having a high quality and an arbitrary position from a low-quality image. Specifically, the apparatus for realizing the super-resolution technique calculates the first geometric parameter related to the shape of the subject from the positional information of the light source and the viewpoint and the normal information representing the surface shape of the subject, and the normal information space Spatial expansion is performed on an expression that is a geometric parameter having a higher resolution component than the general resolution, the first geometric parameter is corrected using the expanded expression, and the corrected first geometric parameter is converted into a reflection model. By applying this, a high-quality image is generated.

  In addition, for example, a multi-view image encoding method that performs encoding by regarding the parallax between viewpoints as a motion vector has been proposed (Patent Document 3). In this multi-viewpoint image encoding method, when a stereoscopic image compression apparatus multiplexes and compresses a multi-viewpoint image constituting a stereoscopic image into one image, the motion vector is decomposed into an offset vector and a local motion vector, Insert into the video stream. Then, when the stereoscopic image decompression apparatus receives and decompresses the moving image stream, the stereoscopic image decompression device combines the offset vector and the local motion vector into one motion vector, and restores the stereoscopic image using the combined motion vector.

JP 2006-127241 A Special table 2011-522316 gazette JP 2005-110113 A

  However, in conventional 3D televisions with different image quality depending on the viewpoint, viewers feel tired and intoxicated in their eyes and mind, feel the difference in image quality when one eye is closed, or feel image quality deterioration like blurring. There was a problem.

  In addition, the multiple-resolution super-resolution technique described in Patent Document 1 described above generates a high-resolution image from a plurality of low-resolution image sequences, and actively uses a case where there is a difference in image quality between images. Is not considered. In addition, since the multi-image super-resolution technique is premised on that aliasing distortion exists in the low-resolution image sequence, it does not operate effectively when aliasing distortion is not added at the time of imaging.

  Further, the texture super-resolution technique using the three-dimensional model described in Patent Document 2 described above requires construction of a three-dimensional model and rendering for each viewpoint in order to apply to a multi-view image. , Hardware and operations become extremely large.

  In addition, in the method of performing encoding by regarding the parallax described in Patent Document 3 as a motion vector, encoding is performed by regarding an image sequence with different viewpoints as a moving image, so that the resolution differs for each viewpoint. The case is not considered.

  As described above, Patent Documents 1 to 3 do not mention any technique for improving the image quality of a low-quality image in a multi-viewpoint image having a difference in image quality. If the image quality of a low-quality image can be improved, problems such as eye fatigue and sickness in a conventional 3D television with different image quality depending on the viewpoint can be solved.

  Accordingly, the present invention has been made to solve the above-described problems, and an object of the present invention is to change a low-quality image (low-quality viewpoint image) from one viewpoint to a high-quality image (high-quality viewpoint image) from another viewpoint. ) Based multi-viewpoint image processing apparatus, method and program.

In order to solve the above-described problem, a multi-viewpoint image processing apparatus according to claim 1 of the present invention is a multi-viewpoint image processing apparatus that uses a high-quality viewpoint image to improve the image quality of a low-quality viewpoint image. Based on the parallax estimated by the parallax estimator and the parallax estimated by the parallax estimator, performing the association for each partial region between the high-quality viewpoint image and the low-quality viewpoint image, Image correction means for performing parallax compensation on the high-quality viewpoint image, and generating an image with improved image quality of the low-quality viewpoint image using the parallax-compensated high-quality viewpoint image, The image correction unit extracts a reinforcement signal image from the high-quality viewpoint image, and performs parallax compensation for each of the partial regions on the reinforcement signal image based on the parallax estimated by the parallax estimation unit; For each partial area, The low quality of the viewpoint images, and superimposes the reinforcing signal image parallax compensation corresponding, the reinforcing signal image is an image of the spatial high-frequency component, and wherein the. According to such a configuration, when the high-viewpoint image and the low-quality viewpoint image are included in the multi-viewpoint image, the image quality of the low-quality viewpoint image can be improved. For example, even in a system in which the transmission bandwidth can be reduced and a difference in image quality occurs between transmission images for the left eye and the right eye, the multi-view image processing apparatus of the present invention can reduce the image quality of a low-quality viewpoint image. This can improve the image quality difference between the eyes. Further, according to such a configuration, since the high frequency information to be interpolated is added while the information of the low-quality viewpoint image remains, the image quality of the low-quality viewpoint image is retained while maintaining the information viewed from the actual viewpoint. Can be improved.

A multi-viewpoint image processing apparatus according to claim 2 is a multi-viewpoint image processing apparatus that uses a high-quality viewpoint image to improve the image quality of a low-quality viewpoint image, wherein the high-quality viewpoint image and the low-quality viewpoint image Parallax estimation means for associating each partial area with each viewpoint image and estimating the parallax, and parallax compensation for the high-quality viewpoint image based on the parallax estimated by the parallax estimation means Image correction means for generating an image with improved image quality of the low-quality viewpoint image using the high-quality viewpoint image compensated for parallax, and the image correction means includes the high-quality viewpoint A reinforcement signal image is extracted from the image, and based on the parallax estimated by the parallax estimation means, parallax compensation is performed on the reinforcement signal image for each partial area, and the low-quality viewpoint is obtained for each partial area. The image corresponding to the view Superimposing a reinforcing signal image compensation, the reinforcing signal image is an image of a texture component, wherein the. According to such a configuration, only the non-contour vibration component is added to the low-quality viewpoint image, and thus the image quality of the low-quality viewpoint image is improved while preventing artifacts such as ringing in the edge portion. Can do.

The multi-view image processing apparatus according to claim 3 is the multi-view image processing apparatus according to claim 2 , wherein the image correction means extracts a texture component from the high-quality viewpoint image using a total variation filter. It is characterized by that. According to such a configuration, since the total variation norm that is easy to calculate is regularized, it is possible to improve the image quality of the low-quality viewpoint image while preventing artifacts such as ringing of the edge portion at a low calculation cost.

A multi-viewpoint image processing apparatus according to claim 4 is a multi-viewpoint image processing apparatus that uses a high-quality viewpoint image to improve the image quality of a low-quality viewpoint image, wherein the high-quality viewpoint image and the low-quality viewpoint image Parallax estimation means for associating each partial area with each viewpoint image and estimating the parallax, and parallax compensation for the high-quality viewpoint image based on the parallax estimated by the parallax estimation means Image correction means for generating an image with improved image quality of the low-quality viewpoint image using the high-quality viewpoint image compensated for parallax, and the image correction means includes the high-quality viewpoint A reinforcement signal image is extracted from the image, and based on the parallax estimated by the parallax estimation means, parallax compensation is performed on the reinforcement signal image for each partial area, and the low-quality viewpoint is obtained for each partial area. The image corresponding to the view Superimposing a reinforcing signal image compensation, the reinforcing signal image is an image of a color difference component, and wherein the. According to such a configuration, the color difference to be interpolated is added while leaving the information of the low-quality viewpoint image, so that the image quality of the low-quality viewpoint image is improved while retaining the information viewed from the actual viewpoint. Can do.

A multi-viewpoint image processing method according to claim 5 is a multi-viewpoint image processing method for improving the image quality of a low-quality viewpoint image using a high-quality viewpoint image, wherein the high-quality viewpoint image and the low-quality viewpoint image A first step of inputting the viewpoint image, a second step of performing an association for each partial area between the high-quality viewpoint image and the low-quality viewpoint image, and estimating a parallax; First, a parallax compensation is performed on the high-quality viewpoint image based on the estimated parallax, and an image in which the image quality of the low-quality viewpoint image is improved is generated using the parallax-compensated high-quality viewpoint image. And the third step extracts a reinforcement signal image from the high-quality viewpoint image, and for each partial region with respect to the reinforcement signal image based on the estimated parallax Parallax compensation is performed for each partial area. Serial to low quality of the viewpoint images, and superimposes the reinforcing signal image parallax compensation corresponding, the reinforcing signal images, high spatial frequency components of the image, is an image of the image or the color difference component of the texture component, and wherein the .

A multi-viewpoint image processing program according to a sixth aspect causes a computer to function as the multi-viewpoint image processing apparatus according to any one of the first to fourth aspects.

  As described above, according to the present invention, a low-quality image from one viewpoint can be refined based on a high-quality image from another viewpoint, and the image quality of the low-quality image can be improved.

It is a block diagram which shows the structural example of the multiview image processing apparatus by embodiment of this invention. It is a flowchart which shows the process of a multiview image processing apparatus. FIG. 3 is a block diagram illustrating a configuration example of an image correction unit according to the first embodiment. 3 is a flowchart illustrating processing of an image correction unit according to the first exemplary embodiment. FIG. 6 is a block diagram illustrating a configuration example of an image correction unit according to a second embodiment. 6 is a flowchart illustrating processing of an image correction unit according to the second exemplary embodiment. FIG. 10 is a block diagram illustrating a configuration example of an image correction unit according to a third embodiment. FIG. 10 is a block diagram illustrating a configuration example of an image correction unit according to a fourth embodiment.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
[Multi-viewpoint image processing device]
First, the configuration of a multi-viewpoint image processing apparatus according to an embodiment of the present invention will be described. FIG. 1 is a block diagram illustrating a configuration example of a multi-view image processing apparatus, and FIG. 2 is a flowchart illustrating processing of the multi-view image processing apparatus. The multi-viewpoint image processing apparatus 1 includes a parallax estimation unit 10 and an image correction unit 20. The multi-viewpoint image processing apparatus 1 inputs a high-quality image shot from one viewpoint (viewpoint 1) and a low-quality image shot from another viewpoint (viewpoint 2). The image quality of the image quality image is improved and output as an improved image of viewpoint 2.

  Note that the multi-viewpoint image processing apparatus 1 also inputs a low-quality image taken from another viewpoint (viewpoint V; V is an integer of 3 or more), improves the low-quality image of the viewpoint V, and improves the viewpoint V. You may make it output as an image. In this case, the multi-viewpoint image processing apparatus 1 may have a configuration in which a low-quality image of “viewpoint 2” described later is replaced with a low-quality image of “viewpoint V”.

  Further, the high-quality image and the low-quality image may be images taken with cameras or lenses having different resolutions or focal lengths of the imaging lenses. Assume that the association (calibration) between the high-quality image and the low-quality image in consideration of the origin, magnification, and aberration is performed by the low-quality image calibration means not shown in FIG. In this calibration, when the optical axis and the front optical principal point of the lens that shoots a high-quality image and the lens that shoots a low-quality image are matched, the position and size of the subject on both images match. It means the association between image coordinates.

  The high-quality image and the low-quality image may be those quantized by different quantization methods, for example, those quantized with different bit depths (quantization bit numbers). It is assumed that the process of matching the quantization method for the high-quality image and the low-quality image is also performed by the low-quality image calibration unit not shown in FIG.

  Hereinafter, let L be a low-quality image, H be a high-quality image, and b be a function that matches the quantization dynamic range and characteristic curve of the low-quality image L to the quantization dynamic range and characteristic curve of the high-quality image H. To do. The pixel value of the image coordinate (P) (P is a two-dimensional vector) in the high-quality image H is H (P), and the pixel value of the image coordinate (p) (p is a two-dimensional vector) in the low-quality image L is L (P).

Further, the association from the image coordinates P on the high-quality image H to the image coordinates p on the low-quality image L by the calibration is represented by a vector function c as in the following equation.

  When the multi-viewpoint image processing apparatus 1 inputs the high-quality image H of the viewpoint 1 and the low-quality image L of the viewpoint 2 (step S201), the low-quality image calibration unit (not shown) calibrates the low-quality image L of the viewpoint 2. To match the image coordinate system of the high-quality image H, and match the low-quality image L that matches the image coordinate system of the high-quality image H to the quantization characteristics of the high-quality image H to generate a calibrated low-quality image M (Step S202). The calibrated low image quality image M is output to the parallax estimation means 10 and the image correction means 20. For example, the low-quality image calibration unit matches the resolution of the low-quality image L with the resolution of the high-quality image H, and matches the bit depth of the low-quality image L with the bit depth of the high-quality image H. Although the image coordinate system of the calibrated low-quality image M matches the image coordinate system of the high-quality image H by the low-quality image proofing means, there is a parallax between the two images.

  Note that low-quality image calibration means (not shown) may be included in the parallax estimation means 10 and the image correction means 20. In this case, the parallax estimation unit 10 and the image correction unit 20 input the low quality image L of the viewpoint 2 and generate a calibrated low quality image M. Further, when the image coordinate system and the quantization characteristics match between the low image quality image L and the high image quality image H, the processing by the low image quality image calibrating means is unnecessary.

Here, the calibrated low-quality image M, which is an image in which the low-quality image L is matched with the image coordinate system of the high-quality image H by calibration and matched with the quantization characteristics of the high-quality image H, It is expressed by an expression.

  In the calculation of the above formula (2), when c (P) is a non-lattice point (one or both of horizontal coordinates and vertical coordinates are non-integer points), the multi-viewpoint image processing apparatus 1 Is a pixel value L (c (P) based on pixel values on lattice points near c (P) in the low-quality image L by interpolation (for example, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.). ) Are determined, the characteristics of the quantization are compensated by the function b, and b (L (c (P))) is obtained to obtain a calibrated low-quality image M. In this case, the multi-viewpoint image processing apparatus 1 calculates pixel values on lattice points in the vicinity of c (P) in the low-quality image L, compensates the quantization characteristics by the function b, and performs an interpolation operation. The result may be regarded as b (L (c (P))), and a calibrated low-quality image M may be obtained.

Further, the multi-viewpoint image processing apparatus 1 performs an interpolation operation (functional F; an image F [L] is an image of a continuous domain in which pixel values at decimal pixel positions are defined by the interpolation operation) as in the following expression. After determining the pixel value of the decimal pixel position, the pixel value of the image coordinate c (P) may be obtained to obtain a calibrated low-quality image M.

ｆ（ｐ）は、例えば、Ｓｉｎｃ関数、Ｌａｎｃｚｏｓ関数に基づき定義することができる。 For example, the interpolation operation F can be performed by convolution of filter coefficients whose impulse response is f (p), as in the following equation.

f (p) can be defined based on, for example, a Sinc function or a Lanczos function.

  The parallax estimation means 10 inputs the high-quality image H of the viewpoint 1 and also inputs the calibrated low-quality image M of the viewpoint 2 from the low-quality image calibration means (not shown), and the high-quality image H and the calibrated low-quality image M Are matched for each partial region by pattern matching between and, the parallax is estimated, and a parallax vector (parallax map D) is generated (step S203). The disparity vector is output to the image correction unit 20. The association by pattern matching between the high-quality image H and the calibrated low-quality image M is also the association by pattern matching between the high-quality image H and the low-quality image L.

ここで、視差マップＤ（Ｐ）は、マッチングした部分領域を基準にした、高画質画像Ｈの画像座標Ｐと校正済み低画質画像Ｍの画像座標Ｐとの間の差をいい、画像座標Ｐにおいて、その差の大きさ及び方向を示すべクトル（視差ベクトル）で表したものをいう。Ｂは部分領域（ブロック領域）、Ｓは探索領域を表す。 For example, the parallax estimation means 10 uses the block matching method to obtain a portion between the pixel value H (P) of the high-quality image H and the pixel value M (P) of the calibrated low-quality image M as shown in the following equation. Matching for each region is performed, parallax is estimated, and a parallax map D (P) is generated.

Here, the parallax map D (P) refers to the difference between the image coordinates P of the high-quality image H and the image coordinates P of the calibrated low-quality image M with reference to the matched partial region. 1, the vector is a vector (parallax vector) indicating the magnitude and direction of the difference. B represents a partial area (block area), and S represents a search area.

  The equation (5) is used to obtain the minimization result of the sum of absolute values (SAD value; Sum of Absolute Difference) between the high-quality image H and the calibrated low-quality image M as the parallax. Instead of this, the result of minimizing the sum of squares of the difference (SSD value; Sum of Squared Difference) or the result of maximizing the normalized cross-correlation may be obtained as the parallax. Further, weighting (for example, weighting monotonously decreasing in accordance with the distance from the block center) may be performed in the block of the partial area. Alternatively, the parallax may be obtained only by the thinned image coordinates without obtaining the parallax for all the image coordinates P, and the parallax may be defined by interpolation processing for the thinned image coordinates. Prior to the parallax estimation, filter processing (for example, a Gaussian filter) may be applied to either or both of the high-quality image H and the calibrated low-quality image M. Thereby, the influence of noise can be suppressed and parallax can be obtained robustly.

  The image correction unit 20 inputs the high-quality image H of the viewpoint 1, receives the calibrated low-quality image M of the viewpoint 2 from the low-quality image calibration unit (not shown), and receives the parallax vector (parallax map D) from the parallax estimation unit 10 , The parallax compensation of the high-quality image H is performed using the parallax map D, and the improved image in which the image quality is improved with respect to the low-quality image L based on the parallax-compensated high-quality image H and the calibrated low-quality image M K is generated (step S204), and the improved image K is output to the outside (step S205). The details of the image correction means 20 will be described in Examples 1 to 4 described later.

[Example 1]
First, the image correction unit 20 of the first embodiment will be described in detail. The image correcting unit 20 according to the first embodiment replaces a partial region of the calibrated low-quality image M with a high-quality image H subjected to parallax compensation, and generates an improved image K.

  FIG. 3 is a block diagram illustrating a configuration example of the image correction unit 20 according to the first embodiment, and FIG. 4 is a flowchart illustrating processing of the image correction unit 20 according to the first embodiment. The image correction unit 20-1 includes an image replacement unit 22 including a parallax compensation unit 21. The image correction unit 20-1 receives the high-quality image H of the viewpoint 1, receives the calibrated low-quality image M of the viewpoint 2 from the low-quality image calibration unit, and receives the parallax vector (parallax map D) from the parallax estimation unit 10. Is input, the high-quality image H is parallax-compensated based on the parallax map D, the result is pasted on the calibrated low-quality image M (the image is replaced for each partial region), and the improved image K of the viewpoint 2 is Generate and output to the outside.

  Specifically, when the image correction unit 20-1 inputs the high-quality image H, the parallax map D, and the calibrated low-quality image M (step S401), the parallax compensation unit 21 of the image replacement unit 22 The pixel value H (P) and the parallax vector D (P) of the parallax map D are input, and the calculation (parallax compensation) of the pixel value H (P + D (P)) shown in the upper part of the equation (6) shown below is performed. By performing (calculation), parallax compensation is performed on the high-quality image H, and a pixel value H (P + D (P)) with parallax compensation is generated (step S402).

  The image replacement means 22 receives the pixel value M (P) of the calibrated low-quality image, and determines whether or not the corresponding partial area of the high-quality image H exists in the partial area on the calibrated low-quality image M. Determination is made (step S403). Specifically, as shown in the condition part on the right side of Equation (6) described later, the image replacement unit 22 uses the calibrated low-quality image in which the coordinate value P + D (P) of the image obtained by parallax compensation of the high-quality image H is obtained. It is determined whether or not it exists in M images. When the partial area of the corresponding high-quality image H exists in the partial area on the calibrated low-quality image M (step S403: Y), the image replacement means 22 The pixel value H (P + D (P)) moved by parallax compensation for the corresponding partial area of the high-quality image H is pasted (step S404). In other words, in the partial area of the calibrated low-quality image M, the pixel value M (P) of the calibrated low-quality image M is moved by parallax compensation for the corresponding partial area of the high-quality image H. P + D (P)). On the other hand, when the corresponding partial area of the high-quality image H does not exist in the partial area on the calibrated low-quality image M (step S403: N), the image replacement unit 22 performs the partial area of the calibrated low-quality image M. The pixel value M (P) of the calibrated low image quality image M is used as it is (step S405). Then, the image replacement unit 22 generates the pixel value K (P) of the improved image K by the processing of step S404 and step S405 (step S406), and outputs the improved image K to the outside (step S407).

前記式（６）の下段に示すＭ（Ｐ）は、高画質画像Ｈを視差補償した結果、参照される高画質画像Ｈの画素が校正済み低画質画像Ｍの範囲外に位置する場合に、当該画素の代わりに校正済み低画質画像Ｍの画素値を充てることを示している。 As described above, the image correcting unit 20-1 moves the corresponding partial area of the high-quality image H with respect to the partial area on the calibrated low-quality image M as shown in the following equation (6) (parallax compensation). ) To generate an improved image K.

M (P) shown in the lower part of the equation (6) is obtained when the pixel of the high-quality image H referred to is located outside the range of the calibrated low-quality image M as a result of parallax compensation of the high-quality image H. It shows that the pixel value of the calibrated low-quality image M is used instead of the pixel.

  As described above, according to the multi-viewpoint image processing apparatus 1 including the image correction unit 20-1 of the first embodiment, the low-quality image calibration unit converts the low-quality image L into the high-quality image H by calibration. Match the coordinate system and match the quantization characteristics of the high-quality image H to generate a calibrated low-quality image M, and the parallax estimation means 10 is between the high-quality image H and the calibrated low-quality image M. The parallax map D is generated by associating each partial area by pattern matching. Then, the parallax compensation means 21 performs parallax compensation on the high-quality image H based on the parallax map D, and the image replacement means 22 corresponds to the partial area of the high-quality image H corresponding to the partial area on the calibrated low-quality image M. Is present, the high-quality image H with parallax compensation is pasted on the partial area of the calibrated low-quality image M, and the corresponding partial area of the high-quality image H is in the partial area on the calibrated low-quality image M. If it does not exist, the improved image K is generated using the calibrated low-quality image M as it is. Thereby, the partial region of the calibrated low-quality image M is replaced with the high-quality image H. Therefore, the low-quality image L of the multi-viewpoint images can be refined based on the high-quality image H, and the image quality of the low-quality image L can be improved.

  Further, for example, even in a system in which the transmission band can be reduced and the image quality difference occurs between the transmission images for the left eye and the right eye, the image quality of the low-quality image L can be improved by using the multi-viewpoint image processing apparatus 1. Since the image quality is improved, the difference in image quality between both eyes can be reduced. Also, with conventional 3D televisions with different image quality depending on the viewpoint, viewers feel tired and intoxicated in their eyes and mind, feel the difference in image quality when one eye is closed, or feel image quality deterioration like blurring. However, using the multi-viewpoint image processing apparatus 1 can solve these problems.

(Modification of Example 1)
In the image correction unit 20-1 of the first embodiment, as a first modification, the image replacement unit 22 pastes a corresponding partial area of the high-quality image H on the partial area on the calibrated low-quality image M. When attaching, as shown in the following formula (7), the pixel value H (P) of the high-quality image H with parallax compensation is pasted or calibrated according to the reliability of the parallax vector D (P). The pixel value K (P) of the improved image K may be generated by selecting whether to use the pixel value M (P) of the completed low-quality image M as it is.

  The reliability of the disparity vector D (P) is determined by the disparity estimation unit 10 that generates the disparity map D based on the evaluation value at the time of block matching. For example, the disparity estimation unit 10 determines that the disparity vector D (P) is reliable when the sum of absolute values of differences (SAD value) is equal to or less than a predetermined threshold in the calculation process of the equation (5). Is determined to be unreliable when the sum of the absolute values of is greater than a predetermined threshold. The parallax estimation unit 10 outputs the reliability of the parallax vector D (P) determined in this way to the image replacement unit 22 of the image correction unit 20-1.

As a second modification, when the image replacement unit 22 pastes the corresponding partial area of the high-quality image H on the partial area on the calibrated low-quality image M, the following equation (8) is given. Thus, the pixel of the high-quality image H that has been subjected to parallax compensation according to the reliability λ of the parallax vector D (P) (λ is a real number between 0 and 1 and the higher the λ, the higher the reliability). The proportional value ratio between the value H (P) and the pixel value M (P) of the calibrated low-quality image M is controlled, and the pixel value H (P) of the high-quality image H subjected to parallax compensation and the pixel of the calibrated low-quality image M The pixel value K (P) of the improved image K may be generated using the value M (P).

Λは広義単調減少関数とし、その値域は０以上１以下の実数とする。 The second modification will be specifically described below. For example, when the disparity estimation unit 10 obtains the disparity vector D (P) by the block matching method that minimizes the sum of absolute values of differences (SAD value) or the sum of squares of differences (SSD value), the following equation is obtained. In addition, the value of the reliability λ is determined using a function Λ such that the reliability λ increases as the minimum value e of these error values (SAD value, SSD value) decreases.

Λ is a broad monotone decreasing function, and its range is a real number between 0 and 1.

For example, the function Λ can be defined based on a predetermined threshold value θ as in the following equation.

Further, the function Λ can be defined by a polygonal line function based on two predetermined threshold values θ ₀ and θ ₁ (θ ₀ <θ ₁ ) as in the following equation.

Γは広義単調増加関数とし、その値域は０以上１以下の実数とする。 Further, for example, when the disparity estimation unit 10 obtains the disparity vector D (P) by the block matching method that maximizes the cross-correlation, a function that increases the reliability λ as the maximum value r of the correlation value increases. The value of the reliability λ is determined using Γ.

Γ is a monotonically increasing function in a broad sense, and its range is a real number between 0 and 1.

For example, the function Γ can be defined based on a predetermined threshold value θ as in the following equation.

Further, the function Γ can be defined by a polygonal line function based on two predetermined threshold values θ ₀ and θ ₁ (θ ₀ <θ ₁ ) as in the following equation.

  As described above, according to the multi-viewpoint image processing apparatus 1 including the image correction unit 20-1 according to the modification of the first embodiment, the parallax estimation unit 10 calculates the expression (5) indicating the block matching process. In the processing, the reliability of the disparity vector D (P) is obtained based on the sum of absolute values of differences (SAD value). That is, when the absolute value of the difference is small, the disparity vector D (P) is reliable, or the reliability of the disparity vector D (P) is increased, and when the absolute value of the difference is large, the disparity vector D (P ) Is not reliable or the reliability of the disparity vector D (P) is low. Then, the image replacement unit 22 selects whether to use the parallax-compensated high-quality image H or the calibrated low-quality image M according to the reliability of the parallax vector D (P), or parallax An improved image K is generated by controlling the proration ratio between the compensated high-quality image H and the calibrated low-quality image M. That is, when the reliability of the parallax vector D (P) is high, the high-quality image H with parallax compensation is used, or its proportionality is increased, and when the reliability of the parallax vector D (P) is low, calibration is performed. The low-quality image M already used is used, or the proration ratio is made high. Thereby, the low-quality image L of the multi-viewpoint images can be further refined based on the high-quality image H, and the image quality of the low-quality image L can be further improved.

[Example 2]
Next, the image correcting unit 20 of the second embodiment will be described in detail. The image correction unit 20 according to the second embodiment performs parallax compensation on the high frequency component (spatial high frequency component) of the high quality image H, superimposes the parallax compensated high frequency component on the calibrated low quality image M, and improves the image K. Is generated.

  FIG. 5 is a block diagram illustrating a configuration example of the image correction unit 20 according to the second embodiment, and FIG. 6 is a flowchart illustrating processing of the image correction unit 20 according to the second embodiment. The image correcting unit 20-2 includes an image superimposing unit 25 including a high-pass filter unit 23 and a parallax compensation unit 24. The image correction unit 20-2 receives the high-quality image H of the viewpoint 1, inputs the calibrated low-quality image M of the viewpoint 2 from the low-quality image calibration unit, and receives the parallax vector (parallax map D) from the parallax estimation unit 10. , A high-pass filter is applied to the high-quality image H, and the applied high-quality image H (high-frequency component) is parallax-compensated based on the parallax map D, and the result is calibrated low-quality image Superimpose on M, generate an improved image K of viewpoint 2, and output to the outside.

  Specifically, when the image correction unit 20-2 inputs the high-quality image H, the parallax map D, and the calibrated low-quality image M (step S601), the high-pass filter unit 23 applies the high-quality image H to the high-quality image H. Then, a high-pass filter is applied, and the result is output as the reinforcement signal image G to the image superimposing means 25 (step S602). The high-pass filter means 23 preferably has a high-pass characteristic that has a spectral envelope corresponding to the image quality difference between the high-quality image H and the calibrated low-quality image M, but this is not necessarily the case. It doesn't matter.

  For example, the high-pass filter unit 23 may generate the reinforcement signal image G by convolving a digital filter with a tap coefficient having a high-pass characteristic. Simply apply a Laplacian filter.

Further, for example, the high-pass filter unit 23 converts the high-quality image H into the frequency domain, then multiplies the frequency response having the high-pass frequency characteristic, and converts the result into the spatial domain. The reinforcement signal image G may be generated. At this time, the Fourier transform, the cosine transform, the sine transform, the Hadamard transform, the Walsh transform, or a discrete version of the transform from the spatial domain to the frequency domain is performed as a high-speed algorithm. Things can be used. Further, for the transformation from the frequency domain to the spatial domain, an inverse transformation method may be used in contrast to the above-described transformation method from the spatial domain to the frequency domain. Specifically, the high-pass filter means 23 Fourier transforms the high-quality image H to convert it into a spectral image, and the magnitude of the spatial frequency (two-dimensional spatial frequency (u, v ) Is set to (u ² + v ² ) ^1/2 ), and a high frequency spectrum image is generated with a spectral component having a value equal to or less than a predetermined value as 0. Then, the high-pass filter means 23 performs an inverse Fourier transform on the high-frequency spectrum image to generate a reinforcement signal image G.

  Further, for example, the high-pass filter unit 23 may generate the reinforcement signal image G by two-dimensional wavelet decomposition. Specifically, the high-pass filter means 23 converts the input high-quality image H into a horizontal low frequency and vertical low frequency (LL) component, horizontal high frequency and vertical low frequency (HL) component, horizontal low frequency and vertical. The signal is decomposed into a high frequency (LH) component, a horizontal high frequency component, and a vertical high frequency (HH) component, all LL components are replaced with 0, and then the inverse signal of the two-dimensional wavelet decomposition is executed to generate the reinforcing signal image G. As a basis function used for wavelet decomposition, for example, a Haar basis or a Doubechies basis can be used.

  The parallax compensation unit 24 of the image superimposing unit 25 receives the pixel value G (P) of the reinforcement signal image G from the high-pass filter unit 23 and the parallax vector D (P) of the parallax map. By performing the calculation of αG (P + D (P)) shown in the upper part of Expression (15) (the calculation of multiplying the result of parallax compensation by a coefficient α (α is a positive real number)), the reinforcement signal image G is parallax-compensated. The coefficient α is multiplied to generate a pixel value αG (P + D (P)) (step S603).

  The image superimposing means 25 inputs the pixel value M (P) of the calibrated low quality image M, and whether or not the corresponding partial region of the reinforcing signal image G exists in the partial region on the calibrated low quality image M. Is determined (step S604). Specifically, the image superimposing means 25 determines whether or not there is a corresponding partial region of the reinforcing signal image G by performing the same processing as the image replacing means 22 of the first embodiment. When there is a corresponding partial region of the reinforcing signal image G in the partial region on the calibrated low-quality image M (step S604: Y), the image superimposing means 25 The corresponding partial region of the reinforcement signal image G is moved by parallax compensation, and the pixel value αG (P + D (P)) multiplied by the coefficient α is superimposed (step S605). On the other hand, when there is no corresponding partial region of the reinforcing signal image G in the partial region on the calibrated low-quality image M (step S604: N), the image superimposing unit 25 determines that the partial region of the calibrated low-quality image M. The pixel value M (P) of the calibrated low-quality image M is used as it is (step S606). Then, the image superimposing means 25 generates the pixel value K (P) of the improved image K by the processing of step S605 and step S606 (step S607), and outputs the improved image K to the outside (step S608).

前記式（１５）の下段に示すＭ（Ｐ）は、補強信号画像Ｇを視差補償して係数α倍した結果、参照される補強信号画像Ｇの画素が校正済み低画質画像Ｍの範囲外に位置する場合に、当該画素を重畳する代わりに、校正済み低画質画像Ｍの画素値を充てることを示している。 As described above, the image correction unit 20-2 moves the corresponding partial region of the reinforcement signal image G with respect to the partial region on the calibrated low-quality image M (parallax compensation) as shown in the following equation (15). ) And multiply the coefficient α (α is a positive real number) and superimpose to generate an improved image K.

M (P) shown in the lower part of the equation (15) is the result of parallax compensation of the reinforcement signal image G and multiplying by the coefficient α, so that the pixel of the reference reinforcement signal image G is outside the range of the calibrated low-quality image M. In this case, the pixel value of the calibrated low-quality image M is used instead of superimposing the pixel.

  As described above, according to the multi-viewpoint image processing apparatus 1 including the image correction unit 20-2 of the second embodiment, the low-quality image calibration unit calibrates the low-quality image L as in the first embodiment. A calibrated low-quality image M is generated, and the parallax estimation means 10 generates the parallax map D. Then, the high-pass filter unit 23 applies a high-pass filter to the high-quality image H to generate the reinforcement signal image G, and the parallax compensation unit 24 uses the parallax map D to reinforce the signal image. G is parallax-compensated and multiplied by a factor α, and when the image superimposing means 25 has a corresponding partial region of the reinforcing signal image G in the partial region on the calibrated low-quality image M, the calibrated low-quality image M When the reinforcement signal image G multiplied by the coefficient α is superimposed on the partial area and the partial area of the corresponding reinforcement signal image G does not exist in the partial area on the calibrated low-quality image M, the calibrated low The improved image K is generated by using the image quality image M as it is. As a result, the high-frequency component of the high-quality image H is superimposed on the partial region of the calibrated low-quality image M. Therefore, the low-quality image L of the multi-viewpoint images can be refined based on the high-quality image H, and the image quality of the low-quality image L can be improved.

  In addition, since the high-frequency component information to be interpolated is added while the information of the calibrated low-quality image M is left, the image quality of the low-quality image L is improved while retaining the information viewed from the actual viewpoint. Can do. Further, as in the first embodiment, the difference in image quality between both eyes can be reduced, and the problem of the conventional stereoscopic television in which the image quality differs depending on the viewpoint can be solved.

(Modification of Example 2)
In the image correcting unit 20-2 of the second embodiment, as a first modification, the image superimposing unit 25 superimposes the corresponding partial region of the reinforcing signal image G on the partial region on the calibrated low-quality image M. In this case, as shown in the following equation (16), superimposition of the reinforcement signal image G is executed according to the reliability of the disparity vector D (P), or the pixel value of the calibrated low image quality image M is used as it is. The pixel value K (P) of the improved image K may be generated by selecting whether to use it.

  As in the case of the modification of the first embodiment, the reliability of the disparity vector D (P) is evaluated by the disparity estimation unit 10 that generates the disparity map D (for example, the evaluation value at the time of block matching (for example, the equation (5)) It is determined based on the sum of absolute values of differences in the arithmetic processing (SAD value).

As a second modification, the image superimposing means 25 shows the following equation (17) when superimposing the corresponding partial region of the reinforcement signal image G on the partial region on the calibrated low-quality image M. Thus, the pixel value of the calibrated low-quality image M according to the reliability λ of the disparity vector D (P) (λ is a real number between 0 and 1 and the reliability is higher as λ is larger). The superimposition amount of the pixel value G (P) of the reinforcement signal image G subjected to parallax compensation with respect to M (P) is controlled, the pixel value M (P) of the calibrated low image quality image M, and the pixel value of the reinforcement signal image G subjected to parallax compensation. The pixel value K (P) of the improved image K may be generated using G (P).

  The specific example of the second modified example in the second embodiment is similar to the first modified example and the second modified example in the first embodiment using the function Λ or the function Γ, and the reliability Determine the value of λ.

  As described above, according to the multi-viewpoint image processing apparatus 1 including the image correcting unit 20-2 according to the modified example of the second embodiment, the disparity vector D (P) is reliable or the disparity vector D (P) is reliable. When the degree is high, the high-frequency component of the high-quality image H is superimposed on the partial region of the calibrated low-quality image M. Therefore, the low-quality image L of the multi-viewpoint images can be further refined based on the high-quality image H, and the image quality of the low-quality image L can be further improved.

Example 3
Next, the image correcting unit 20 of the third embodiment will be described in detail. The image correction unit 20 according to the third embodiment performs disparity compensation on the texture component (fluctuation component) of the high-quality image H, and superimposes the disparity-compensated texture component on the calibrated low-quality image M to generate the improved image K. It is characterized by that.

  FIG. 7 is a block diagram illustrating a configuration example of the image correction unit 20 according to the third embodiment. The image correcting unit 20-3 includes an image superimposing unit 25 including a texture extracting unit 26 and a parallax compensating unit 24. The image correction unit 20-3 receives the high-quality image H of the viewpoint 1, receives the calibrated low-quality image M of the viewpoint 2 from the low-quality image calibration unit, and receives the parallax vector (parallax map D) from the parallax estimation unit 10. , The texture component is extracted from the high-quality image H, the texture component of the high-quality image H is parallax-compensated based on the parallax map D, and the result is superimposed on the calibrated low-quality image M. The improved image K is generated and output to the outside.

  The processing of the image correcting unit 20-3 according to the third embodiment is the same except for step S602 among steps S601 to S608 shown in FIG.

  Specifically, when the image correction unit 20-3 inputs the high-quality image H, the parallax map D, and the calibrated low-quality image M (corresponding to step S601), the texture extraction unit 26 extracts the texture component from the high-quality image H. And the texture component is output to the image superimposing means 25 as the reinforcement signal image G (corresponding to step S602).

  For example, the texture extraction means 26 is applied with a filter that separates the skeleton component (structure component) and the texture component such as a total variation filter, and the skeleton component and the texture component are extracted from the high-quality image H by using the total variation regularization method. The separated texture components may be output as the reinforcing signal image G. Since the separation method of the skeleton component and the texture component by the total variation regularization method is known, the description thereof is omitted here. For details, see `` Jerome Gilles and Yves Meyer: Properties of BV-G Structures + Textures Decomposition Models.Application to Road Detection in Satellite Images, IEEE Transactions on Image Processing, vol.19, no.11, November 2010. '' Please refer.

  The parallax compensation unit 24 of the image superimposing unit 25 inputs the pixel value G (P) of the reinforcement signal image G from the texture extraction unit 26 and also inputs the parallax vector D (P) of the parallax map, and the equation (15) By performing αG (P + D (P)) calculation (calculation by multiplying the parallax-compensated result by a factor α (α is a positive real number)), the reinforcement signal image G is parallax-compensated and multiplied by a factor α. , A pixel value αG (P + D (P)) is generated (corresponding to step S603). Thereafter, the image superimposing means 25 performs the same processing as in steps S604 to S608 shown in FIG.

  As described above, the image correcting unit 20-3 determines whether or not there is a corresponding partial region of the reinforcing signal image G by performing the same processing as the image replacing unit 22 of the first embodiment. As shown in (15), the partial region of the corresponding reinforcing signal image G is moved (parallax compensation) with respect to the partial region on the calibrated low-quality image M, multiplied by a factor α (α is a positive real number), and superimposed. By doing so, an improved image K is generated. The same applies to the image correction means 20-4 of Example 4 described later.

  As described above, according to the multi-viewpoint image processing apparatus 1 including the image correction unit 20-3 according to the third embodiment, the low-quality image calibration unit calibrates the low-quality image L as in the first and second embodiments. Thus, the calibrated low-quality image M is generated, and the parallax estimation means 10 generates the parallax map D. Then, the texture extraction unit 26 extracts a texture component from the high-quality image H to generate a reinforcement signal image G, and the parallax compensation unit 24 performs parallax compensation on the reinforcement signal image G based on the parallax map D to obtain a coefficient α. The image superimposing means 25 performs parallax compensation on the partial region of the calibrated low-quality image M when the partial region of the corresponding reinforcement signal image G exists in the partial region on the calibrated low-quality image M. If the reinforcement signal image G multiplied by the coefficient α is superimposed and there is no corresponding partial area of the reinforcement signal image G in the partial area on the calibrated low-quality image M, the calibrated low-quality image M is used as it is. An improved image K is generated. As a result, the texture component of the high-quality image H is superimposed on the partial region of the calibrated low-quality image M. Therefore, the low-quality image L of the multi-viewpoint images can be refined based on the high-quality image H, and the image quality of the low-quality image L can be improved.

  Further, since only the non-contour vibration component that is a texture component is added to the calibrated low-quality image M, the image quality of the low-quality image L can be improved while preventing artifacts such as ringing at the edge portion. Can do. When the total variation filter is used for the texture extraction unit 26, the total variation norm that is easy to calculate is regularized, so that the image quality of the low-quality image L can be improved with a low calculation cost.

  Further, in order to add the information of the texture component to be interpolated while keeping the information of the calibrated low-quality image M, the low-quality image L is held while retaining the information viewed from the actual viewpoint as in the second embodiment. Image quality can be improved. Further, as in the first embodiment, the difference in image quality between both eyes can be reduced, and the problem of the conventional stereoscopic television in which the image quality differs depending on the viewpoint can be solved.

  Note that the image correction means 20-3 of the third embodiment can apply a modification similar to the first and second modifications in the second embodiment.

Example 4
Next, the image correcting unit 20 of the fourth embodiment will be described in detail. The image correcting unit 20 of the fourth embodiment corrects the color difference component of the high-quality image H when the high-quality image H is a color image and the low-quality image L is a monochrome image, and calibrated the low-quality image M. On the other hand, the improved image K is generated by superimposing the color difference components subjected to parallax compensation.

  FIG. 8 is a block diagram illustrating a configuration example of the image correction unit 20 according to the fourth embodiment. The image correcting unit 20-4 includes an image superimposing unit 25 including a color difference extracting unit 27 and a parallax compensating unit 24. The image correction unit 20-4 receives the high-quality image H (color image) of the viewpoint 1, receives the calibrated low-quality image M (calibrated monochrome image) of the viewpoint 2 from the low-quality image calibration unit, and estimates the parallax. A disparity vector (disparity map D) is input from the means 10, a color difference is extracted from the high-quality image H, the color difference of the high-quality image H is parallax-compensated based on the disparity map D, and the result is calibrated low-quality image Superimpose on M, generate an improved image K of viewpoint 2, and output to the outside.

  The processing of the image correcting unit 20-4 of the fourth embodiment is the same except for step S602 in steps S601 to S608 shown in FIG.

  Specifically, when the image correction unit 20-4 inputs the high-quality image H, the parallax map D, and the calibrated low-quality image M (corresponding to step S601), the color difference extraction unit 27 calculates the color difference from the high-quality image H. The color difference is extracted and output to the image superimposing means 25 as the reinforcing signal image G (corresponding to step S602).

  For example, the color difference extraction unit 27 can perform ITU-R BT. 601 and Cb signal and Cr signal, or ITU-R BT. The Pb signal and the Pr signal defined in 709 may be extracted.

  The parallax compensation unit 24 of the image superimposing unit 25 inputs the pixel value G (P) of the reinforcement signal image G from the color difference extraction unit 27 and also inputs the parallax vector D (P) of the parallax map, and the equation (15) By performing αG (P + D (P)) calculation (calculation by multiplying the parallax-compensated result by a factor α (α is a positive real number)), the reinforcement signal image G is parallax-compensated and multiplied by a factor α. , A pixel value αG (P + D (P)) is generated (corresponding to step S603). Thereafter, the image superimposing means 25 performs the same processing as in steps S604 to S608 shown in FIG.

  As described above, the image correction unit 20-4 moves (parallax compensation) the corresponding partial region of the reinforcing signal image G with respect to the partial region on the calibrated low-quality image M as shown in the equation (15). Then, the coefficient α is multiplied (α is a positive real number), and the improved image K is generated by superimposing. That is, the image superimposing unit 25 of the image correcting unit 20-4 moves the partial region of the reinforcement signal image G that is the corresponding color difference image to the partial region of the calibrated low-quality image M that is the luminance image (parallax compensation). Then, the improved image K is generated by combining them.

  As described above, according to the multi-viewpoint image processing apparatus 1 including the image correcting unit 20-4 of the fourth embodiment, the low-quality image calibrating unit calibrates the low-quality image L (monochrome image) and calibrated low The image quality image M is generated, and the parallax estimation means 10 generates the parallax map D. Then, the color difference extraction unit 27 extracts the color difference from the high-quality image H (color image) to generate the reinforcement signal image G, and the parallax compensation unit 24 performs the parallax compensation on the reinforcement signal image G based on the parallax map D. When the image superimposing means 25 multiplies the coefficient α and the partial region of the reinforced signal image G that is the corresponding color difference image exists in the partial region on the calibrated low-quality image M that is the luminance image, The reinforcement signal image G multiplied by the coefficient α by parallax compensation is superimposed on the partial area of the image M, and the partial area of the reinforcement signal image G that is the corresponding color difference image is superimposed on the partial area on the calibrated low-quality image M. If it does not exist, the improved image K is generated using the calibrated low-quality image M as it is. Thereby, the color difference of the high-quality image H is superimposed on the partial region of the calibrated low-quality image M. Therefore, the low-quality image L of the multi-viewpoint images can be refined based on the high-quality image H, and the image quality of the low-quality image L can be improved.

  In the system that transmits the high-quality image H and the low-quality image L from the transmission side to the multi-viewpoint image processing apparatus 1, the transmission side transmits a part of the multi-viewpoint images to the low-quality image L without color information. Therefore, the transmission band necessary for obtaining the same image quality can be reduced.

  Further, in order to add the information of the color difference to be interpolated while keeping the information of the calibrated low-quality image M, the low-quality image L of the low-quality image L is retained while retaining the information viewed from the actual viewpoint as in the second embodiment. The image quality can be improved. Further, as in the first embodiment, the difference in image quality between both eyes can be reduced, and the problem of the conventional stereoscopic television in which the image quality differs depending on the viewpoint can be solved.

  In addition, the image correction means 20-4 of Example 4 can apply the modification similar to the 1st modification in Example 2, and a 2nd modification.

  Although the present invention has been described with reference to the first to fourth embodiments, the present invention is not limited to the first to fourth embodiments, and various modifications can be made without departing from the technical idea thereof. For example, in the first to third embodiments, the difference in image quality between the high-quality image H and the low-quality image L includes the high-quality image H and the low-quality image L in addition to the difference in resolution and the difference in quantization method. It may be caused by a difference in encoding parameters when encoded and transmitted. The encoding parameter is, for example, a QP (Quantization Parameter) value used in a quantization step for orthogonal transform coefficients. An image encoded using a small value (fine quantization) QP value is a high quality image H, and an image encoded using a large value (coarse quantization) QP value is a low image quality image L. is there. In this case, in the system that transmits the high-quality image H and the low-quality image L from the transmission side to the multi-viewpoint image processing apparatus 1, the transmission side transmits a part of the high-quality image H of the multi-viewpoint images with high compression. Therefore, the transmission band necessary for obtaining the same image quality can be reduced.

  When the difference in image quality between the high-quality image H and the low-quality image L is caused by the difference in resolution, the transmission side sets some of the multi-viewpoint images as the low-resolution image L with low resolution. Since transmission is possible, a transmission band necessary for obtaining the same image quality can be reduced.

  In addition, when the difference in image quality between the high-quality image H and the low-quality image L is caused by the difference in the quantization method of the pixel value, the transmission side, for example, selects some of the multi-viewpoint images. Can be transmitted as a low-gradation low-quality image L with a small bit depth, so that the transmission band necessary for obtaining the same image quality can be reduced.

  Note that a normal computer can be used as the hardware configuration of the multi-viewpoint image processing apparatus 1 according to the embodiment of the present invention. The multi-viewpoint image processing apparatus 1 includes a computer having a volatile storage medium such as a CPU and a RAM, a non-volatile storage medium such as a ROM, an interface, and the like. Each function of the parallax estimation means 10 and the image correction means 20 provided in the multi-viewpoint image processing apparatus 1 is realized by causing the CPU to execute a program describing these functions. These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc. You can also send and receive.

DESCRIPTION OF SYMBOLS 1 Multiview image processing apparatus 10 Parallax estimation means 20 Image correction means 21, 24 Parallax compensation means 22 Image replacement means 23 High-pass filter means 25 Image superimposition means 26 Texture extraction means 27 Color difference extraction means

Claims (6)

A multi-viewpoint image processing apparatus that uses a high-quality viewpoint image to improve the image quality of a low-quality viewpoint image,
A parallax estimation unit that associates each partial area between the high-quality viewpoint image and the low-quality viewpoint image and estimates a parallax;
Based on the parallax estimated by the parallax estimation means, parallax compensation is performed on the high-quality viewpoint image, and the quality of the low-quality viewpoint image is improved using the parallax-compensated high-quality viewpoint image. An image correcting means for generating a processed image,
The image correcting means is
Extracting a reinforcing signal image from the high-quality viewpoint image,
Based on the parallax estimated by the parallax estimation means, perform parallax compensation for each partial region on the reinforcement signal image,
For each partial region, the corresponding parallax-compensated reinforcement signal image is superimposed on the low-quality viewpoint image ,
The multi-viewpoint image processing apparatus , wherein the reinforcement signal image is an image of a spatial high-frequency component . A multi-viewpoint image processing apparatus that uses a high-quality viewpoint image to improve the image quality of a low-quality viewpoint image,
A parallax estimation unit that associates each partial area between the high-quality viewpoint image and the low-quality viewpoint image and estimates a parallax;
Based on the parallax estimated by the parallax estimation means, parallax compensation is performed on the high-quality viewpoint image, and the quality of the low-quality viewpoint image is improved using the parallax-compensated high-quality viewpoint image. An image correcting means for generating a processed image,
The image correcting means is
Extracting a reinforcing signal image from the high-quality viewpoint image,
Based on the parallax estimated by the parallax estimation means, perform parallax compensation for each partial region on the reinforcement signal image,
For each partial region, the corresponding parallax-compensated reinforcement signal image is superimposed on the low-quality viewpoint image,
The multi-viewpoint image processing apparatus, wherein the reinforcement signal image is a texture component image. The image correcting means is
The multi-viewpoint image processing apparatus according to claim 2 , wherein a texture component is extracted from the high-quality viewpoint image using a total variation filter. A multi-viewpoint image processing apparatus that uses a high-quality viewpoint image to improve the image quality of a low-quality viewpoint image,
A parallax estimation unit that associates each partial area between the high-quality viewpoint image and the low-quality viewpoint image and estimates a parallax;
Based on the parallax estimated by the parallax estimation means, parallax compensation is performed on the high-quality viewpoint image, and the quality of the low-quality viewpoint image is improved using the parallax-compensated high-quality viewpoint image. An image correcting means for generating a processed image,
The image correcting means is
Extracting a reinforcing signal image from the high-quality viewpoint image,
Based on the parallax estimated by the parallax estimation means, perform parallax compensation for each partial region on the reinforcement signal image,
For each partial region, the corresponding parallax-compensated reinforcement signal image is superimposed on the low-quality viewpoint image,
The multi-viewpoint image processing apparatus, wherein the reinforcement signal image is a color difference component image. A multi-viewpoint image processing method for improving the image quality of a low-quality viewpoint image using a high-quality viewpoint image,
A first step of inputting the high-quality viewpoint image and the low-quality viewpoint image;
A second step of performing an association for each partial region between the high-quality viewpoint image and the low-quality viewpoint image and estimating a parallax;
Based on the estimated parallax, parallax compensation is performed on the high-quality viewpoint image, and an image in which the quality of the low-quality viewpoint image is improved is generated using the parallax-compensated high-quality viewpoint image. A third step,
The third step is
Extracting a reinforcing signal image from the high-quality viewpoint image,
Based on the estimated parallax, parallax compensation is performed on the reinforcement signal image for each partial region,
For each partial region, the corresponding parallax-compensated reinforcement signal image is superimposed on the low-quality viewpoint image ,
The multi-viewpoint image processing method , wherein the reinforcement signal image is an image of a spatial high-frequency component, an image of a texture component, or an image of a color difference component . A multi-viewpoint image processing program for causing a computer to function as the multi-viewpoint image processing apparatus according to any one of claims 1 to 4 .

Images