TW201519635A - Method and apparatus for correcting the multi-view images - Google Patents

Abstract

Method for correctig the multi-view images is applied to correct brightness and color of each of a plurality of view images before generating a three-dimensional image using these view images. First, choose one from the view images as a reference image according the brightness of the view images, and the other of the view images is though as a plurality of correction images. Compute global disparity information of the correction images according to the reference image, and cut the correction images according to the individual global disparity information. Adjust brightness of the correction images base on an average of brightness of the reference image. Generate a color correcting information according to each two adjacent correction images corrected brightness, and adjust color of each correction image according to the color correcting information.

Description

Multi-view image correction device and method thereof

The invention relates to a pre-processing device for a three-dimensional stereoscopic image and a method thereof, in particular to a multi-view image correction device and a method thereof.

The 3D naked-eye stereoscopic display technology applied to a three-dimensional display device (for example, a stereoscopic television) generally synthesizes a three-dimensional image (3D image) having a stereoscopic effect by using a multi-view image, and the multi-view image refers to using multiple parts. The camera set in different positions synchronously captures a plurality of viewing angle images having different viewing angles obtained by the dynamic scene (the multi-view image and the plurality of viewing angle images referred to in the specification of the present invention have the same meaning unless otherwise specified), and more The view image contains a plurality of view images captured from different angles. Due to the change of ambient brightness, the angle of the camera, and the correction before the camera is taken, the angles of view of different angles of view are different, which affects the subsequent use. Stereoscopic imaging effect of 3D images synthesized by viewing angle images.

In a known 3D image synthesis technology, the depth information of the image is obtained while capturing the scene image required for synthesizing the 3D image by using the camera, and the color information obtained by the camera can be combined with the depth information to generate a stereo image, and the depth is obtained. There are many different ways of information, including: (1) through deep camera; (2) making Use two cameras to capture two different perspective images, and use two different perspectives to create depth information of one of the perspective images. (3) Use multiple cameras to capture multi-view images and calculate multiple images. The disparity vector between the view images is converted to the depth image information using the parallax vector value, but if multiple cameras are used to capture only one set of image and depth information for synthesis, the synthetic image will be generated due to the effect of Occlusion. Holes; (4) use a single camera to capture images and use image features to convert into depth information. Since depth image information cannot have high-resolution information like color image information, insufficient resolution will affect the stereoscopic rendering effect. In addition, depth image camera is expensive and difficult to obtain, so it is not widely used.

In general, the three-dimensional imaging effects of 3D images that affect multi-view image synthesis include: the masking problem caused by capturing different angles of view images from different angles, and the difference in brightness between the angles of view images of different viewing angles. The problem of obscuration means that the same scene is blocked by the front object in the view image captured from a certain angle of view, and is not blocked by the front object in the view image captured by other angles, so each of the multi-view images is When the angle of view image is color-corrected, it will cause the information to be filled with errors and result in uneven color. The reason for the difference in brightness is that when using multiple cameras to simultaneously capture dynamic scenes, the brightness of each view image in the multi-view image is inconsistent due to the influence of the scene and the brightness of the light, resulting in obvious brightness for the user to watch. difference.

In addition to the brightness correction, the multi-view image needs to be adjusted in terms of color. A known color adjustment method is histogram comparison (Histogram). Matching), this method is to adjust the color information through the statistical information of the known image information, but this method must be accurately calculated, otherwise as long as some of the micro errors have great distortion effect on the color. Then, the point and point correction on the image is not realized in the multi-view image. The main reason is that there will be displacement information on the multi-view image. If the displacement information is not obtained, the image will have the same color distortion when the color is corrected.

The invention provides a correction device and a method for multi-view images applied to a three-dimensional display device, which are used for solving the problems of shielding problems, brightness difference problems and color distortions in the process of synthesizing 3D images using multi-view images.

An embodiment of the method includes: selecting one of a plurality of different view images as a reference image, and using the remaining view images as a plurality of corrected images; calculating overall disparity information of each corrected image according to the reference image, and Correcting the overall parallax information of the image to perform image cropping on each corrected image; correcting the brightness information of each corrected image based on the average brightness of the reference image so that the brightness average of each corrected image corrected by the brightness information is closest to the reference image The average brightness value; and the correction image closest to the reference image from the overall parallax information starts to generate a color correction information according to the corrected image and the previous image corrected by the brightness information, and selectively corrects the corresponding brightness information correction according to the color correction information. After correcting the color information of the image.

In an embodiment of the invention, a brightness average value closest to a threshold is selected from a plurality of different perspective images as a reference image.

According to an embodiment of the method of the present invention, the overall disparity estimation mechanism calculates the overall disparity information of each corrected image according to the reference image by using a sum of the absolute difference (SAD) formula, according to the overall parallax of each corrected image. The information is image-cutted for each view image, and the most similar area is found between the reference image and each corrected image, and unnecessary image information is deleted, the image quality is improved, and the error rate of block matching is reduced.

According to an embodiment of the method of the present invention, performing image cropping on each view image includes deleting leftmost and rightmost image information of the plurality of different view images.

According to an embodiment of the method of the present invention, in the color correction step, the corrected image adjacent to the reference image is corrected, and then the corrected corrected image is used to perform color correction on the adjacent other corrected image. The displacement information is reduced, and the correction quality of the image color information is improved, and a better effect can be obtained.

Another aspect of the present invention includes a correction device for a multi-view image of a three-dimensional display device for solving the problem of obscuration, brightness difference, and color distortion faced in synthesizing 3D images using multi-view images.

The present invention provides an embodiment of an apparatus for implementing the above method, comprising: a reference information processing unit, an overall disparity estimating unit, a storage unit, and a color correction unit. The reference information processing unit calculates the brightness average value, selects one of the plurality of different view images as a reference image, and uses the remaining view image as a plurality of corrected images, and uses the reference image as a correction reference for brightness and color, and the reference image is The brightness and color information is transmitted to the later stage, and then according to the parameters The test image calculates the overall parallax information of each corrected image, and then performs image cropping on each view image according to the overall parallax information of each corrected image, and then uses the brightness correcting unit to correct the brightness information of each corrected image based on the average brightness of the reference image. The brightness average value of each corrected image corrected by the brightness information is closest to the brightness average value of the reference image. Then, the color correction unit generates a color correction information based on the corrected image of the entire parallax information closest to the reference image, and generates a color correction information according to the corrected corrected image and the previous image, and selectively corrects the corresponding brightness information correction according to the color correction information. After correcting the color information of the image.

The details of other functions and embodiments of the present invention will be described below in conjunction with the drawings.

10‧‧‧Reference Information Processing Unit

11‧‧8-bit right shifter

12‧‧‧ accumulator

13‧‧‧Subtractor

14‧‧‧Absolute encoder

15‧‧‧ Comparator

20‧‧‧Integral Parallax Estimation Unit

21a‧‧‧First subtractor

21b‧‧‧second subtractor

22a‧‧‧First absolute encoder

22b‧‧‧Second absolute encoder

23a‧‧‧First accumulator

23b‧‧‧Second accumulator

24‧‧‧Adder

25‧‧‧ Comparator

30‧‧‧Brightness correction unit

31‧‧‧ Comparator

32‧‧‧Incrementer

33a‧‧‧ register

33b‧‧‧Displacement register

34‧‧‧Subtractor

35‧‧‧Absolute encoder

36a‧‧‧Adder

36b‧‧‧Adder

37‧‧‧ comparator

38‧‧‧Subtractor

39‧‧‧Multiplier

40‧‧‧ storage unit

41‧‧‧ memory controller

50‧‧‧Color Correction Unit

C‧‧‧variable coefficient

D‧‧‧displacement

D1‧‧‧ displacement

D2‧‧‧ displacement

View1~Viewn‧‧‧ view image

VR/VT‧‧‧V color space information

UR/UT‧‧‧U color space information

Y‧‧‧Fixed image brightness information

L1‧‧‧Preset brightness value

S1‧‧‧Sort circuit

SRAM1‧‧‧ first memory

SRAM2‧‧‧ second memory

SRAM3‧‧‧ third memory

SRAM4‧‧‧ fourth memory

SAD‧‧‧ the sum of absolute differences

CDF‧‧‧ cumulative distribution function

S1‧‧‧Select one of the multi-view images as the reference image and use the remaining view images as the plurality of corrected images

S2‧‧‧ Calculate the overall parallax information of each corrected image based on the reference image, and perform image cropping on each view image based on the overall parallax information of each corrected image

S3‧‧‧ corrects the brightness information of each corrected image based on the average brightness of the reference image so that the brightness average of each corrected image corrected by the brightness information is closest to the brightness average of the reference image

S4‧‧‧ The color correction information is generated from the corrected image with the overall parallax information closest to the reference image, and the corrected image and the previous image are corrected according to the brightness information, and the corresponding brightness information is selectively corrected according to the color correction information. Correct the color information of the image

FIG. 1 is a flow chart showing the steps of a method for correcting a multi-view image according to an embodiment of the present invention.

FIG. 2 is a functional block diagram of a multi-view image correction device according to an embodiment of the present invention.

3A-3B is a computational example diagram of the overall disparity estimation mechanism, showing the result of the overall disparity estimation mechanism calculating the sum of the absolute differences of the overall disparity information of each corrected image.

Figure 4 is a diagram showing an example of cropping an image in the method of the present invention, showing the position of the leftmost and rightmost image information in the multi-view image.

5A-5C are schematic diagrams (1) of the luminance information correction performed by the adjustable histogram equalization algorithm according to an embodiment of the present invention.

6A-6C are schematic diagrams of the brightness information correction performed by the adjustable histogram equalization algorithm according to an embodiment of the present invention (2).

7A-7C are schematic diagrams of the brightness information correction performed by the adjustable histogram equalization algorithm according to an embodiment of the present invention (3).

FIG. 8 is a circuit diagram of a reference information processing unit according to an embodiment of the present invention.

FIG. 9 is a circuit diagram of an overall parallax estimation unit according to an embodiment of the present invention.

FIG. 10 is a schematic diagram showing a probability distribution function and an adjustable cumulative distribution function in a brightness correction unit according to an embodiment of the present invention.

Figure 11 is a circuit diagram of a conversion function in a luminance correction unit according to an embodiment of the present invention.

Figure 12 is a circuit diagram of a conversion function having a function of reducing brightness in a brightness correction unit according to an embodiment of the present invention.

Figure 13 is a circuit diagram of a storage unit according to an embodiment of the present invention.

FIG. 14 is a diagram showing color correction information of a U color space after storing a color correction in a storage unit according to an embodiment of the present invention; Schematic diagram.

FIG. 15 is a diagram showing color correction information of a V color space after a color correction is stored in a storage unit according to an embodiment of the present invention; Schematic diagram.

FIG. 16 is a structural diagram of a U color space data statistical circuit in a storage unit according to an embodiment of the present invention.

FIG. 17 is a structural diagram of a U color space data statistical circuit after color correction in a storage unit according to an embodiment of the invention.

Please refer to "FIG. 1". An embodiment of the method of the present invention includes the following steps: S1. From a multi-view image (assuming a plurality of different perspectives) Select one of View1~Viewn as the reference image (ViewR) and use the remaining view image as the plurality of corrected images; S2. Calculate the overall parallax of each corrected image based on the reference image using a Global Disparity Estimation mechanism Information, and image clipping of each corrected image according to the overall parallax information of each corrected image; S3. Correcting the brightness information of each corrected image based on the average brightness of the reference image, so that the corrected images after the brightness information are corrected The average value of the brightness is closest to the average value of the brightness of the reference image; and S4. The color correction information is generated according to the correction image corrected by the brightness information and the previous image, and the color correction information is corrected according to the color correction information. The information selectively corrects the color information of the corrected image after the brightness information is corrected.

Referring to FIG. 2, an embodiment of the apparatus for implementing the above method includes: a reference information processing unit 10, an overall disparity estimation unit 20, a brightness correction unit 30, a storage unit 40, and a color correction unit 50. . The reference information processing unit 10 is configured to receive a plurality of view images View1~Viewn, select one of the view images as the reference image ViewR, and use the remaining view images as the plurality of corrected images. The overall disparity estimating unit 20 is electrically connected to the reference information processing unit 10. The overall parallax estimating unit 20 is configured to calculate the overall parallax information of each corrected image according to the reference image ViewR, and perform image cropping on each corrected image according to the overall parallax information of each corrected image. The brightness correction unit 30 is electrically connected to the reference information processing unit 10. The brightness correcting unit 30 corrects the brightness information of each of the corrected corrected images based on the average value of the brightness of the reference image ViewR so that the brightness average value of each corrected image after the brightness information correction is closest to the brightness average value of the reference image ViewR. Color correction list The element 50 is electrically connected to the reference information processing unit 10 and the brightness correction unit 30. The color correction unit 50 is configured to generate a color correction information by using a color correction mechanism based on the correction information of the entire parallax information closest to the reference image ViewR, and the color correction information according to the color correction information. Correct the color information of the corrected image after the corresponding brightness information is corrected.

In an embodiment of the apparatus of the present invention, a storage unit 40 can also be utilized for storing data generated during the process. The storage unit 40 includes a memory controller 41 and a plurality of memories SRAM1 to SRAM4. The memory controller 41 is used to control the reading and writing operations of the four memory SRAM1~SRAM4. The implementation of the storage unit 40 may take the form of a separate circuit architecture or may be integrated into the various units of the aforementioned apparatus.

The method of the invention is to enable the multi-view image to adjust the brightness and color of each view image before synthesizing the three-dimensional image to obtain a better synthetic effect. In order to make the multi-view image corrections taken by all the cameras tend to be similar, one of the multi-view images is selected as the reference image ViewR, and the remaining view images are used as the corrected corrected images, and the reference image ViewR is used in the subsequent processing. The baseline corrects these corrected images. Moreover, this correction method is also more effective than separately correcting each view image.

[Selecting an implementation of a reference image]

One embodiment of selecting one of the multi-view images as the reference image ViewR is to calculate the brightness average of all the view images, and use the view image with the brightness average close to a threshold T as the reference image, and then use the reference image ViewR. Reference ViewR information is used as a correction reference for correcting the brightness and color of the remaining corrected images. The reference image information includes at least the brightness information and color information of the reference image. In general, the image brightness is represented by a value between 0 and 255. One of the embodiments uses 128 as the threshold value, which avoids the problem that other view images after brightness correction are too dark or too bright; It should be noted that the implementation of the threshold value T is not limited to 128, and a suitable threshold value may be determined depending on the situation.

In terms of setting the condition setting of the reference image, one embodiment of the present invention uses the luminance average value of each view image to approach the threshold value of 128 as the reference image. However, this is not the only embodiment. Any one of the brightness average value, the median value, the common area brightness value, the image sharpness, and the image contrast of all the view images may be considered as the condition for selecting the reference image. In addition, a plurality of reference images may be set to perform multi-view image brightness and color correction. Those who are familiar with the technical field in this technical field should fully understand and change the technical content of the present invention and implement it.

[Implementation of calculation of overall parallax information for each corrected image based on reference image]

Calculating the overall parallax information of each corrected image according to the reference image is a solution for the image displacement and the difference of the image information caused by different shooting angles in the multi-view image. One embodiment of the method of the present invention utilizes a The global disparity estimation mechanism calculates the overall disparity information of each corrected image based on the reference image. The overall disparity estimation mechanism uses absolute difference The sum of the absolute difference (SAD) formula calculates the overall disparity information of each corrected image according to the reference image, and then finds the most similar area between the reference image and each corrected image, and deletes unnecessary image information in the corrected image, thereby improving Image quality and error rate for reducing block matching.

According to an embodiment of the method of the present invention, the overall disparity estimation mechanism obtains the corrected image sum by the following formula (1.1) and formula (1.2) by calculating the sum of absolute differences (SAD) of the overall disparity information of each corrected image. The overall parallax information of the reference image. The overall disparity information includes: the displacement amount d and the position (including the horizontal position x and the vertical position y). In the following formulas (1.1) and (1.2), R represents a reference image, and T represents a corrected image (Corrected View), which means a view image other than the reference image (if not specified, the corrected image mentioned below is Refers to other perspective images other than the reference image).

SAD _global ( R , T , d )=Σ _{( x , y )} | R ( x , y )- T ( x + d , y )|(1.1)

SAD _global ( R , T , d )=Σ _{( x , y )} | R ( x , y )- T ( x , y + d )|(1.2)

According to an embodiment of the method of the present invention, wherein the SAD value of each displacement amount d is calculated by re-transmission, and the minimum SAD value is found from the SAD value of each displacement amount d by using the above formula (1.3) to find a reference. The largest similar area between the image and each corrected image. And, the image is cropped for each corrected image, and the similar area is retained. Image information, delete image information outside the similar area. Therefore, image quality can be improved and the error rate of block matching can be reduced. In other words, the smaller the SAD value of the displacement amount d, the closer the similarity of the two images is. The method of the present invention considers the horizontal axis and the vertical axis displacement, and the position of the horizontal axis position (x) and the vertical axis (y) are also obtained after calculating the SAD value. The image of the corrected image can be cropped according to the overall parallax information, and the image information of the similar area is retained, and the image information outside the similar area is deleted. In other words, the image information required is retained, and unnecessary image information is deleted to achieve a complete effect.

For example, as shown in the following figure, the "two images shown in Figure 3A" are used to calculate the SAD values of different displacement amounts d through the overall parallax estimation mechanism. (See Figure 3B, SAD value distribution map). It can be found that the minimum SAD value is 977264 when the displacement amount d is equal to 6. That is to say, when the displacement amount is 6, the two images have the most similar regions. Among them, the most similar areas of the two images can be as shown in the range selected in Figure 3A.

According to an embodiment of the invention, performing image cropping on each of the corrected images includes deleting the leftmost and rightmost image information of the corrected image (ie, the image except the most similar region). For example, in the example shown in Figure 4, when shooting a multi-view image with multiple cameras, the leftmost image of the view image View1 taken by the camera on the far left (the shaded portion of View1 in Figure 4), and The rightmost image of Viewn image captured by the rightmost camera (the shaded portion of Viewn in Figure 4) cannot be captured by other cameras, and this part of the image is different from each view image and can be deleted.

[Verification step of image brightness change affecting displacement]

In an embodiment of the method of the present invention, the step of verifying the displacement of the image brightness affects the displacement, and the main function is to verify whether the image has a different displacement value calculated before and after the brightness change due to the brightness change. One implementation of this verification step is to first change the brightness of the image through an image software to change its brightness information, and then use the SAD formula to calculate whether the minimum SAD value is different. The results of the experimental data confirm that the SAD value will increase and decrease relatively when the brightness of the image changes, but the information of the displacement amount does not differ much, and is close to the original displacement. In other words, the method of the present invention utilizes the overall disparity estimation mechanism to solve the problem of image displacement and image information difference caused by different shooting angles in each view image in a multi-view image, and is indeed feasible.

According to an embodiment of the method of the present invention, after the reference image is acquired, the “verification step of affecting the displacement of the image brightness change” is performed, and then the overall parallax information of each corrected image is calculated according to the reference image by using the global disparity estimation mechanism. A detailed implementation of the verification step will be exemplified below with respect to the circuit architecture embodiment of the overall disparity estimation unit 20.

[Embodiment of Brightness Correction Step]

The reference image and the corrected image after the image is cropped are obtained, and then the brightness correction is prepared. The brightness correction corrects the brightness information of each corrected image by using an adjustable brightness correction mechanism based on the average brightness of the reference image, so that the brightness average of each corrected image after the brightness information is corrected is closest to the brightness average of the reference image. One implementation manner of the adjustable brightness correction mechanism is an adjustable histogram equalization algorithm (Adjustable Histogram Equalization) proposed by the present invention. Correct the brightness information of each corrected image. The adjustable histogram equalization algorithm includes: a Probability Density Function (PDF) (see Equation 2.1 below), an Adjustable Cumulative Distribution Function (see Equation 2.2 below), and the last use. A conversion function to (see Equation 2.3 below).

The Cumulative Distribution Function (CDF) of the present invention adds a variable coefficient c operation and a cumulative distribution function (CDF) to subtract the absolute value and then rearranges it. In the embodiment of the present invention, the brightness information is adjusted multiple times on the corrected image by sequentially adjusting the variable coefficient c, and the average value of each brightness after the corrected image is adjusted multiple times of the brightness information is calculated, and then from the total brightness average value. Finding the brightness average closest to the reference image is the corrected image for the brightness information correction.

Hypothesis The brightness component of the original corrected image. Rearrange the order of the brightness values, arrange the brightness values from 0 to 255, and use the formula (2.1) to find the probability distribution function (PDF). Then use the Cumulative Distribution Function (see Equation (2.2)) and accumulate the results. After the calculation, the conversion function is used to see the formula (2.3) above, and the brightness average value of the brightness information of the corrected image is adjusted to the brightness average value closest to the reference image, that is, the corrected image of the brightness information correction is completed.

f ( X _k )= X _k +( X _L - X _k ) CDF ( X _k )K=0,1,...L-1 (2.3)

Where n represents the sum of the image brightness, nk represents the individual statistical number of image brightness, and the variable coefficient c is a non-zero value.

In an embodiment of the invention, the conversion function further comprises: a reduced conversion function as shown in the following formula (2.4). In addition to increasing the brightness by using the above formula (2.2), the brightness can also be reduced to achieve the full function of image processing.

f ( X _K )= X _K -( X _L - X _K ) CDF ( X _K )K=0,1,...L-1 2.4)

Hereinafter, through the different variable coefficients c, the effect of correcting the luminance information by the adjustable histogram equalization algorithm proposed by the present invention will be described.

The example shown in "5A" to "5C" is the brightness correction result of the variable coefficient c=0. "Picture 5A" is a statistical diagram of the reference image. "Picture 5B" is a statistical diagram of the image before correction. "5C Figure" is the corrected image chart. The horizontal axis is the luminance value and the vertical axis is the luminance statistical value. When the variable coefficient c is 0, the average brightness of the reference image is 59.342, the average brightness of the image before correction is 46.387, and the average value of the corrected image brightness is 103.21, so it can be found that the corrected image after brightness modification has brightness. Too strong a problem.

The example shown in "Fig. 6A" to "6C" is the brightness correction result of the variable coefficient c = 1. Figure 6A is a statistical diagram of the reference image. Figure 6B is a statistical diagram of the corrected image. Figure 6C shows the corrected image statistics. After the corrected image brightness average is 49.224, we can see that the data of the chart is slightly changed, but the corrected image brightness average is still not close to the brightness average of the reference image, so continue to change the variable coefficient c. Adjustment.

An example shown in "Ath 7A" to "7C" is a brightness correction result of a variable coefficient c = 0.84. "Picture 7A" is a statistical diagram of the reference image. "Picture 7B" is a statistical chart of the corrected image. "Picture 7C" is the corrected image chart. The corrected image brightness average is 59.128. The extended part of the image is similar to the reference image and close to the brightness average of the reference image. The above example can reveal the adjustable brightness correction mechanism proposed by the present invention. By adjusting the variable coefficient c one by one to adjust the brightness information of the corrected image multiple times, calculate the average value of each brightness after the corrected image is adjusted multiple times, and then find the closest reference image from the total brightness average. The average brightness is the corrected image of the brightness information correction.

[Implementation of color correction steps]

In a three-dimensional display device (such as a stereoscopic television), the color information of the image is represented by a YUV color format, and in order to facilitate color correction of the corrected image that completes the brightness information correction, in one embodiment of the present invention, the multi-view image is initially used. The image color information contained in the plurality of different viewing angle images View1~Viewn obtained in the image is the color information of the YUV color space.

In general, the color information of an image taken by a camera is expressed in an RGB color format, so if a multi-view image using the RGB color format is applied to the apparatus and method of the present invention, multi-view of the RGB color format should be handled. The image is first subjected to color domain transformation. The applied color space conversion formula is determined by the color format of the multi-view image before conversion and the color format required after conversion. The color conversion formula listed below is used for explanation. its One of them is not the only example of this conversion formula.

In an embodiment of the method for performing color space conversion, the RGB color space of the reference image can be converted into the YUV color space by using the following formula (4.1), mainly for conforming to the input information of the 3D stereo TV:

Then use the following formula (4.2) to convert the RGB color space of the corrected image to the YUV color space:

One embodiment of the color correction mechanism proposed by the present invention redefines the above formula (4.1) and formula (4.2) by the following color correction formula (4.3) as the color correction mechanism of the present invention, and the formula (5.1) can be seen.

The following formula (5.1) where VR and UR represent the color information of the reference image, and VT and UT represent the color information of the corrected image.

Simplified color correction information For the VR color information of the reference image, Refers to the UR color information of the image. The invention uses the color range for processing. When the color correction information is in a certain range, the correction value is directly filled in with the corresponding color value to avoid the color distortion problem.

An embodiment of the color correction in the method of the present invention performs color correction on the corrected image adjacent to the left and right of the reference image, and performs color correction processing on the reference image and the shadowing effect image in the event of a shadowing effect in the correction process. This gives the blocked image information. Therefore, problems caused by the shadowing effect of the image can be avoided.

In the method of the present invention, the above-described embodiment is adopted, and the color correction is performed first by using the corrected image adjacent to the left and right of the reference image, and then the color correction is performed to the adjacent other corrected image after the correction. In this way, because the amount of displacement adjacent to each other is not larger than the amount of displacement calculated by the reference image for each corrected image, the obtained correction information is relatively improved, and the color correction of the image is also effective. . Therefore, it is possible to avoid excessive shift of the displacement amount and distortion of the color information after the image correction.

It can be understood from the above description that the method of the present invention uses the global disparity estimation mechanism to find the maximum similarity between the reference image and each corrected image, and then corrects the brightness and color. In the color correction step, the reference image is adjacent to the reference image. After correcting the image correction, use the corrected image to correct the adjacent other Correcting the image and then performing color correction can reduce the displacement information and improve the quality of the correction of the color information of the corrected image, so that better results can be obtained.

The technical features of the apparatus for carrying out the above method and the embodiments thereof are set forth in the present invention, as explained later.

The reference information processing unit 10 has an input side and a multi-view image source connection for acquiring multi-view images View1~Viewn, and the reference information processing unit 10 first calculates the brightness average value of each view image, and then selects the brightness average value from the most. A threshold value is used as a reference image ViewR, and the remaining view images are used as a plurality of corrected images, and then the reference image information (ViewR information) and the corrected image information are output, wherein the reference image information includes at least the brightness average value of the reference image, and the correction is performed. The image information includes at least the color information Y, UT, and VT of each corrected image after the color gamut conversion.

There are many possible implementations of the reference information processing unit 10. Here is an example of a circuit architecture for selecting a reference image and an average of the brightness of the output reference image. As shown in Fig. 8, the multi-view image is input to an 8-bit right shifter 11 (8 bit right shift) to avoid the input of the brightness value through the accumulator 12 (accumulator). After the multi-bit scratchpad is needed to store, the first division operation is performed, so that the required number of temporary storage bits is up to eight bits. After the accumulation is completed, it is subtracted from the threshold value 128 by a subtractor 13 (subtractor). Then, the absolute value is taken by the absolute value encoder 14 (absolute) in order to make the calculated value smaller, that is, the value closer to the threshold value 128. Then compare the comparator 15 (comparator minimum value) to find the minimum brightness average Reference image. Then, the reference image information is sent to the overall parallax estimating unit 20, and the brightness average value of the reference image is sent to the brightness correcting unit 30.

As shown in FIG. 2, the overall parallax estimating unit 20 has an input side and a multi-view image source connection for acquiring multi-view images View1~Viewn. The reference data input terminal and the reference information processing unit 10 are connected to obtain the reference image information output by the reference information processing unit 10. The overall disparity estimation unit 20 calculates the overall disparity information of each corrected image according to the reference image by using the overall disparity estimation mechanism, and performs image cropping on each view image according to the overall disparity information of each corrected image. Among them, performing image cropping on each view image retains image information of a similar area and deletes image information other than the similar area.

The overall parallax information of each view image generated by the overall disparity estimation unit 20 can be transmitted to the brightness correction unit 30 (see FIG. 9) through the memory controller 41, and the overall disparity information is as shown in FIG. The displacement amount d of each of the corrected image and the reference image is included.

In conjunction with the "Verification Step of Image Brightness Effect Affecting Displacement" in the foregoing method of the present invention, a circuit architecture implementation example of the overall parallax estimating unit 20 is shown here, for example, "Fig. 9", which uses a reference image ViewR and other viewing angle images. The minimum SAD value is calculated, and the calculation of the two SAD values is performed using a set of test images at the beginning. The calculation of the first SAD value is performed by using the original test image and the reference image through the path of the calculation circuit shown in the lower part of "Fig. 9", and the original test image and the reference image are utilized by a first subtractor 21a. First absolute The value encoder 22a and a first accumulator 23a calculate the first set of displacement amounts d1 having the smallest SAD values. The second SAD value is calculated by first adding a predetermined brightness value L1 to the test image except the reference image. As shown in FIG. 9, the brightness is increased by an adder 24, and then the reference image is calculated by using a second subtractor 21b, a second absolute value encoder 22b, and a second accumulator 23b. Two sets of displacements d2 having a minimum SAD value. Finally, the calculated displacement amount d1 and the displacement amount d2 of the two groups are checked by the comparator 25 to verify whether the two groups obtain the same displacement amount, thereby verifying whether the image after the brightness change affects the displacement amount. Calculation.

The brightness correction unit 30 has an input side and a reference information processing unit 10 connected to obtain a brightness average value of the reference image. The brightness correcting unit 30 corrects the brightness information of each corrected image after the image is cut by the adjustable brightness correction mechanism based on the average brightness of the reference image, so that the brightness average of each corrected image corrected by the brightness information is closest to the reference. The average brightness of the image, and then output the corrected image after brightness correction and its brightness correction information.

The basic core of the brightness correction unit 30 is to implement the adjustable brightness correction mechanism proposed by the present invention. According to the foregoing description, the adjustable brightness correction mechanism of the present invention includes: a probability distribution function (PDF), an adjustable cumulative distribution function, and a conversion function. Several parts. A circuit architecture implementation of these three parts is shown in "Figure 10" to "Figure 11". Among them, a circuit structure of the probability distribution function (PDF) and the adjustable cumulative distribution function is shown in Fig. 10. When the brightness value of the image is input, the brightness value needs to be counted, and a large number of comparators 31 are used to set 0~ The 255 parameter is used as the trigger condition. When the input image is the same as the value of the comparator, the back end is used. The incrementer 32 (incrementer) adds 1 for accumulation, and then temporarily accumulates the result by the register 33a (register) and the shift register 33b (shift register) (that is, the result of the probability distribution function), and the entire image is transmitted. After all, all the image information is also counted.

After calculating the probability distribution function, the adder 36a is used to add each value to obtain an accumulated distribution function value (CDF), and according to the adjustable cumulative distribution function proposed by the present invention, a variable coefficient c and an accumulated distribution are added. The function (CDF) is subtracted by the subtractor 34 and then re-arranged by taking the absolute value by the absolute value encoder 35. A new cumulative distribution value CDF can be obtained by sorting the sorting circuit S1 in a small to large manner, and the effect of brightness increase is performed by a conversion function (see equation (2.3) in the foregoing description). The value of the accumulated distribution function obtained when the variable coefficient c changes once also differs.

After passing through the circuit structure of the conversion function (see "Figure 11"), the circuit structure of the conversion function increases the luminance value one by one according to the coefficient of the accumulated distribution. After calculating the brightness image information, the reference information processing unit 10 can also be used to calculate the brightness average value, and then the calculated result is subtracted and compared with the brightness average value of the reference image by the comparator 37. Among them, the smaller the value, the more the brightness information of the two images is the same. Finally, a correction luminance value (CLV) that is closest to the correction information is found and transmitted to the storage unit 40 for storage. When the next image is sent in and out, the brightness correction information can be directly read out and corrected.

Further, in the method of the present invention, a circuit structure in which the luminance information has a reduced conversion function for reducing the luminance function is shown as shown in Fig. 12. Among them, the circuit structure of the conversion function of "12th picture" and "11th picture" is different from CDF After multiplying by the multiplier 39, the subtractor 38 is used to subtract the input information to reduce the brightness correction information. The circuit structure of the conversion function of "Fig. 11" is used after multiplying the CDF by the multiplier 39 by adding the addition information to the input information to reduce the brightness correction information.

One embodiment of storage unit 40 includes a memory controller 41 and a plurality of memory SRAM1 through SRAM 4. The memory controller 41 is for controlling the reading and writing operations of the four memory SRAM1 to the SRAM 4, and writing the overall parallax information of the respective corrected images output from the overall parallax estimating unit 20 to the first memory SRAM1. The first memory SRAM 1 is used by the brightness correcting unit 30 for brightness information correction (see "Fig. 13"). The second memory SRAM 2 is used to store the corrected images and the brightness correction information corrected by the brightness information. In other words, the second memory SRAM 2 is used to store the reference image and the adjacent corrected image corrected image information. When the data is stored in the memory, when other corrected images (ie other perspective images) are input, we can read the data of the second memory SRAM2 for modification.

As shown in Figure 14, the third memory SRAM3 can store the U color space information UR of the reference image and the color correction information of the U color space after the color correction is stored. . As shown in Figure 15, the fourth memory SRAM4 can store the V color space information VR of the reference image and the color correction information of the V color space after the color correction. .

The color correction unit 50 has an input side connected to the reference information processing unit 10. The color correction unit 50 is configured to obtain color information Y, UT, and VT of the corrected image (that is, each view image), and use the color correction information of the U color space. , color correction information of V color space And the overall parallax information of each corrected image corrects the color information of the mapped position in each corrected image.

In an embodiment of the apparatus of the present invention, in addition, in order to compare the actual data before and after the correction, the present invention compares the reference image with the information before and after the correction according to the statistical method used, and can compare the obtained information. Find out if the information before and after the correction is the same. Among them, an implementation manner of the U color space data statistical circuit architecture is shown in "16th picture". An implementation of the U color space data statistics circuit architecture after color correction is shown in Figure 17.

Embodiments of the present invention utilize an overall disparity estimation mechanism to find the region of greatest similarity between a reference image and a corrected image. After finding the maximum similar area of the reference image and the corrected image through the image cropping, in order to further improve the corrected image quality, the method and the device of the present invention use a reference image as a standard, and use the reference image and each corrected image to perform comparison. Correct. Next, the correction and the color are corrected for each of the corrected images. When the brightness average of the corrected image after the brightness information correction is completed, the closer to the brightness average of the reference image, the better the correction information is. The color correction algorithm allows each corrected image information to be close to the reference image.

It can be seen from the above description that the method and device of the present invention can re-correct image information of multi-view images, reconstruct image brightness and color information, and delete unnecessary image information to improve image quality and reduce image quality. The error rate of block matching. Therefore, the image quality of the multi-view image is improved, and the multi-view image is output to the three-dimensional composite software to establish a stereoscopic image.

While the present invention has been described above by way of example, the present invention is not intended to limit the invention, and the invention may be modified and modified without departing from the spirit and scope of the invention. The scope of patent protection is subject to the terms of the claims attached to this specification.

10‧‧‧Reference Information Processing Unit

20‧‧‧Integral Parallax Estimation Unit

30‧‧‧Brightness correction unit

40‧‧‧ storage unit

41‧‧‧ memory controller

50‧‧‧Color Correction Unit

D‧‧‧displacement

View1~Viewn‧‧‧ view image

VT‧‧‧Fixed V color space information after image space conversion

UT‧‧‧Fixed U color space information after image space conversion

Y‧‧‧Reference image/correction image brightness information

SRAM1‧‧‧ first memory

SRAM2‧‧‧ second memory

SRAM3‧‧‧ third memory

SRAM4‧‧‧ fourth memory

Claims (10)

A multi-view image correction device includes: a reference information processing unit, configured to receive a plurality of view image images, select one of the view image images as a reference according to a brightness average value of each view image and a threshold value The image and the remaining view images are used as a plurality of corrected images, wherein the view angles correspond to different viewing angles; an overall disparity estimating unit is electrically connected to the reference information processing unit for utilizing an overall disparity estimation mechanism according to The reference image is used to calculate the overall disparity information of each of the corrected images, and to perform image cropping on each of the view images according to the overall disparity information of each of the corrected images; a brightness correcting unit is electrically connected to the reference information processing unit for The brightness average of the reference image is corrected by an adjustable histogram equalization algorithm to correct the brightness information of each of the corrected images, so that the brightness average of each corrected image after the brightness information is corrected is closest to the reference. An average brightness of the image; and a color correction unit electrically connected to the reference The processing unit and the brightness correcting unit are configured to generate a color correction information by using the color correction mechanism after the corrected image of the entire parallax information closest to the reference image is started according to the brightness information and the previous image is corrected by a color correction mechanism. The color information of the corrected image corrected by the corresponding brightness information is selectively corrected according to the color correction information. A multi-view image correction device as claimed in claim 1, wherein The overall disparity information calculated by the overall disparity estimation mechanism has at least one displacement of each of the modified image and the reference image, and the overall disparity estimation mechanism calculates a sum of absolute differences of the total disparity information of each of the corrected images ( And obtaining, by the SAD, the displacement amount of each of the corrected image and the reference image, wherein the brightness correction unit further finds the reference image and each of the corrected images by calculating a SAD value of each of the displacement amounts and searching for a minimum SAD value thereof. The largest similar area between the two, the image is cut for each of the modified images, the image information of the similar area is retained, and the image information other than the similar area is deleted, wherein the image cropping of each of the view images further includes the leftmost of the view images. The rightmost image information is deleted. The apparatus for correcting multi-view images as described in claim 1, wherein the color correction mechanism obtains the color correction information by the following formula with A method for correcting a multi-view image includes: selecting one of the view images as a reference image according to brightness of the plurality of view images, wherein the remaining view images are used as a plurality of corrected images; and utilizing an overall disparity estimation mechanism Calculating the overall parallax information of each of the corrected images according to the reference image, and determining the overall view of the corrected image according to the reference image The difference information is used for image clipping of each of the corrected images; and the brightness of each of the corrected images after the clipping is corrected by an adjustable brightness correction mechanism based on the average brightness of the reference image, so that the corrected images of the corrected brightness information are corrected The brightness average value is closest to the brightness average value of the reference image; and the corrected image that is closest to the reference image by the overall disparity information starts to be generated according to the brightness information, and the previous image is generated by a color correction mechanism. a color correction information, and selectively correcting color information of the corrected image corresponding to the brightness information according to the color correction information. The method for correcting a multi-view image according to claim 4, wherein the reference image is set by selecting a brightness average value closest to a threshold from the plurality of view images. The method for correcting multi-view images as described in claim 4, wherein the overall disparity estimation mechanism obtains each of the corrected images and the reference by calculating a sum of absolute differences (SAD) of the total disparity information of each of the corrected images. A displacement amount of the image is obtained by calculating the SAD value of each of the displacement amounts and finding the SAD value of the displacement amount to find the maximum similarity between the reference image and each of the corrected images, and performing image cropping on each of the corrected images. The image information of the similar area is retained, and the image information other than the similar area is deleted, wherein performing image cropping on each of the plurality of view images includes deleting the leftmost and rightmost image information of the plurality of view images. The method for correcting a multi-view image as described in claim 4, wherein the adjustable brightness correction mechanism performs brightness adjustment on the corrected images through an adjustable histogram equalization algorithm, the adjustable histogram, etc. The algorithm includes: a probability distribution function, an adjustable cumulative distribution function, and a conversion function. The method for correcting a multi-view image according to claim 7, wherein the adjustable cumulative distribution function is re-arranged after subtracting an absolute value from a cumulative distribution function by adding a variable coefficient to an accumulated distribution function. Adjusting the variable coefficient successively to adjust the brightness of the corrected image multiple times, calculating the average brightness value of the corrected image after adjusting the brightness multiple times, and finding the closest to the reference image from all the average values of the brightness. The brightness average is used as the corrected image for the brightness correction. The method for correcting a multi-view image as described in claim 4, wherein the color correction mechanism obtains the color correction information by the following formula with The method for correcting a multi-view image according to claim 4, wherein the color correction step corrects the corrected image adjacent to the reference image, and then uses the corrected image pair adjacent to the correction. The other corrected images are then color corrected.