JP6893068B1 - Image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image process for synthesizing an image subjected to sharpening processing and an image subjected to blackout processing. SOLUTION: In performing image processing for synthesizing an image subjected to sharpening processing and an image subjected to blackout processing, a dark portion of the image is blackened by using a flat frame used in the blackout processing. By increasing the ratio of sharpening processing and increasing the ratio of sharpening processing in bright areas, the drawbacks of sharpening processing, that is, the disadvantages of tending to have excessive contrast in dark areas of the image, and the disadvantages of blackout processing, that is, contrast in bright areas. Processing that makes up for the shortage becomes possible. [Selection diagram] Fig. 1

Description

The present invention relates to an image processing method for synthesizing an image that has been subjected to a sharpening process for converting an image in which an extremely low contrast portion is mixed into an easy-to-see screen and an image that has been subjected to a correction process for blackout of the image.

Clarification processing that converts an image with extremely bright and dark parts mixed in the same screen such as an indoor window or an image with extremely low contrast parts such as underwater and fog into an easy-to-see screen, and a shutter speed of 1 / Conventionally known methods are used for nighttime shooting with a video camera that cannot be lengthened by 30 seconds or more, and a method for dealing with the blackout phenomenon that occurs when the exposure, gain, and aperture are limited when the subject is backlit by the automatic exposure function.

As the conventional sharpening process, there are Patent Documents 1 to 5. Patent Document 1 describes an image pickup element that photoelectrically converts an optical image and outputs an electric signal, a signal processing means that processes the electric signal output from the image pickup element to generate a video signal, and the signal processing means. An image pickup device including a histogram circuit that generates a histogram from an output video signal and an exposure control means that controls exposure using the histogram detected by the histogram circuit is disclosed.

Patent Document 2 describes a step of reading light from a manuscript to generate image data, a step of creating a density distribution histogram from the image data, and a step of the density distribution near both ends of the density distribution with respect to the total number of data of the image data. An image processing method including a step of generating a density correction curve based on the ratio of the number of data and a step of performing density correction of image data with the density correction curve is disclosed.

Patent Document 3 describes an imaging means that captures a subject and acquires image data of the captured image, and a predetermined pixel portion of the captured image composed of the image data acquired by the imaging means, and the brightness level is in a predetermined range. For the pixel portion consisting of the pixels inside, the brightness value of each pixel in the pixel portion is converted so as to increase the brightness interval between the pixels while maintaining the vertical relationship of the relative brightness between the pixels. An image pickup apparatus including a gradation correction means for performing adjustment correction is disclosed.

Patent Documents 4 and 5 are proposals of the present inventor, and the content of Patent Document 4 is that image data in pixel units is captured in one pass from a captured image, and the captured image data is decomposed into a specific color space in pixel units. After that, a brightness histogram is generated, this image brightness information is read out with a predetermined reading pattern, and the pixels at the specific position of the reading pattern are based on the average histogram excluding the pixels at the specific position (center) in the reading pattern. It is a method of setting the brightness of.

In Patent Document 5, when the tone map is obtained by locally dividing the input signal and the contrast of the input signal is improved by correcting the tone map, the input luminance signal of the tone map is used for a region where the luminance change is small. Is converted into an output luminance signal, a gradient limit is provided so that the output luminance signal does not change beyond a certain level to suppress the luminance change, and further, the brightness of the entire region reduced by the luminance limitation is adjusted as a whole. Is disclosed.

Patent Documents 6 to 9 are available as conventional blackout correction processing. Patent Document 6 describes a method of restoring an image taken in a situation where the brightness difference is large such as backlight. Specifically, as described in paragraphs (0038) and (0039), a histogram is created, and when the frequency at zero luminance value is larger than the threshold value, it is determined that the image is underexposed, and that portion is determined. It is described to correct the brightness of.

In Patent Document 7, when the brightness difference between the dark part and the bright part of the image becomes large, the normal imaging mode is switched to the composite imaging mode, and blackout correction is performed to reduce the brightness difference. It is described that each time-exposure image signal is automatically exposed and controlled.

In Patent Document 8, a long-exposure image signal having a long exposure time and a short-exposure image having a short exposure time are combined, a luminance integrated value and a histogram are generated in the composite image signal, and blackout in the composite image signal is generated from the luminance histogram. It is described that the target luminance integrated value is set based on the detection result, and the exposure compensation control of the imaging unit is performed using the target luminance integrated value.

Patent Document 9 discloses an imaging device capable of acquiring the remaining battery capacity. This imaging device is provided with a determination means for displaying a battery segment when the remaining capacity of the battery is low or when the battery is out of order. In this determination means, the number of black or white pixels exceeds a predetermined value. When present, it is stated that the determination is made to display the histogram.

Japanese Unexamined Patent Publication No. 2002-142150 Japanese Unexamined Patent Publication No. 2003-051944 Japanese Unexamined Patent Publication No. 2007-124087 Japanese Patent No. 4386959 Japanese Patent No. 6126054 Japanese Unexamined Patent Publication No. 2011-223173 JP-A-2002-084449 Japanese Unexamined Patent Publication No. 2008-228058 Japanese Unexamined Patent Publication No. 2016-092798

The process disclosed in Patent Documents 1 to 3 described above takes a long time. According to the methods disclosed in Patent Documents 4 and 5, the processing time can be shortened and real-time processing of moving images becomes possible. However, even with the method described in Patent Document 4, there may be a case where the variation in brightness is too large or noise cannot be removed.

The contents disclosed in Patent Document 5 include reading captured image data such as AHE (Adaptive Histogram Equalization) and CLAHE (Constract Limited AHE) in a predetermined pattern, dividing the captured image into a plurality of blocks, and dividing each small area. Histogram is generated and a tone map is created for each area.

When a tone map is created for each small area in this way, the boundary line of the small area may be vaguely visible.
Furthermore, since AHE and CLAHE scan images to create a histogram for each area and then create a tone map, a lot of high-speed memory is required for real-time processing, and independent processing is performed on a pixel-by-pixel basis. It can be said that it is not suitable for GPU and FGPA because it cannot be done.

In Patent Documents 6 to 9, a histogram of brightness is created and the histogram is flattened when performing backlight and shadow correction (blackout correction). In order to flatten the histogram, it is necessary to count the histogram of the entire screen, which requires a large memory and increases the treatment time.

Further, the sharpening process has a problem that the contrast becomes excessive in a dark part, and the blackout correction process tends to lack the contrast in a bright part. Not proposed.

The image processing method according to the present invention in order to solve the above problems is an image processing in which an image subjected to sharpening processing and an image subjected to blackout processing are combined, and the flat frame used in the blackout processing is used. Using the principle, the dark part of the image has a higher ratio of blackout processing, and the bright part has a higher ratio of sharpening processing.

In the specific example of the sharpening process, when processing the captured image data on a pixel-by-pixel basis, n reference pixels (n is an integer) are set around the pixel of interest, which is the scanning position of the image scan, and the pixel of interest The pixel value (brightness) and the pixel value (brightness) of each reference pixel are sequentially compared, the number of reference pixels that were less than or equal to the pixel value of interest is counted, and in parallel with this, the histogram value of each reference pixel is calculated. Compare with the tilt limit value set in advance by +1 and count the number of reference pixels that satisfy both the condition that the reference pixel value is less than or equal to the attention pixel value and the condition that the pixel histogram value is less than or equal to the tilt limit value. By doing so, the true count values of both sides are obtained, and the true count values of both sides are proportionally divided and output.

In a specific example of the blackout processing, a Blur value with blurred brightness is obtained by performing Gaussian Blur processing on the brightness of each pixel in the Y plane (luminance plane) memory of the input image, and this Blur value is normalized to 0. The distribution information takes a value of ~ 1.0, and a threshold value is set between 0 and 1.0, which is a normalized value. All pixels with a value larger than this threshold value are set to 1.0, and dark areas with a value smaller than the threshold value are set. Determines the brightness magnification (n) of the darkest pixel, corrects the distribution information so that the inverse number (1 / n) is the lowest value of the distribution information, and corrects the brightness distribution information of the dark part (1 / n to 1.0). Is the denominator to divide the brightness of the input image.

According to the present invention, it is possible to simultaneously solve the problems that remain unsolved even in the sharpening process and the blackout correction process.

The present invention presupposes that the captured image data is processed independently in pixel units without being divided into small areas. That is, since the histogram for each area is not generated, the processing speed is greatly increased and the corrected image can be displayed in real time.

Instead of actually using a lot of memory to create a histogram and tonemap for each area as in the past, you can get the same result as creating a histogram and tonemap by executing logic.

No memory for histograms and tone maps is required, and implementation is possible using only the line buffer. Since the delay is only for the line buffer and does not require full-screen statistics such as histogram acquisition, there is no delay in frame units.

Since the logic itself is simple, it can be mounted in an empty area of a small FPGA or other video circuit, and because there is no logic or memory proportional to the pixel depth (number of bits) such as a histogram array, it has a high bit depth of 36 bits or 48 bits. But the circuit scale is almost the same.

A block diagram illustrating the basics of a logical image processing method according to the present invention. (A) is a diagram explaining the horizontal Blur processing in the blackout processing, and (b) is a diagram explaining the vertical Blur processing. Diagram explaining the basics of sharpening processing Virtual tone map averaging histograms of reference pixels corresponding to FIG. The figure which shows the implementation example of the sharpening process Figure 5 Corresponding virtual tone map Diagram showing another implementation example of sharpening processing Virtual tone map corresponding to FIG. Diagram showing another implementation example of sharpening processing Virtual tone map corresponding to FIG. Diagram showing another implementation example of sharpening processing Virtual tone map corresponding to FIG. Diagram showing another implementation example of sharpening processing Virtual tone map corresponding to FIG. Diagram showing another implementation example of sharpening processing Virtual tone map corresponding to FIG. (A) is the original image, (b) is the image after the blackout processing, (c) is the image after the sharpening processing, and (d) is the image after image composition (the present invention).

As shown in FIG. 1, the input image is subjected to blackout processing (FC: flat collector) and sharpening processing in parallel, and the image after blackout processing and the image after sharpening processing are used for blackout processing. It is synthesized and output using the principle of the flat frame that was used.

In the blackout process, as shown in FIG. 1, the input image is color-separated, and the tint (CbCr), the brightness (Y), and the three primary colors (RGB) are extracted. In the present invention, the brightness (Y) is subjected to Blur processing and flat processing, and color combination with tint (CbCr) and three primary colors (RGB) is performed to obtain an output image. In the present invention, a flat frame (an image having uniform brightness) is not created, but only the principle of the flat frame is used.

In the Blur process, a roughly variable image of brightness, that is, an image with blurred brightness is created. In this example, Gaussian Blur processing was performed to blur the image using a Gaussian function.

In the Gaussian Blur processing, for example, the horizontal Blur processing shown in FIG. 2A is performed, and the vertical Blur processing shown in FIG. 2B is performed on the horizontal Blur-processed line buffer to perform the blur plane. To create.

At the time of the horizontal Blur processing, the minimum value (Ymin) of the input video is obtained by averaging the past 4 frames of the minimum value of the input image.
Further, while performing Gaussian Blur processing, the minimum value (Bmin) and the maximum value (Bmax) of the Blur image are obtained.

In implementing the logic of the present invention, Gaussian calculation is not performed using a Gaussian table with radius R. Further, the radius R is 30 pixels at the maximum, and 62 line buffers (30 × 2 + center 1 + horizontal calculation 1) are used.

In the illustrated example, since the kernel size is 61 × 61, the Gaussian Blur processing is divided into horizontal Blur and vertical Blur in order to reduce the processing area, but a kernel filter may be simply used without dividing.

After creating the Blur plane by the above Gaussian Blur process, generate a flat frame. Generated from the Blur plane and the lowest input video value (Ymin). As described above, the present invention does not actually create a flat frame, but merely uses the principle of the flat frame.

In flat frame generation, the Blur plane is normalized to be distribution information that takes a value from 0 to 1.0, a threshold is set between the normalized values 0 to 1.0, and all pixels with values larger than this threshold are set. Set it to 1.0, determine the brightness magnification (n) of the darkest pixel for the dark pixel with a value smaller than the threshold value, and modify the distribution information so that the reciprocal (1 / n) is the lowest value of the distribution information.

For example, if 0.5 is set as the threshold value, among the normalized values 0 to 1.0, 0 to 0.5 are 0 to 1.0, and 0.5 and above are all 1.0.
By performing such correction, pixels with a brightness of 0.5 or more in the input image will be 1.0 in the distribution information, and will change from the original image in later processing (dividing the input image with the brightness distribution information as the denominator). do not. On the other hand, since the denominator of the dark part gradually becomes smaller, the dark part is corrected so that the darker the original image, the higher the magnification.

Flat processing is performed following flat frame generation. In this flat processing, the current frame is processed from the Blur plane and the input video minimum value (Ymin). That is, it is a process of dividing the luminance (Y) and the three primary colors (RGB) of the input image by the Blur plane.

When the vertical Blur value of the pixel of interest is determined in the above, processing from the creation of the Blur plane to the acquisition and correction of distribution information becomes possible, so real-time processing can be performed with only the delay of the line buffer without using the frame buffer. It is possible.

Flat processing includes normal processing and color burst processing. In normal processing, only brightness (Y) is processed and combined with tint (CbCr) before output, and in color burst processing, instead of brightness (Y). The same calculation is performed for the three primary color (RGB) planes.

The general calculation formula for flat processing is as follows.
Output image F (x, y) = Input image Y (x, y) * 256 / Flat frame Blur (x, y)
256 is when the bit depth is 8 bits

When the Blur plane correction (distribution information correction) formula is applied to the above formula, the output image becomes the following formula.

Next, the sharpening process will be described with reference to FIGS. 3 and later. In FIG. 3, P0 is a pixel of interest, and the brightness (luminance) of the pixel of interest P0 is adjusted. Further, FIG. 4 is a virtual tone map in which the execution results obtained from the reference pixels (8 in P1 to P8 in FIG. 3) when processed by the method of FIG. 3 are graphed at all the brightnesses that the reference pixels can have. Is.
The present invention obtains only one conversion result from a pixel of interest and a reference pixel to a pixel of interest, and does not output a tone map.

The processing procedure first compares the brightness of the attention pixel P0 and the reference pixels P1 to P8 around it, counts the number of reference pixels having a brightness smaller than the brightness of the attention pixel P0, and counts the number of reference pixels having a brightness smaller than the brightness of the attention pixel P0. The brightness of is corrected according to a predetermined algorithm.

For example, when the number of reference pixels having a brightness smaller than the brightness of the pixel of interest P0 is one and the number of reference pixels having a brightness larger than the brightness of P0 is seven, the maximum brightness that can output the value of the reference pixel is 1. Correct the brightness to 1/8. The correction algorithm is not limited to this.

When implementing the above operation on FPGA or CPU, move the pixel of interest one by one in the row direction, and perform this process in parallel for each row to correct the brightness of all pixels and smooth the brightness. Do.
When mounting on a GPU, these operations are independent for each pixel, so the brightness of all pixels is corrected and the brightness is smoothed by performing parallel processing on multiple cores at the same time.

On the other hand, the virtual tone map shown in FIG. 4 is not actually generated by the algorithm, and shows the execution result of the logic in an easy-to-understand manner for all the input luminances.

There is room for improvement in the above implementation example. Specifically, in the virtual tone map of FIG. 2, there are three places where the output suddenly increases with respect to the input, specifically, there are three places where the inclination is 45 ° or more. This place is a place where the brightness becomes prominent and large like noise.

FIG. 5 is a diagram showing an implementation example in which the above is improved, FIG. 6 is a virtual tone map corresponding to FIG. 5, and the dotted line in FIG. 6 is a virtual tone map when the logic of FIG. 4 is executed.

In order to eliminate such a portion where the brightness is prominent, the pixel value (luminance) of the reference pixels P1 to P8 is compared with the pixel value of the attention pixel P0 in order, and it is determined whether or not it is equal to or less than the pixel value of the attention pixel P0.

In parallel with this, the histogram value of the reference pixel is incremented by +1 and compared with the tilt limit value (45 °) to determine whether or not it is equal to or less than the tilt limit value.

Then, the number of reference pixels for which both of the above two judgments are satisfied is counted as a true count value for both, and the brightness of the pixel of interest is output based on the true count values for both.
By the above processing, as shown by the solid line in FIG. 6, an easy-to-see image without a portion where the brightness changes prominently can be obtained.

The number of reference pixels that are less than or equal to the pixel value of interest and are less than or equal to this tilt limit value compared to the preset tilt limit value by adding +1 to the histogram value of each reference pixel is used as the true count value for both. While the tilt limit is taken into consideration by counting, the entire image may become dark in this state. The configuration for correcting this is shown in the virtual tone maps of FIGS. 7 and 8.

In the implementation example shown in FIG. 7, the number of reference pixels that satisfy both the pixel value of the pixel of interest P0 or less and the tilt limit value or less is set as the true count value, and is further larger than the tilt limit value. Count the number of reference pixels (non-count value) of the histogram, set the brightness offset value as an external parameter with a value of 0 to n, and set (non-count value x offset value / n) to the true count value of both. In addition, it outputs.
Here, the offset value is determined by what percentage of the brightness (a + b) of the end point value reduced by the inclination limit is raised.
Output = true count value on both sides + (non-count value x offset value / n)
(N: number of reference pixels, eg 128 or 256)
(Offset value is 0 to n)

As a result, as shown in FIG. 8, the depopulated tone map characteristics of the image do not change, but the entire image becomes brighter by the offset.

FIG. 9 shows an implementation example in which the above-mentioned offset amount is automatically calculated for each region. In this implementation example, the read reference pixel value and the attention pixel value are compared, and the histogram value of the reference pixel is incremented by +1. If the histogram value of the reference pixel is larger than the tilt limit value, it is counted as a non-count value.

In parallel with the above, simply add the reference pixel values that were less than or equal to the preset tilt limit value, divide by the number of reference pixels to calculate the average value, and divide this by the maximum pixel brightness (256 for 8 bits). The value is proportionally divided from 0 to n to obtain the offset value. That is, by using the average brightness of the reference pixel as it is as the offset value, the brightness of the pixel of interest P0 can be matched with the brightness of the surroundings.

Then, as shown below, (non-count value x offset value / n) is added to both true count values and output.
Output = true count value on both sides + (non-count value x offset value / n)

FIG. 10 is a virtual tone map obtained by the above processing. In this implementation example, a tone map matching the brightness of the surroundings can be automatically generated by using the average brightness of the reference pixels as an offset value as it is.

By the above-mentioned image processing, the portion where the brightness is extremely outstanding with respect to the surroundings is eliminated and the entire image is not darkened, but the contrast of the entire image may be insufficient. An implementation example for solving this problem is shown in FIG.

In this implementation example, the number of reference pixels that satisfy both the pixel value of the pixel of interest P0 or less and the tilt limit value or less is set as the true count value, and the reference of the histogram that is larger than the tilt limit value is used. Count the number of pixels (non-counting value).

Then, by setting the contrast intensity value as an external parameter having a value of 0 to n and multiplying the true count values of both of them by {n / (n-non-count value × intensity value / n)}, attention is paid. The brightness of pixel P0 is output.
Output = both true count value x n / (n-non-count value x intensity value / n)
(N: number of reference pixels, eg 128 or 256)
(Intensity value is 0 to n)

FIG. 12 is a virtual tone map of the image obtained by the above processing. In the above-mentioned implementation example, the end point value (maximum brightness) reduced by the tilt limit is uniformly raised by the offset, but in this implementation example, it is multiplied. The tilt is also changing because the matching process is performed.

FIG. 13 shows an implementation example in which the offset function and the contrast function are simultaneously exhibited. In this implementation example, the number of reference pixels that satisfy both the pixel value of the pixel of interest P0 or less and the tilt limit value or less is set as the true count value, and the tilt limit set in advance in parallel with the above. The reference pixel value that was less than or equal to the value is simply added, divided by the number of reference pixels to calculate the average value, and this is used as the offset value.

Then, the true count values of both of the above are multiplied by {n / (n-non-count value × intensity value / n)}, and further {non-count value × (n-intensity value) / n × offset value / n} is obtained. In addition, output. That is, output = {n / (n-non-count value × intensity value / n)} + {non-count value × (n-intensity value) / n × offset value / n}.
(N: number of reference pixels, eg 128 or 256)
(Offset value and intensity value are 0 to n)

By each of the above processes, the part where the brightness is extremely different from the surroundings (noise, etc.) is eliminated, and the contrast of the entire image can be corrected. However, such processing tends to be biased toward a specific brightness in an area occupied by pixels having the same value. For example, in a dark image, when the pixel value is biased to 0, there is a problem that the image becomes whitish.

FIG. 15 is a diagram obtained by further improving the portion shown in FIG. 5. In this implementation example, two comparators for comparing the reference pixel value and the attention pixel value are prepared, and the brightness is smaller than the attention pixel value and the tilt is limited. The number of reference pixels that are less than or equal to the value and the number of reference pixels that are equal to or less than the value of the pixel of interest and are less than or equal to the tilt limit value are individually counted.

The count number of the brightness equal to the pixel value of interest is proportionally added to the count number of the brightness smaller than the pixel value of interest obtained in the previous section according to the brightness of interest.
By doing so, as shown in FIG. 16, a tone map in which the end point (maximum brightness) is fixed and the start point moves to the origin (0,0) can be obtained.

As described above, once the blackout correction image (FC output image) and the sharpening output image are obtained, these images are combined using the principle of flat frame. In the present invention, in compositing, instead of simply blending the blackout correction image and the sharpening output image, the blackout correction image is distributed more in the dark part and the sharpening output image is more distributed in the bright part. To.

When the above is implemented by the CPU, the flat frame is actually generated and then the calculation is performed between the frames.
On the other hand, in the case of real-time processing with a circuit such as FPGA, instead of actually creating a flat frame for one frame buffer, a line buffer is used and a ring buffer for the blur diameter (a structure that repeatedly uses as much as necessary). ) Is used for real-time processing. That is, instead of creating a flat frame for one screen, a horizontally long thin flat frame is generated for each scan line in synchronization with the screen scan.

It is the same as the flat frame used for blackout correction in that real-time processing is performed with a delay of the diameter of the flat frame using the line buffer, but in the case of blackout correction, the brightness information is blurred with Gaussian. Later, the level was corrected to make it a flat frame, but the level correction at the time of screen composition may be different from the correction value of the blackout correction, so the line buffer is kept in a state where the brightness is only blurred with Gaussian. In addition, different level corrections are performed for blackout correction and screen composition. Since the level correction is completed only by addition and multiplication, it can be calculated in real time immediately before output.

FIG. 17A is an original image, in which blackouts occur in dark areas and there are unclear areas. (B) is an image after the blackout processing, and although the blackout has been eliminated, the contrast of the bright part is insufficient and the electric wire is overexposed. (C) is an image after the sharpening process, in which the contrast in the dark portion is excessively emphasized and the particles in the backlit portion are disturbed. (D) is an image after image composition (the present invention), in which the problems of blackout processing and sharpening processing are both solved, the electric wire can be visually recognized, and the particles in the backlit portion are arranged.

Claims (2)

It is an image process that combines an image that has undergone sharpening processing and an image that has undergone blackout processing , and the blackout processing is a Gaussian Blur that determines the brightness of each pixel in the Y plane (luminance plane) memory of the input image. The Blur value with blurred brightness is obtained by processing, and this Blur value is normalized to obtain distribution information that takes a value from 0 to 1.0, and a threshold value is set between the normalized value 0 to 1.0, and this threshold value is set. The brightness magnification (n) of the darkest pixel is determined for all dark pixels with a value smaller than the threshold value, and the inverse number (1 / n) is set as the minimum value of the distribution information. An image processing method characterized in that the distribution information is corrected as described above, and the brightness of the input image is divided using the corrected brightness distribution information (1 / n to 1.0) of the dark part as the denominator. In the image treatment method according to claim 1, in the sharpening process, when processing the captured image data in pixel units, n (n is an integer) reference around the pixel of interest which is the scanning position of the image scanning. Set the pixels, compare the pixel value (brightness) of the noteworthy pixel with the pixel value (brightness) of each reference pixel in sequence, count the number of reference pixels that were less than or equal to the noteworthy pixel value, and parallel this. The histogram value of each reference pixel is incremented by +1 and compared with the preset tilt limit value, and both the condition that the reference pixel value is less than or equal to the attention pixel value and the condition that the pixel histogram value is less than or equal to the tilt limit value are set. An image processing method characterized in that a true count value for both sides is obtained by counting the number of reference pixels to be satisfied, and the true count values for both sides are proportionally divided and output.

Images