CN114998135A - Image enhancement method and device, field programmable logic gate array and equipment - Google Patents

Abstract

The disclosure relates to an image enhancement method, an image enhancement device, a field programmable gate array and a field programmable gate array device. Obtaining a luminance domain representation of an image, the luminance domain representation being used to characterize a distribution of pixel values of the image in a luminance domain; calculating a stretch coefficient based on the luminance domain representation; and updating pixel values of the image in the luminance domain based on the stretch coefficient. Therefore, the overall brightness of the image can be adjusted without influencing the color of the image (namely, without distortion), so that the contrast is enhanced, the calculation amount can be reduced to a certain extent, and the method is friendly to hardware.

Description

Image enhancement method, apparatus, field programmable logic gate array, and apparatus

technical field

The present disclosure relates to the technical field of image enhancement, and in particular, to an image enhancement method, device, field programmable logic gate array and device suitable for enhancing images with low contrast and improper exposure.

Background technique

Image enhancement technology is suitable for scenes that enhance image quality and the preprocessing part of various image processing and analysis modules. This technology improves the quality of the image by emphasizing or changing the local or overall features of the image and adapts to special needs. Therefore, image enhancement technology has developed rapidly in recent years.

Histogram equalization is a more commonly used technique in image enhancement technology. The histogram of an image is a graphical representation of the intensity and the probability of occurrence of intensity values in a given image. Image histogram equalization is a method of equalizing the probability distribution of intensity values in an image. Due to the limitations of the camera device or the characteristics of the scene, the captured image pixels will be concentrated in some parts of the pixel value range, resulting in a dim image without a sense of hierarchy, and the scene information cannot be truly reproduced, that is, the image contrast is low. The histogram of the image describes the distribution of the pixel values of the image on the value domain, and shows the distribution characteristics of the image pixel values. The above problem of low image contrast can be represented by constructing a histogram of image pixel values. At the same time, adjusting the histogram of the image can change the relative characteristics such as the contrast of the image.

The current color image enhancement methods using histogram equalization mainly include the following:

1. Generate a mapping relationship according to the cumulative histogram of image color values to enhance image contrast. In order to protect the overall brightness or other purposes, this method needs to divide the sub-regions on the cumulative histogram and calculate the cumulative density function separately. There are many hyperparameters in the process of dividing the sub-regions, which is difficult to manually select and difficult to adapt to different scenarios.

2. Use a locally adaptive scheme to generate digital images that exhibit improved exposure and tonal uniformity by employing histogram equalization on a local basis, even though the stretching and compression of pixel brightness levels adapts to the local distribution of pixels within a pixel. The adjustment effect of this method for local areas is good, but discontinuous edges are easily generated between different areas, and the overall effect is not good due to the emphasis on local effects.

3. Perform histogram equalization on the RGB channels in the RGB domain color space, and the tone will change. Therefore, a tone preservation module needs to be added. For example, wavelet transform is used to preserve the wavelet detail coefficients of the image and reconstruct the result after histogram equalization. . This kind of method has high computational complexity and introduces the problem of tone distortion.

It can be seen that the existing implementation scheme of histogram equalization has the problem of image distortion or excessive calculation, so an image enhancement method with good effect and hardware-friendly that can realize real-time processing is required.

SUMMARY OF THE INVENTION

A technical problem to be solved by the present disclosure is to provide an image enhancement method that has better effects, is adaptable to different scenarios, and is hardware-friendly and can realize real-time processing.

According to a first aspect of the present disclosure, an image enhancement method is provided, comprising: acquiring a luminance domain representation of an image, where the luminance domain representation is used to characterize the pixel value distribution of the image in the luminance domain; calculating a stretching coefficient based on the luminance domain representation ; and updating the pixel values of the image in the luminance domain based on the stretch factor.

Optionally, the step of calculating the stretch coefficient based on the luminance domain representation includes: calculating the luminance domain pixel mean value based on the luminance domain representation; Pixel mean is negatively correlated and positively correlated with expected luminance.

Optionally, the tensile coefficient k is calculated based on the following formula,

或者

or

Among them, b is the average value of pixels in the luminance domain, y is the expected value of luminance, and p(x) represents the proportion of pixels with pixel value x in the image.

Optionally, the method further includes: limiting the value of the stretch coefficient to a first value interval.

Optionally, the method further includes: calculating the first numerical interval based on the value range limit parameter and the average value of pixels in the luminance domain.

Optionally, the first numerical interval is [(1-μ)*b, μ+(1-μ)*b], where μ is a value range limitation parameter, and b is a pixel mean value in the luminance domain.

Optionally, the step of updating the pixel value of the image in the luminance domain based on the stretching coefficient includes: adjusting the pixel value based on the pixel value, the average pixel value in the luminance domain and the stretching coefficient.

Optionally, adjust the pixel value _xi based on the following formula,

Among them, y _i is the adjusted pixel value, k is the stretching coefficient, b is the average pixel value of the luminance domain, x _min is the minimum pixel value of the image in the luminance domain, and x _max is the maximum pixel value of the image in the luminance domain.

Optionally, the method further includes: quantizing the operations involved in the execution of the method into fixed-point operations, and entrusting the method to a field programmable logic gate array for execution.

Optionally, the field programmable logic gate array adopts a multi-port BRAM memory, and the field programmable logic gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, adjacent in the image. The pixel columns are assigned to different sub-depths, and the field programmable logic gate array takes the pixel value of the read image in the luminance domain as the address, reads the sub-depth assigned to the pixel column where the pixel value is located, and reads the Add 1 to the output value and write back to the original address. After reading all the pixels, the field programmable logic gate array adds the values stored in the storage addresses corresponding to the same pixel value in different sub-depths, and adds the result. As the number of pixel values, the luminance domain representation is obtained.

According to a second aspect of the present disclosure, there is also provided an image enhancement apparatus, comprising: an acquisition module configured to acquire a luminance domain representation of an image, where the luminance domain representation is used to characterize pixel values of the image in the luminance domain distribution; a calculation module for calculating a stretch coefficient based on the pixel value distribution; and an update module for updating pixel values of the image in the luminance domain based on the stretch coefficient.

According to a third aspect of the present disclosure, there is also provided a field programmable logic gate array for executing the method described in the first aspect, wherein the operation involved in the method is a fixed-point number operation obtained by quantization.

Optionally, the field programmable logic gate array adopts a multi-port BRAM memory, and the field programmable logic gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, adjacent in the image. The pixel columns are assigned to different sub-depths, and the field programmable logic gate array takes the pixel value of the read image in the luminance domain as the address, reads the sub-depth assigned to the pixel column where the pixel value is located, and reads the Add 1 to the output value and write back to the original address. After reading all the pixels, the field programmable logic gate array adds the values stored in the storage addresses corresponding to the same pixel value in different sub-depths, and adds the result. As the number of pixel values, the luminance domain representation is obtained.

According to a fourth aspect of the present disclosure, there is also provided an imaging device, comprising: an imaging device and a field programmable logic gate array, the field programmable logic gate array is used to perform the method described in the first aspect above, In order to enhance the image captured by the imaging device, the operation involved in the method is a fixed-point number operation obtained by quantization.

According to a fifth aspect of the present disclosure, there is provided a computing device, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, causes the processor to execute the above-mentioned first the method described in the aspect.

According to a sixth aspect of the present disclosure, there is provided a computer program product comprising executable code, when the executable code is executed by a processor of an electronic device, the processor is caused to execute as described in the first aspect above Methods.

According to a seventh aspect of the present disclosure, there is provided a non-transitory machine-readable storage medium on which executable codes are stored, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the above-mentioned first step. The method described in one aspect.

Therefore, the present disclosure adjusts the pixel value of the image in the luminance domain by acquiring the luminance domain representation of the image and using the stretch coefficient calculated based on the luminance domain representation, so that the image color is not affected (ie, will not be distorted). At the same time, contrast enhancement can be achieved, and the calculation amount can be reduced to a certain extent, which is friendly to hardware.

Description of drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the more detailed description of the exemplary embodiments of the present disclosure taken in conjunction with the accompanying drawings, wherein the same reference numerals generally refer to the exemplary embodiments of the present disclosure. same parts.

FIG. 1 shows a schematic flowchart of an image enhancement method according to an embodiment of the present disclosure.

Figure 2 shows the visualization of the pixel value adjustment formula.

FIG. 3 shows a schematic diagram of the overall processing flow of the hardware.

FIG. 4 shows a schematic diagram of data flow transmission and BRAM instantiation.

Figure 5A shows an overall darker image.

FIG. 5B is an image after contrast enhancement of the image shown in FIG. 5A using the present disclosure.

FIG. 6 shows a schematic structural diagram of an image enhancement apparatus according to an embodiment of the present disclosure.

FIG. 7 shows a schematic structural diagram of an imaging device according to an embodiment of the present disclosure.

FIG. 8 shows a schematic structural diagram of a computing device according to an embodiment of the present disclosure.

Detailed ways

Preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

In view of the problems of low contrast and inappropriate exposure (image is dark or too bright) of the image, the present disclosure proposes that the component of the image in the luminance domain (that is, the luminance component or the luminance domain component) can be obtained, and the luminance level of the image can be considered Adjust (stretch) the brightness component in the case of not affecting the color of the image (that is, without distortion), adjust the overall brightness of the image, achieve contrast enhancement, and reduce the amount of calculation to a certain extent. friendly.

FIG. 1 shows a schematic flowchart of an image enhancement method according to an embodiment of the present disclosure.

Referring to FIG. 1, in step S110, a luminance domain representation of the image is obtained.

The luminance domain representation is used to characterize the distribution of pixel values in the luminance domain (ie, luminance channel) of an image. The pixel value distribution, that is, the distribution of pixel values, may be the number of pixels per pixel value, or the pixel ratio of each pixel value.

The pixel value of the image in the luminance domain reflects the luminance value of the image pixel.

The luminance domain representation may be a component of the image in the luminance domain, that is, the luminance component or the luminance domain component of the image.

The luminance domain representation may also be a statistical result obtained by performing statistics on luminance components of an image.

For example, the luminance domain representation may be a histogram derived from statistics of luminance components of an image. The histogram marks the number of each pixel value of the image in the luminance domain, or the ratio of the number of each pixel value to the total number of pixels.

An image refers to an image to be enhanced, specifically an image with low contrast and inappropriate exposure, such as a relatively dim or too bright image.

The images can be RGB images or images in other color spaces such as HSV, YUV, YCbCr. There is no luminance component in the RGB color space, so if the image is an image in a color space with no luminance component such as RGB, the image needs to be converted to another color space with a luminance component to obtain the luminance domain representation of the image.

For example, HSV (hue, saturation, lightness) is a color space created by A.R. Smith in 1978 according to the intuitive characteristics of color, also known as the Hexcone Model, where the V channel is the luminance channel, so Can convert RGB images to HSV color space.

For another example, in the YUV color space, "Y" represents the brightness (Luminance or Luma), that is, the grayscale value; while "U" and "V" represent the chromaticity (Chrominance or Chroma), where the Y channel That is, the brightness channel, "brightness" is obtained by linear transformation of the RGB input signal, so the RGB image can also be converted to the YUV color space.

At step S120, the stretch factor is calculated based on the luminance domain representation.

The stretch factor is used to enhance the contrast of the image to solve the problem of low contrast and inappropriate exposure in the image. The main reason for the problem of low image contrast is that the image pixels are concentrated in some parts of the pixel value range, resulting in a dim or too bright image, without a sense of hierarchy, and unable to truly reproduce the scene information, that is, the image contrast is low.

The "stretching" in the stretching coefficient mentioned in the present disclosure refers to shifting the pixel values that are originally relatively concentrated in the luminance domain of the image to the pixel value domain (such as 0 to 255, or normalized 0 to 1). A wider value range is stretched to improve the contrast of the image by balancing the distribution of pixel values.

In other words, the stretch factor is used to adjust the pixel value distribution of the image in the luminance domain. The "stretching" in the stretching coefficient is a generalized stretching, which refers to stretching the pixel value distribution represented by the luminance domain representation of the image to enhance the contrast, rather than restricting that each pixel value must be stretched. stretch. That is to say, when the stretching coefficient is used to adjust a specific pixel value, it may include not only increasing the pixel value (stretching), but also decreasing the pixel value (scaling).

When enhancing the contrast of an image based on the stretching coefficient, it is necessary to consider the original brightness of the image and protect the original brightness of the image to a certain extent to avoid brightness distortion. To this end, the present disclosure proposes that the stretch coefficient can be calculated based on the luminance domain representation, so that the value of the stretch coefficient is highly correlated with the image luminance level, so that the luminance can be protected.

As an example, the luminance domain pixel mean value may be calculated based on the luminance domain representation, and the stretch factor may be calculated based on the luminance domain pixel mean value and the luminance expected value. Among them, the size of the stretch coefficient is negatively related to the mean value of the pixels in the luminance domain, and positively related to the expected value of luminance. The luminance domain pixel mean is used to characterize the average of all pixel values in the luminance domain of the image. The expected brightness value is used to represent the expected mean brightness value after adjusting the pixel values of the image in the brightness domain. The stretch factor can be used to make the luminance domain pixel average of the updated image closer to the desired luminance value than before the update.

The expected value of brightness is generally 0.5 (the result after numerical normalization), which is half of the image pixel value range. From a visual point of view, if the average pixel value of the brightness domain is near this value, the overall visual effect of the image is better. The expected value of brightness generally does not change, that is, it does not need to change as the image changes. However, the expected brightness value can still be designed as an adjustable parameter. In practical applications, if you want the image to be darker or brighter, it can be achieved by adjusting the expected brightness value.

Optionally, the stretch factor can be calculated based on the difference between the luminance domain pixel mean and the desired luminance value. The stretch factor can be designed such that when the average value of pixels in the luminance domain is lower than the expected value of luminance, the value of the stretch coefficient is greater than that when the average value of pixels in the luminance domain is higher than the expected value of luminance, and the value of the stretch coefficient is the same as that in the luminance domain. Pixel mean is negatively correlated.

For example, when the average value of pixels in the luminance domain is lower than the expected value of luminance, the value of the stretching coefficient is larger, and the lower the average value of pixels in the luminance domain is, the larger the value of the stretching coefficient is; The value of the stretch factor is smaller, and the higher the average pixel value in the luminance domain, the smaller the stretch factor.

For example, the stretch factor k can be calculated based on the following formula

b is the mean value of pixels in the luminance domain, y is the expected value of luminance, and p(x) represents the proportion of pixels with pixel value x in the image. The luminance domain pixel value in the formula is the normalized result, the highest pixel value is 1, and the lowest pixel value is 0.

表示像素值大于亮度域像素均值的所有像素的像素平均值；

表示大于亮度域像素均值的像素占比与亮度域像素均值的乘积；

表示像素值小于亮度域像素均值的所有像素的像素平均值；

表示亮度域像素均值。(1-b)y represents the product of the difference between the highest pixel value and the mean value of the pixels in the luminance domain and the expected value of luminance;

Indicates the pixel average value of all pixels whose pixel value is greater than the pixel average value of the luminance domain;

Represents the product of the proportion of pixels greater than the mean value of pixels in the luminance domain and the mean value of pixels in the luminance domain;

Indicates the pixel average value of all pixels whose pixel value is less than the pixel average value of the luminance domain;

Represents the luminance domain pixel mean.

In order to facilitate understanding, the calculation formula (1) of the tensile coefficient k can be simplified as

-delta是(x-b)p(x)在b和1之间的积分，即高于亮度域像素均值的所有像素的像素值中高出均值的部分的平均值。in

-delta is the integral of (xb)p(x) between b and 1, that is, the average of the above-mean pixel values of all pixels above the luminance-domain pixel mean.

In the above formula, the pixel mean value b in the luminance domain and the expected luminance value y are both floating point numbers between 0 and 1, which are dimensionless units. Therefore, the final calculated stretching coefficient k is a dimensionless unit.

It can be seen from formula (2) that the size of the stretching coefficient k is calculated based on the ratio of the luminance domain pixel mean value b to the luminance expected value y, and the luminance domain pixel mean value b. When the pixel mean value b in the luminance domain is lower than the expected brightness value y, the k value is larger, and the lower the pixel mean value b in the luminance domain is, the larger the k value is; on the contrary, when the pixel mean value b in the luminance domain is higher than the expected luminance value y, the k value is smaller, and The higher the pixel mean value b in the luminance domain, the smaller the k is; when the mean pixel value b in the luminance domain is equal to the expected luminance value y, the value of k is equal to the expected luminance value y.

The calculation principle of the stretching coefficient k can be summarized as follows: the stretching coefficient k is calculated according to the difference between the image brightness level and the desired brightness level (that is, the expected brightness value), and the stretching coefficient k is highly correlated with the image brightness level, which plays a role in protecting the brightness. . In contrast, the current mainstream contrast enhancement algorithms do not consider the brightness level of the original image.

Considering that the stretching coefficient is too large or too small, the image will be changed too obviously, and local overexposure or dim situation will occur, the present disclosure proposes that the value of the stretching coefficient can be An interval can be regarded as a reasonable range of values. The first value interval may be a preset range irrelevant to the image, or may be a range determined based on the luminance domain representation of the image.

As an example, the first value interval may be calculated based on the value range limit parameter and the luminance domain pixel mean value. The value range limit parameter is used to limit the value range of the stretching coefficient. In practical applications, the value of the value range limit parameter can be adjusted according to the effect of the algorithm. For example, if the image processed by the algorithm is too bright, you can Appropriately reduce the value of the value range limit parameter to suppress the stretching effect.

As mentioned earlier, the Range Limit parameter controls the range of values for the stretch factor. In actual testing, the value is 0.25. This value generally does not change. However, this value is still adjustable, and in some extreme cases, this value affects the actual stretching effect. If the scene is too dark or too bright, this value can be appropriately increased to enhance the stretching effect; otherwise, if the output image is too dark or bright, the value can be decreased to weaken the stretching effect.

The first value interval may be [(1-μ)*b, μ+(1-μ)*b], where μ is a limit parameter of the value range, and b is the average value of pixels in the luminance domain. Therefore, the limit range of k can be calculated according to the original input image mean value b and the value range limit parameter μ, and k can be updated

Wherein, when (1-μ)*b<k<μ+(1-μ)*b, the value of k is equal to the value calculated according to the calculation formula (1) or (2) of the elongation coefficient k.

In step S130, the pixel values of the image in the luminance domain are updated (ie adjusted) based on the stretch coefficient.

Adjustments may be made for at least some (eg, all) pixel values of the image in the luminance domain based on the stretch factor. Among them, for pixel values with different values, a non-linear adjustment method can be used to adjust, so as to uniformly stretch the pixel values of the image in the luminance domain to the pixel value domain. As an example, the pixel value may be adjusted based on the pixel value, the luminance domain pixel mean, and the stretch factor. For example, the pixel value x _i can be adjusted based on the following formula

Among them, y _i is the adjusted pixel value, k is the stretching coefficient, b is the average pixel value of the luminance domain, x _min is the minimum pixel value of the image in the luminance domain, and x _max is the maximum pixel value of the image in the luminance domain. The luminance domain pixel value in the formula is the normalized result, the highest pixel value is 1, and the lowest pixel value is 0.

The function of the pixel value adjustment formula (4) is to uniformly stretch the image pixel value to the range of [0,1]. When the pixel value is less than the mean value of the image pixel value, the formula compares the input pixel value with the mean b, and distributes it to [0,k); otherwise, it stretches to the range of [k,1]. Figure 2 shows a visualization of this formula. As shown in Figure 2, the abscissa is the pixel value before stretching, the ordinate is the pixel value after stretching, and the entire stretching process is nonlinear stretching, where k72_b11 indicates that the stretching coefficient k is 72, and the average pixel value in the luminance domain b is 11.

So far, the image enhancement method of the present disclosure has been exemplarily described with reference to FIG. 1 .

The present disclosure also provides a hardware adaptive quantization scheme based on the above method.

The present disclosure can quantify the operations involved in the execution of the method (such as floating-point number operations) into fixed-point number operations, and assign the method to hardware for execution. (Field Programmable Gate Array, FPGA) implementation.

FPGA belongs to a kind of semi-custom circuit in integrated circuit, which consists of a series of Configurable Logic Blocks (CLB, Configurable Logic Block) and programmable connections. It can be quickly compiled and programmed into a combination of transistor circuits through a hardware description language, so that the algorithm can be quickly deployed and accelerated on the hardware platform. It has the characteristics of extremely high computing energy consumption ratio, rapid iteration, customization, etc. It is especially suitable for current artificial intelligence and related applications in data centers.

FPGA can be used to implement the method shown above in conjunction with Figure 1, and the rich DSP, BRAM, CLB and other configurable resources inside the FPGA can be used to process data streams in parallel, and real-time enhancement of 4K, FPS60 and other types of video images can be achieved.

When using FPGA to implement the above method, the parameters involved in the method can be limited to non-negative integers within the pixel value range (such as 0 to 255), and the integration process can be changed to equivalent multiply-accumulate, so that the Floating-point arithmetic is quantized into fixed-point arithmetic. The range of pixel values refers to the range of pixel values of the image to be enhanced.

Taking the method to use the above-mentioned calculation formula (1) of the stretching coefficient k, the value restriction formula (3) of the stretching coefficient k, and the pixel value adjustment formula (4) to enhance the image as an example, the method based on this method is used to enhance the image. The scheme after hardware adaptation and quantization is as follows.

1. Give the expected value y of the image pixel value after histogram equalization and the limiting parameter μ of the value range of the expected value.

2. Count the number of pixel points corresponding to each pixel value, and obtain the pixel value distribution histogram of the image. The value range of the abscissa of the histogram is limited to 0 to 255. If the pixel width of the image is higher than 8bit, the histogram can be increased. The value range of the abscissa, for example, if the input pixel value range is 0 to 4095, the value range of the abscissa of the histogram is also 0 to 4095;

3. Calculate the mean value b of the pixel value according to the pixel value of the input image, and take the upper 8 bits of the bit bit b_;

4. Calculate the cumulative distribution function using the cumulative multiplication method according to the following formula;

5. Substitute the result of the above cumulative distribution function into the following formula, calculate the histogram stretching coefficient k, and fix the value after calculating the division;

6. Calculate the limit range of k according to the mean value b of the original input image and the limit parameter μ of the value range and update k;

7. Calculate a new pixel value based on the input pixel value x and the above-mentioned stretching coefficient k and the image mean value b;

FIG. 3 shows a schematic diagram of the overall processing flow of the hardware.

As shown in FIG. 3 , the hardware processing flow mainly includes three steps: histogram statistics, k-value calculation, and adjustment of low-point new pixel values for pixel values. Taking FPGA as an example, the specific implementation of the above three steps is as follows.

Step P1, perform histogram statistics on the image data frame by frame

Histogram statistics is to count the number of occurrences of each pixel value in a frame of image in the luminance domain.

FPGAs can use multi-port (eg dual-port) BRAM memory.

The FPGA can divide the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 storage addresses, and adjacent pixel columns in the image are assigned to different sub-depths to avoid when the same data appears in a row, it cannot be timely. Update BRAM.

The FPGA takes the pixel value of the read image in the luminance domain as the address, reads the sub-depth assigned to the pixel column where the pixel value is located, adds 1 to the read value and writes back to the original address.

After the FPGA reads all the pixels, it adds the values stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of the pixel value, so that the histogram statistics result can be obtained.

The specific implementation steps of FPGA statistical histogram are as follows:

① Take the high 8bit data and get the pixel range of 0-255;

② Instantiate a BRAM with a depth of 512, address 0-255 is used to store the number of pixels in even columns, and addresses 256-511 are used to store the number of pixels in odd columns;

③ When taking the pixel value of the even column, the obtained pixel value is used as the address to read the BRAM address 0-255; when the pixel value of the odd column is taken, the obtained pixel value is shifted to the left by 1 bit, and the BRAM address 256-511 is read as the address;

④Add 1 to the read data, and rewrite the corresponding address;

⑤ After storing a whole frame of data, read the data of 0-255 and 256-511 in the BRAM at the same time, add the data of address 0 to the data of address 256, and add them in turn to get the number of each pixel value of 0-255. number;

⑥ Data streaming and BRAM instantiation are shown in Figure 4.

The FPGA adopts the method of interleaved storage to the true dual-port RAM with a depth of 512 and 256 addresses before and after the parity column, which avoids the use of frequency multiplication and cross-clock domain operation, and also reduces the use of BRAM blocks and saves resources.

As shown in Figure 4, the number of columns of pixel values in the image data can start from the 0th column, and the read pixel value of the 0th column is 0. At this time, the read pixel value 0 can be used as the address in BRAM address 0 Read within -255, that is, read address 0, then add 1 to the value in address 0 (the initial value is 0) and rewrite address 0. The pixel value of the first column is read as 18, which belongs to an odd column. The read pixel value 18 can be shifted to the left by one bit, and the value of 273, which is the sum of 17 and 256, is used as the address, which is in the BRAM address 256-511. A read is performed, that is, address 273 is read, and the value in address 273 (initial value is 0) is incremented by 1 and address 273 is rewritten. By analogy, the traversal of all pixel values in the image data is completed.

Step P2, according to the statistical results of the histogram, calculate the stretching coefficient k

The specific steps for the FPGA to calculate the stretching coefficient k are as follows:

①At the same time of histogram statistics, the input pixels are accumulated to obtain the actual pixel sum, and divided by the number of pixels in each frame to obtain the pixel mean value b, which is also the brightness domain pixel mean value mentioned above.

② With b as the boundary, the data of BRAM address 0-b is multiplied by 0-b to get sum0; the data of BRAM address b-255 is accumulated to get sum; the data of BRAM address b-255 is multiplied by b-255, Add up to get sum1.

③ According to formula (5), the divisor and dividend are obtained separately, and sent to the divider to obtain the k value;

④According to formula (6), using the mean value b obtained in ① in step P2, calculate the limit range of k and update k;

Step P3: Calculate the new pixel value after stretching of each pixel of the current frame according to the formula

The specific steps for the FPGA to calculate the new pixel value are as follows:

①At the same time of statistical histogram, the maximum pixel xmax and the minimum pixel xmin are obtained by comparing the size of the input pixel value, and registered;

②According to formula (7), calculate the new pixel value.

FIG. 5A is an overall rather dim image.

For the image shown in FIG. 5A , according to the above scheme, the stretching coefficient k is calculated to be 84, the image pixel mean value b is 27, the expected value y of the image pixel value is set to 127, and the value range limit parameter μ is 64.

The output result obtained by enhancing the image shown in FIG. 5A according to the above scheme is shown in FIG. 5B .

It can be seen that, for an image with a lower contrast and an overall darker image, the present disclosure can achieve contrast enhancement without causing image distortion. In addition, for images with low contrast and too bright overall, the present disclosure can also achieve contrast enhancement without causing image distortion.

To sum up, the present disclosure provides an image enhancement algorithm based on histogram equalization that works in the luminance domain, which can solve the problems of low contrast and inappropriate exposure of the image (eg, the image is relatively dim or too bright), and can Protect image brightness. In addition, the present disclosure also improves and optimizes hardware features, so that the algorithm can realize real-time processing on the hardware platform, especially in the face of high resolution and high frame rate application scenarios, it can still ensure a good histogram equalization effect. .

The image enhancement method of the present disclosure can also be implemented as an image enhancement device. FIG. 6 shows a schematic structural diagram of an image enhancement apparatus according to an embodiment of the present disclosure. The functional units of the image enhancement apparatus may be implemented by hardware, software, or a combination of hardware and software implementing the principles of the present disclosure. Those skilled in the art can understand that the functional units described in FIG. 6 can be combined or divided into sub-units, so as to realize the principle of the above invention. Accordingly, the description herein may support any possible combination, or division, or further definition of the functional units described herein.

The following briefly describes the functional units that the image enhancement apparatus can have and the operations that each functional unit can perform. For the details involved, reference may be made to the above related descriptions, which will not be repeated here.

Referring to FIG. 6 , the image enhancement apparatus 600 includes an acquisition module 610 , a calculation module 620 and an update module 630 .

The obtaining module 610 is configured to obtain the luminance domain representation of the image, and the luminance domain representation is used to characterize the pixel value distribution of the image in the luminance domain.

The calculation module 620 is used to calculate the stretching coefficient based on the distribution of pixel values.

As an example, the calculation module 620 may calculate the luminance domain pixel mean value based on the luminance domain representation, and calculate the stretching coefficient based on the luminance domain pixel mean value and the luminance expectation value, wherein the magnitude of the stretching coefficient is negatively correlated with the luminance domain pixel mean value and is related to the luminance expected value positive correlation. For example, the calculation module may calculate the stretch factor using the above-mentioned formula (1).

The image enhancement apparatus 600 may further include a limiting module, configured to limit the value of the stretching coefficient to the first value interval.

The image enhancement apparatus 600 may further include a second calculation module, configured to calculate the first numerical interval based on the value range limitation parameter and the mean value of the pixels in the luminance domain. For the first value interval, please refer to the above related description.

The updating module 630 is configured to update the pixel value of the image in the luminance domain based on the stretching coefficient.

The update module 630 may adjust the pixel value based on the pixel value, the luminance domain pixel mean, and the stretch coefficient. For example, the update module 630 may adjust the pixel values using the above-mentioned equation (3).

The present disclosure also provides a field programmable logic gate array, which can quantize the operations involved in the execution of the above-mentioned methods into fixed-point operations, and execute the quantized method.

The present disclosure also provides an imaging apparatus. FIG. 7 shows a schematic structural diagram of an imaging device according to an embodiment of the present disclosure. As shown in FIG. 7 , the imaging apparatus 700 may include an imaging device 710 and a field programmable logic gate array 720 . The imaging device 710 is used for capturing imaging. The field programmable logic gate array is used to execute the method shown above in conjunction with FIG. 1 to enhance the image captured by the imaging device, wherein the operation involved in the method is a fixed-point number operation obtained by quantization. For the specific implementation process of the field programmable logic gate array, reference may be made to the above related description.

FIG. 8 shows a schematic structural diagram of a computing device that can be used to implement the above image enhancement method according to an embodiment of the present disclosure.

Referring to FIG. 8 , computing device 800 includes memory 810 and processor 820 .

The processor 820 may be a multi-core processor, or may include multiple processors. In some embodiments, processor 820 may comprise a general-purpose main processor and one or more special-purpose co-processors, such as a graphics processing unit (GPU), a digital signal processor (DSP), and the like. In some embodiments, the processor 820 may be implemented using customized circuits, such as Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs).

Memory 810 may include various types of storage units, such as system memory, read only memory (ROM), and persistent storage. The ROM may store static data or instructions required by the processor 820 or other modules of the computer. Persistent storage devices may be readable and writable storage devices. Permanent storage may be a non-volatile storage device that does not lose stored instructions and data even if the computer is powered off. In some embodiments, persistent storage devices employ mass storage devices (eg, magnetic or optical disks, flash memory) as persistent storage devices. In other embodiments, persistent storage may be a removable storage device (eg, a floppy disk, an optical drive). System memory can be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data that the processor needs at runtime. In addition, memory 810 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read only memory), and magnetic and/or optical disks may also be employed. In some embodiments, memory 810 may include a removable storage device that is readable and/or writable, such as a compact disc (CD), a digital versatile disc (eg, DVD-ROM, dual-layer DVD-ROM), Read-only Blu-ray Discs, Ultra-Density Discs, Flash Cards (eg SD Cards, Min SD Cards, Micro-SD Cards, etc.), Magnetic Floppy Disks, etc. Computer readable storage media do not contain carrier waves and transient electronic signals transmitted over wireless or wire.

Executable codes are stored on the memory 810, and when the executable codes are processed by the processor 820, the processor 820 can be caused to perform the above-mentioned image enhancement method.

The image enhancement method, apparatus, field programmable logic gate array, and apparatus according to the present disclosure have been described in detail above with reference to the accompanying drawings.

Furthermore, the method according to the present disclosure can also be implemented as a computer program or computer program product comprising computer program code instructions for performing the above-mentioned steps defined in the above-mentioned method of the present disclosure.

Alternatively, the present disclosure can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) having executable code (or computer program, or computer instruction code stored thereon) ), when the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or a computing device, a server, etc.), the processor is caused to perform each step of the above-mentioned method according to the present disclosure .

Those skilled in the art will also appreciate that the various exemplary logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems and methods in accordance with various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

Various embodiments of the present disclosure have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or improvement over the technology in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims (17)

1. An image enhancement method, comprising: obtaining a luminance domain representation of the image, where the luminance domain representation is used to characterize the pixel value distribution of the image in the luminance domain; computing a stretch factor based on the luminance domain representation; and The pixel values of the image in the luminance domain are updated based on the stretch factor. 2. The method of claim 1, wherein the step of calculating a stretch factor based on the luminance domain representation comprises: computing a luminance domain pixel mean based on the luminance domain representation; The stretch coefficient is calculated based on the luminance domain pixel mean value and the luminance expected value, wherein the magnitude of the stretch coefficient is negatively correlated with the luminance domain pixel mean value and positively correlated with the luminance expected value. 3. The method according to claim 2, wherein the elongation coefficient k is calculated based on the following formula,

or

Wherein, b is the mean value of the pixels in the luminance domain, y is the expected value of luminance, and p(x) represents the proportion of pixels with a pixel value of x in the image. 4. The method of claim 1, further comprising: The value of the stretch factor is limited to a first value interval. 5. The method of claim 4, further comprising: The first numerical interval is calculated based on the value range limit parameter and the mean value of pixels in the luminance domain. 6. The method of claim 5, wherein, The first numerical interval is [(1-μ)*b, μ+(1-μ)*b], where μ is a limit parameter of the value range, and b is the average value of pixels in the luminance domain. 7. The method of claim 1, wherein updating pixel values of the image in the luminance domain based on the stretch factor comprises: The pixel value is adjusted based on the pixel value, the luminance domain pixel mean, and the stretch factor. 8. The method of claim 7, wherein the pixel values x _i are adjusted based on the following formula,

Among them, y _i is the adjusted pixel value, k is the stretching coefficient, b is the pixel mean value in the luminance domain, x _min is the minimum pixel value of the image in the luminance domain, and x _max is the image in the luminance domain. Maximum pixel value. 9. The method of any one of claims 1 to 8, further comprising: The operations involved in the execution of the method are quantized into fixed-point operations, and the method is executed by a field programmable logic gate array. 10. The method of claim 9, wherein, The field programmable logic gate array adopts multi-port BRAM memory, The field programmable logic gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 memory addresses, and adjacent pixel columns in the image are assigned to different sub-depths , The field programmable logic gate array uses the read pixel value of the image in the luminance domain as an address, reads the sub-depth allocated by the pixel column where the pixel value is located, and adds 1 to the read value. and write back the original address, After reading all the pixels, the field programmable logic gate array adds the values stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of the pixel values to obtain the total number of pixels. Said luminance domain representation. 11. An image enhancement device, comprising: an acquisition module, configured to acquire a luminance domain representation of an image, where the luminance domain representation is used to characterize the pixel value distribution of the image in the luminance domain; a calculation module for calculating a stretch factor based on the distribution of pixel values; and An update module, configured to update the pixel value of the image in the luminance domain based on the stretch coefficient. 12. A field programmable logic gate array for performing the method of any one of claims 1 to 8, wherein the operation involved in the method is a quantized fixed-point number operation. 13. The field programmable logic gate array of claim 12, wherein, The field programmable logic gate array adopts multi-port BRAM memory, The field programmable logic gate array divides the depth of the multi-port BRAM memory into at least two sub-depths, each sub-depth corresponds to at least 256 memory addresses, and adjacent pixel columns in the image are assigned to different sub-depths , The field programmable logic gate array uses the read pixel value of the image in the luminance domain as an address, reads the sub-depth allocated by the pixel column where the pixel value is located, and adds 1 to the read value. and write back the original address, After reading all the pixels, the field programmable logic gate array adds the values stored in the storage addresses corresponding to the same pixel value in different sub-depths, and uses the addition result as the number of the pixel values to obtain the total number of pixels. Said luminance domain representation. 14. An imaging device, comprising: an imaging device and a field programmable logic gate array, The field programmable logic gate array is used to execute the method according to any one of claims 1 to 8, so as to enhance the image captured by the imaging device, wherein the operation involved in the method is obtained by quantization fixed-point arithmetic. 15. A computing device comprising: processor; and A memory having executable code stored thereon which, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 8. 16. A computer program product comprising executable code which, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1 to 8. 17. A non-transitory machine-readable storage medium having executable codes stored thereon which, when executed by a processor of an electronic device, cause the processor to execute any of claims 1 to 8 one of the methods described.

Images