WO2021189222A1

WO2021189222A1 - Image enhancement processing method and device

Info

Publication number: WO2021189222A1
Application number: PCT/CN2020/080774
Authority: WO
Inventors: 曹忆南; 张文红; 邵涛
Original assignee: 华为技术有限公司
Priority date: 2020-03-24
Filing date: 2020-03-24
Publication date: 2021-09-30
Also published as: CN115298688A

Abstract

Provided are an image enhancement processing method and device. The image enhancement processing method comprises: acquiring a histogram of an image to be processed; dividing the histogram into n sub-histograms, n being a positive integer; calculating a mapping curve of each of the n sub-histograms, a first sub-histogram being any one of the n sub-histograms; synthesizing mapping curves of the n sub-histograms to obtain a mapping curve of the histogram; performing, according to the mapping curve of the histogram, enhancement processing on said image. The prevent invention not only achieves the purpose of fine adjustment, but also considers the adjustment of both brightness distribution and contrast distribution, achieving the purpose of balancing.

Description

Image enhancement processing method and device

Technical field

This application relates to image processing technology, and in particular to an image enhancement processing method and device.

Background technique

The image signal processor (Image Signal Processor, ISP) is mainly responsible for processing the signal output by the image sensor, and the signal is usually the RAW image format (RAW Image Format) output by the image sensor. ISP includes a series of processing sub-modules such as brightness, contrast, color, noise, zoom, etc., which can process RAW format images into images suitable for human observation or display. Among them, the brightness and contrast processing sub-module is mainly responsible for adjusting the brightness distribution and contrast of the image signal. Usually a global tone mapping curve (Global Tone Mapping Curve, GTMC) is used to remap the global histogram of the image to make the global brightness of the image The distribution is more reasonable. How to maximize the role of GTMC has become one of the key factors to improve image quality and ensure video stability.

In the related art, the local brightness distribution information of each image block in the image is analyzed first to obtain the histogram of each image block, and then the equalization operation is performed on each histogram to obtain the color scale map of each image block Curve (Tone Mapping Curve, TMC), the role of TMC is to map the brightness of the input pixel to the output brightness to achieve the purpose of adjusting the image brightness distribution, such as image brightness enhancement, contrast adjustment and other processing often use TMC as an adjustment means . The least square fitting is performed on the TMC of all image blocks on the image to obtain the GTMC of the whole image, and finally the image is enhanced globally according to GTMC.

However, the above method adopts indifferent adjustment of the brightness of all image blocks, which may cause problems such as large local noise of the image and poor controllability.

Summary of the invention

The present application provides an image enhancement processing method and device, which can not only achieve the purpose of fine adjustment, but also adjust the brightness distribution and contrast distribution to achieve the purpose of balance.

In a first aspect, the present application provides an image enhancement processing method, including: acquiring a histogram of an image to be processed; dividing the histogram into n sub-histograms, where n is a positive integer; calculating each of the n sub-histograms The mapping curve of each sub-histogram; synthesize the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram; perform enhancement processing on the image to be processed according to the mapping curve of the histogram.

This application divides the histogram of the image to be processed into multiple sub-histograms, and obtains corresponding mapping curves for each sub-histogram respectively, and then synthesizes these mapping curves to obtain the mapping curve of the histogram, so as to realize the enhancement processing of the image to be processed, Separate processing of the sub-histograms can not only achieve the purpose of fine adjustment, but also take into account the overall distribution of the image and achieve the purpose of balance.

In a possible implementation manner, the calculating the mapping curve of each sub-histogram of the n sub-histograms includes: extracting multiple features of each sub-histogram; determining each sub-histogram according to the multiple features Mapping curve. It should be noted that at least one feature can also be extracted from the feature of the sub-histogram, which is not specifically limited.

In a possible implementation manner, the multiple features include one or more of a brightness average value, a brightness standard deviation, and a pixel percentage of a sub-histogram, and the pixel percentage is the number of pixels included in the sub-histogram The percentage of the number of pixels included in the histogram.

In a possible implementation manner, the determining the mapping curve of each sub-histogram according to the multiple features includes: calculating the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple features; The gain curve coefficient and the equalization curve weight determine the mapping curve of each sub-histogram.

The key to obtaining the mapping curve of the sub-histogram in this application is the calculation of the gain curve coefficient and the equalization curve weight, which can parameterize the generation process of the mapping curve and improve the output efficiency and accuracy. In addition, the gain curve coefficient and the equalization curve weight are introduced to each sub-histogram, and the brightness distribution and contrast distribution of each sub-histogram can be adjusted to achieve the purpose of equalization.

In a possible implementation manner, the calculating the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple features includes: inputting the multiple features into the first histogram obtained by pre-training to enhance The coefficient generator obtains the gain curve coefficients, and the first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; the plurality of features are input into the second histogram enhancement coefficient generator obtained by pre-training to obtain the The equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.

In a possible implementation manner, the calculating the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple features includes: combining the multiple features with the multiple adjacent sub-histograms The features are all input to the first histogram enhancement coefficient generator obtained by pre-training to obtain the gain curve coefficient, and the first histogram enhancement coefficient generator is used to calculate the gain curve coefficient; The multiple features of the neighbor histogram are all input to a second histogram enhancement coefficient generator obtained by pre-training to obtain the equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.

In a possible implementation manner, the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator include an adaptive neuro-fuzzy system or a neural network.

In a possible implementation manner, each sub-histogram in the n sub-histograms is a histogram processed by low-pass filtering.

The sub-histogram obtained in this way can have a smooth transition from the data to the data 0 on the ordinate, without causing a cliff-like mutation, so as to ensure the smooth change of the extracted feature value, and then to ensure the calculated mapping curve The smooth change of the image is used to ensure the stability of the image brightness when the mapping curve is used to process the image.

In a possible implementation manner, before dividing the histogram into n sub-histograms, the method further includes: performing brightness transformation processing on the histogram; and dividing the histogram into n sub-histograms , Including: dividing the histogram after the brightness transformation into n sub-histograms.

In a possible implementation manner, after dividing the histogram into n sub-histograms, the method further includes: performing brightness transformation processing on each sub-histogram in the n sub-histograms.

In a possible implementation manner, the synthesizing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram includes: splicing the mapping curves of the n sub-histograms to obtain the The mapping curve of the histogram, the value at the junction of the mapping curves of two adjacent sub-histograms takes the average value, the maximum value or the minimum value, and the average value, the maximum value or the minimum value is the two adjacent sub-histograms The average, maximum or minimum of the two values of the mapping curve at the connection ends of each other.

In order to ensure the monotonicity of GTMC, the image brightness maintains the original relative relationship, and there will be no reverse.

In a possible implementation manner, the method further includes: obtaining the adaptive neuro-fuzzy system by training according to a first set parameter, where the first set parameter includes input feature membership function parameters, output membership function parameters, rules function.

In a possible implementation manner, the method further includes: obtaining the neural network by training according to a second set parameter, where the second set parameter includes different combinations of the multiple features, membership function calculation nodes, and rule functions Design the corresponding output value.

In a possible implementation manner, the calculation to obtain the mapping curve of the first sub-histogram according to the gain curve coefficient and the equalization curve weight includes:

The mapping curve of the first sub-histogram is calculated according to formula (1):

y _i =α _i ×y _{i_gain} +β _i ×y _{i_hist} (1)

Wherein, y _i represents the mapping curve of the first sub-histogram, γ _i represents the gain curve coefficient, β _i represents the weight of the equalization curve, α _i =1-β _i , y _{i_gain} represents the first sub-histogram The gain curve of the histogram,

j∈[0,m], y _{i_hist} represents the equalization curve of the first sub-histogram.

In a second aspect, the present application provides an image enhancement processing device, including: an acquisition module for acquiring a histogram of an image to be processed; a processing module for dividing the histogram into n sub-histograms, where n is a positive integer Calculate the mapping curve of each sub-histogram in the n sub-histograms; synthesize the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram; compare the mapping curve of the histogram according to the mapping curve of the histogram Process the image for enhancement.

In a possible implementation manner, the processing module is specifically configured to extract multiple features of each sub-histogram; and determine the mapping curve of each sub-histogram according to the multiple features.

In a possible implementation manner, the processing module is specifically configured to calculate the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple characteristics; determine according to the gain curve coefficient and the equalization curve weight The mapping curve of each sub-histogram.

In a possible implementation manner, the processing module is specifically configured to input the multiple features into the first histogram enhancement coefficient generator obtained by pre-training to obtain the gain curve coefficients, and the first histogram The graph enhancement coefficient generator is used to calculate the gain curve coefficients; the second histogram enhancement coefficient generator obtained by pre-training the multiple features is input to obtain the equalization curve weight, and the second histogram enhancement coefficient generator is used for Calculate the weight of the equilibrium curve.

In a possible implementation manner, the processing module is specifically configured to input the multiple features and the multiple features of the neighboring sub-histograms into the first histogram enhancement coefficient generator obtained by pre-training The gain curve coefficient is obtained, and the first histogram enhancement coefficient generator is used to calculate the gain curve coefficient; and the multiple features and the multiple features of the adjacent sub-histograms are input into pre-trained The second histogram enhancement coefficient generator obtains the equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.

In a possible implementation manner, the processing module is further configured to perform brightness conversion processing on the histogram; divide the histogram after the brightness conversion into n sub-histograms.

In a possible implementation manner, the processing module is further configured to perform brightness transformation processing on each sub-histogram in the n sub-histograms.

In a possible implementation manner, the processing module is specifically configured to splice the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram, and the mapping curve of the two adjacent sub-histograms is The value at the connection takes the average value, maximum value or minimum value, and the average value, maximum value or minimum value is the average value of the two values of the mapping curves of the two adjacent sub-histograms at the connection ends of each other, and the maximum value Value or minimum value.

In a possible implementation manner, the processing module is further configured to train the adaptive neuro-fuzzy system according to a first set parameter, and the first set parameter includes input feature membership function parameters, output membership Degree function parameters, rule functions.

In a possible implementation, the processing module is further configured to train the neural network according to a second set parameter, where the second set parameter includes the multiple features, membership function calculation nodes, Different combination designs in the rule function correspond to output values.

In a possible implementation manner, the processing module is specifically configured to calculate the mapping curve of the first sub-histogram according to formula (1):

y _i =α _i ×y _{i_gain} +β _i ×y _{i_hist} (1)

In a third aspect, the present application provides an image enhancement processing device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are processed by the one or more The processor executes, so that the one or more processors implement the method according to any one of the above-mentioned first aspects.

In a fourth aspect, the present application provides a computer-readable storage medium that stores instructions, and when the instructions are run on a computer, they are used to execute any of the above-mentioned instructions in the first aspect. method.

Description of the drawings

Fig. 1 exemplarily shows a schematic structural diagram of a terminal device 100;

Fig. 2 shows an exemplary structural schematic diagram of the image acquisition device;

FIG. 3 is a flowchart of an embodiment of an image enhancement processing method provided by an embodiment of the application;

Figure 4 shows an exemplary schematic diagram of the gamma curve;

Fig. 5 shows an exemplary coefficient diagram of the filter;

FIG. 6 is a schematic structural diagram of an embodiment of an image enhancement processing apparatus of this application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be described clearly and completely in conjunction with the accompanying drawings in this application. Obviously, the described embodiments are part of the embodiments of this application. , Not all examples. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first", "second", etc. in the specification embodiments, claims, and drawings of this application are only used for the purpose of distinguishing description, and cannot be understood as indicating or implying relative importance, nor can it be understood as indicating Or imply the order. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device need not be limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

It should be understood that in this application, "at least one (item)" refers to one or more, and "multiple" refers to two or more. "And/or" is used to describe the association relationship of associated objects, indicating that there can be three types of relationships, for example, "A and/or B" can mean: only A, only B, and both A and B , Where A and B can be singular or plural. The character "/" generally indicates that the associated objects before and after are in an "or" relationship. "The following at least one item (a)" or similar expressions refers to any combination of these items, including any combination of a single item (a) or a plurality of items (a). For example, at least one of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c" ", where a, b, and c can be single or multiple.

The terminal equipment of this application can also be called user equipment (UE), which can be deployed on land, including indoor or outdoor, handheld or vehicle-mounted; it can also be deployed on the water (such as ships, etc.); it can also be deployed on In the air (for example, airplanes, balloons, satellites, etc.). The terminal device can be a terminal device 100 (mobile phone), a tablet computer (pad), a computer with wireless transceiver function, a virtual reality (VR) device, an augmented reality (AR) device, a monitoring device, and a smart phone. This application is not limited to screens, smart TVs, wireless devices in remote medical (remote medical), or wireless devices in smart home (smart home).

Fig. 1 exemplarily shows a schematic structural diagram of a terminal device 100. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 151, and a wireless communication module 152 , Audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, SIM card interface 195 and so on. The sensor module 180 may include a gyroscope sensor 180A, an acceleration sensor 180B, a proximity light sensor 180G, a fingerprint sensor 180H, a touch sensor 180K, and an image sensor 180N (Of course, the terminal device 100 may also include other sensors, such as temperature sensors, pressure sensors, Distance sensors, magnetic sensors, ambient light sensors, air pressure sensors, bone conduction sensors, etc. (not shown in the figure).

It can be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU) Wait. Among them, the different processing units may be independent devices or integrated in one or more processors. The controller may be the nerve center and command center of the terminal device 100. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.

A memory may also be provided in the processor 110 to store instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.

When the processor 110 integrates different devices, such as integrated CPU and GPU, the CPU and GPU can cooperate to execute the method provided in the embodiment of the present application. For example, part of the algorithm in the method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain Faster processing efficiency.

The display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode). emitting diode, AMOLED, flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 194, and N is a positive integer greater than one.

The camera 193 (a front camera or a rear camera, or a camera can be used as a front camera or a rear camera) is used to capture still images or videos. Generally, the camera 193 may include photosensitive elements such as a lens group and an image sensor, where the lens group includes a plurality of lenses (convex lens or concave lens) for collecting light signals reflected by the object to be photographed and transmitting the collected light signals to the image sensor . The image sensor generates an original image of the object to be photographed according to the light signal.

The internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions. The processor 110 executes various functional applications and signal processing of the terminal device 100 by running instructions stored in the internal memory 121. The internal memory 121 may include a storage program area and a storage data area. Among them, the storage program area can store operating system, application program (such as camera application, WeChat application, etc.) codes and so on. The storage data area can store data created during the use of the terminal device 100 (such as images and videos collected by a camera application) and the like.

In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

Of course, the code of the method provided in the embodiments of the present application may also be stored in an external memory. In this case, the processor 110 may run the code stored in the external memory through the external memory interface 120.

The function of the sensor module 180 is described below.

The gyroscope sensor 180A may be used to determine the movement posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (ie, x, y, and z axes) can be determined by the gyro sensor 180A. That is, the gyro sensor 180A can be used to detect the current motion state of the terminal device 100, such as shaking or static.

The acceleration sensor 180B can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). That is, the acceleration sensor 180B can be used to detect the current motion state of the terminal device 100, such as shaking or static.

The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light to the outside through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100.

The gyroscope sensor 180A (or the acceleration sensor 180B) may send the detected motion state information (such as angular velocity) to the processor 110. The processor 110 determines whether it is currently in the hand-held state or the tripod state based on the motion state information (for example, when the angular velocity is not 0, it means that the terminal device 100 is in the hand-held state).

The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, and so on.

Touch sensor 180K, also called "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”. The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. The visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the terminal device 100, which is different from the position of the display screen 194.

Exemplarily, the display screen 194 of the terminal device 100 displays a main interface, and the main interface includes icons of multiple applications (such as a camera application, a WeChat application, etc.). The user clicks the icon of the camera application in the main interface through the touch sensor 180K, triggers the processor 110 to start the camera application, and turns on the camera 193. The display screen 194 displays an interface of the camera application, such as a viewfinder interface.

The image sensor 180N uses the photoelectric conversion function of the photoelectric device to convert the light signal on the photosensitive surface into an electrical signal proportional to the light signal. It can generate raw image data (RAW format image) of the object to be photographed based on the light signal.

The wireless communication function of the terminal device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 151, the wireless communication module 152, the modem processor, and the baseband processor.

The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna can be used in combination with a tuning switch.

The mobile communication module 151 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the terminal device 100. The mobile communication module 151 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 151 can receive electromagnetic waves by the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 151 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave radiation via the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 151 may be provided in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 151 and at least part of the modules of the processor 110 may be provided in the same device.

The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 151 or other functional modules.

The wireless communication module 152 can provide applications on the terminal device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellites. System (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 152 may be one or more devices integrating at least one communication processing module. The wireless communication module 152 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 152 can also receive the signal to be sent from the processor 110, perform frequency modulation and amplification, and convert it into electromagnetic waves to radiate through the antenna 2.

In some embodiments, the antenna 1 of the terminal device 100 is coupled with the mobile communication module 151, and the antenna 2 is coupled with the wireless communication module 152, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).

In addition, the terminal device 100 may implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc. The terminal device 100 may receive a key 190 input, and generate a key signal input related to user settings and function control of the terminal device 100. The terminal device 100 may use the motor 191 to generate a vibration notification (such as a vibration notification of an incoming call). The indicator 192 in the terminal device 100 may be an indicator light, which may be used to indicate the charging status, power change, or to indicate messages, missed calls, notifications, and so on. The SIM card interface 195 in the terminal device 100 is used to connect to a SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the terminal device 100.

It should be understood that, in actual applications, the terminal device 100 may include more or less components than those shown in FIG. 1, which is not limited in the embodiment of the present application.

FIG. 2 shows an exemplary structural diagram of an image acquisition device. As shown in FIG. 2, the image acquisition device includes a lens 201, an image sensor 202, and an image signal processor (ISP) 203. The lens 201 is used to capture still images or videos, collect light signals reflected by objects to be photographed, and transmit the collected light signals to the image sensor. The image sensor 202 generates raw image data (RAW format image) of the object to be photographed according to the light signal. ISP 203 is used to process the RAW format image into an image suitable for human observation or display, and in this process, it adjusts the visible light exposure parameters and/or the intensity of the infrared fill light until the automatic exposure (AE) is satisfied. ) The convergence condition of the algorithm. The lens 201 in this embodiment may be the camera 193 in the terminal device shown in FIG. 1, the image sensor 202 may be the image sensor 180N in the sensor module 180 in the terminal device shown in FIG. 1, and the ISP 203 may be as shown in FIG. The processor 110 in the terminal device.

Optionally, the image acquisition device may adopt a single lens plus a single image sensor, or a dual lens plus a dual image sensor, or a single lens plus a beam splitter and a dual image sensor structure. The single lens structure can save cost, while the single image The structure of the sensor can simplify the structure of the camera. This application does not specifically limit this.

FIG. 3 is a flowchart of an embodiment of an image enhancement processing method provided by an embodiment of the application. As shown in FIG. 3, the method in this embodiment may be executed by the above-mentioned terminal device, image acquisition device, or processor. The image enhancement processing method may include:

Step 301: Obtain a histogram of the image to be processed.

The histogram is a kind of brightness statistical graph. Its abscissa is the brightness value of the pixels in the image to be processed, and the ordinate is the number of pixels corresponding to the brightness value (representing how many pixels in the image to be processed are of a certain brightness value) ). It should be noted that the brightness value in the histogram can be the brightness value obtained by sampling according to a certain granularity. For example, the brightness range of the image to be processed is 0-255. If sampling at each integer value, the horizontal of the histogram The coordinates include 256 coordinate values, namely 0, 1, ..., 254, 255; if an integer value is sampled every interval, the abscissa of the histogram includes 128 coordinate values, namely 0, 2, ..., 252, 254; if Every 0.5 integer values are sampled, the abscissa of the histogram includes 512 coordinate values, namely 0, 0.5, ..., 254.5, 255. There is no specific limitation on this application. The number of pixels in the histogram can also be sampled at a certain granularity. For example, the pixel array of the image to be processed is 400×600. If sampling is done for each pixel, the sum of all the pixels in the histogram is 400×600; if sampling every other pixel, the sum of all the pixels in the histogram is half of 400×600. This application is not specifically limited.

Step 302: Divide the histogram into n sub-histograms.

n is a positive integer. The division of the histogram in this application is based on the abscissa of the histogram. To ensure smooth transition between sub-histograms, two adjacent sub-histograms may or may not overlap. For example, the abscissa of the histogram includes 256 coordinate values 0-255, and each histogram includes 64 coordinate values. The histogram can be divided into 4 sub-histograms that do not overlap each other, and the range of the corresponding abscissa is [0,63], [64,127], [128,191] and [192,255]. For another example, the abscissa of the histogram includes 256 coordinate values 0-255, and each histogram includes 70 coordinate values. The histogram can be divided into 4 sub-histograms where two adjacent sub-histograms will overlap, respectively. The corresponding abscissa ranges are [0,69], [61,130], [125,194] and [186,255]. For another example, the abscissa of the histogram includes 256 coordinate values 0-255. The histogram can be divided into 4 sub-histograms that do not overlap each other. The ranges of the corresponding abscissas are [0,69], [70,130], [131,191] and [192,255], the number of coordinate values included in the four sub-histograms are not completely equal. It should be noted that the division of the histogram of the image to be processed can be implemented using multiple rules, such as equal length/unequal length division, overlapping division, non-overlapping division, etc., which are not specifically limited in this application.

Since there are many brightness-related processing modules in the ISP that will have a greater impact on the brightness of the image, the most important impact on the brightness is the gamma curve. Figure 4 shows an exemplary schematic diagram of the gamma curve, such as As shown in FIG. 4, the abscissa represents the original brightness value, the ordinate represents the mapped brightness value, and the gamma curve is used to represent the correspondence between the original brightness value and the mapped brightness value. Therefore, in order to extract more accurate image features in subsequent steps, the histogram may be subjected to brightness conversion processing, which may include gamma conversion or other brightness conversion methods, which is not specifically limited in this application.

In a possible implementation manner, the above-mentioned brightness conversion process may be performed on the histogram of the image to be processed first, and then the histogram after the brightness conversion is divided into n sub-histograms according to the abscissa of the histogram after the brightness conversion.

In a possible implementation manner, after the histogram is divided into n sub-histograms according to the abscissa of the histogram, the above-mentioned brightness transformation processing is performed on each sub-histogram separately.

In addition, after the application divides the histogram of the image to be processed into n sub-histograms, the dynamic range of each sub-histogram can be expanded. As described above, the range of the abscissa of each sub-histogram is a subset of the range of the abscissa of the histogram of the image to be processed, and the range of the abscissa of each sub-histogram can be extended to the abscissa of the histogram of the image to be processed. The range of coordinates, but the value of the ordinate in the extended range is 0. For example, the range of the abscissa is the sub-histogram of [70,130], and the range of the abscissa after expansion is [0,255], but the ordinates corresponding to the abscissas except [70,130] are all 0, indicating that The histogram has no corresponding pixels in these brightness values.

Optionally, the present application can also perform enhancement processing on each sub-histogram, and the enhancement processing can include equalization processing (enlarge or reduce the spacing of the entire histogram proportionally on the abscissa), left shift processing (the entire histogram is Left move x + coordinate units) or right shift processing (the entire histogram moves to the right x- coordinate units), etc. The specific method used to enhance a certain sub-histogram can be based on the brightness distribution of the sub-histogram, or according to the brightness distribution of the sub-histogram and the brightness distribution of the sub-histogram adjacent to the sub-histogram. Sure. This application does not specifically limit this.

Step 303: Calculate the mapping curve of each sub-histogram in the n sub-histograms.

The first sub-histogram is any one of the n sub-histograms. The step 303 is described below by taking the first sub-histogram as an example.

Optionally, the present application may perform low-pass filtering processing on the first sub-histogram, for example, a Gaussian filter with a mean value of 0 filter coefficients and a variance of 10 may be used. Fig. 5 shows an exemplary coefficient diagram of the filter. As shown in Fig. 5, the abscissa represents the brightness of the sub-histogram, and the ordinate represents the weight of the corresponding brightness. For example, if the design sub-histogram includes 16 coordinate values ([16,32]), half of the coordinate values ([8,16] and [32,40]) are taken from the left and right adjacent sub-histograms, and the sub-histogram is added All the coordinate values of the graph compose a sub-histogram containing 32 coordinate values ([8,40]). The weight in this sub-histogram can be set to 1, that is, the horizontal coordinate [16,32] corresponds to the vertical coordinate 1, which corresponds to The weight in the neighbor histogram changes Gaussian. The farther away from the histogram, the smaller the weight, that is, the smaller the abscissa in the abscissa [8,16], the smaller the weight, and the larger the abscissa in the abscissa [32,40]. The smaller the weight. The sub-histogram obtained in this way can have a smooth transition from the data to the data 0 on the ordinate, without causing a cliff-like mutation, so as to ensure the smooth change of the extracted feature value, and then to ensure the calculated mapping curve The smooth change of the image is used to ensure the stability of the image brightness when the mapping curve is used to process the image.

This application first extracts multiple features of the first sub-histogram; calculates the gain curve coefficients and equalization curve weights of the first sub-histogram based on the multiple features; calculates the gain curve coefficients and equalization curve weights of the first sub-histogram based on the gain curve coefficients and the equalization curve weights Map the curve.

The multiple features of the above-mentioned first sub-histogram may include, but are not limited to, features such as gray-level mean value and variance. Preferably, the brightness average value, the brightness standard deviation and the pixel percentage of the first sub-histogram.

Among them, the brightness average refers to the weighted average of all the brightness values included in the first sub-histogram, and the calculation formula can be

r represents the average brightness value, x _i represents the i-th brightness value, a _i represents the weight of the i-th brightness value, and N represents the number of brightness values included in the first sub-histogram. The weight of each brightness value is determined according to the ordinate value corresponding to the brightness value. The higher the ordinate value, the greater the weight value. If the dynamic range of each sub-histogram has been extended, the weight value can be set to 0 for the brightness value corresponding to the ordinate value of 0.

The brightness standard deviation refers to the mean square deviation of all the brightness values and the brightness average included in the first sub-histogram, and the calculation formula can be

δ(r) represents the standard deviation of brightness.

Pixel percentage refers to the proportion of the number of pixels included in the first sub-histogram in the number of pixels included in the histogram of the image to be processed, and its calculation formula can be

Among them, p represents the percentage of pixels, m represents the number of pixels included in the first sub-histogram, and M represents the number of pixels included in the histogram of the image to be processed.

As mentioned above, when dividing the histogram of the image to be processed, there may or may not be overlap between the sub-histograms. In the case of no overlap, when calculating the above eigenvalues, the value range of the parameters in the formula is usually limited to the range of the abscissa of the first sub-histogram, for example, the abscissa of the first sub-histogram The range is [0,63], then when calculating the brightness mean and the brightness standard deviation, _{the value range of x i} is in [0,63], specifically 0, 1, ..., 62, 63, and when counting m, it is in 0, 1,..., 62, 63, these abscissas correspond to the ordinates. Optionally, in this case, the value range of the parameter in the formula may also exceed the range of the abscissa of the first sub-histogram. For example, _{the value range of x i} is in [0,65], specifically 0, 1,..., 64, 65, when counting m, take the value on the ordinate corresponding to the abscissas of 0, 1,..., 64, 65. Or the value range of the parameter in the formula is smaller than the range of the abscissa of the first sub-histogram. For example, _{the value range of x i} is [0,60], specifically 0, 1,..., 59, 60, statistics In the case of m, take the value on the ordinate corresponding to the abscissas of 0, 1, ..., 59, and 60. In the case of overlap, when calculating the above eigenvalues, the value range of the parameter in the formula can only take a subset of the abscissa of the first sub-histogram that does not overlap, for example, the first sub-histogram The range of the abscissa is [0,69], where [61,69] overlaps with the adjacent sub-histogram, then when calculating the brightness mean and the brightness standard deviation, _{the value range of x i} is in [0,60], Specifically, they are 0, 1,..., 59, 60, and when counting m, take values on the ordinates corresponding to the abscissas of 0, 1,..., 59, and 60. Optionally, in this case, the value range of the parameter in the formula can also be a subset that does not overlap in the range of the abscissa of the first sub-histogram, or the range of the abscissa of the first sub-histogram. There are overlapping subsets in, for example, _{the value range of x i} is in [0,69], specifically 0, 1,..., 68, 69, and m is in the range of 0, 1,..., 68, 69 These abscissas correspond to values on the ordinates. Optionally, in this case, the value range of the parameter in the formula can also be a subset that does not overlap in the range of the abscissa of the first sub-histogram, or the range of the abscissa of the first sub-histogram. For example, _{the value range of x i} is in [0,69], specifically 0, 1,..., 64, 65, and m is in the range of 0, 1,..., 64 and 65 are taken on the ordinate corresponding to these abscissas.

It should be noted that when extracting the feature value of the sub-histogram in this application, the value range of the parameter in the formula can only take all or part of the value within the range of the abscissa of the corresponding sub-histogram, or it can take the corresponding The value of the extended part outside the range of the abscissa of the sub-histogram can only take the subset of the corresponding sub-histogram that does not overlap within the range of the abscissa, or it can also take the corresponding sub-histogram that does not have the range of the abscissa All or part of overlapping subsets and overlapping subsets. This application does not specifically limit this.

This application calculates the gain curve coefficients and equalization curve weights of the first sub-histogram according to the above multiple features may include: inputting multiple features into the first histogram enhancement coefficient generator obtained by pre-training to obtain the gain curve coefficients, first The histogram enhancement coefficient generator is used to calculate the gain curve coefficients; the second histogram enhancement coefficient generator obtained by pre-training multiple features is input to obtain the equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.

The first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator include an Adaptive Network-based Fuzzy Inference System (ANFIS) or a neural network. The first histogram enhancement coefficient generator is used to output gain curve coefficients, and the second histogram enhancement coefficient generator is used to output equalization curve weights.

The following takes ANFIS as an example to illustrate the output process of gain curve coefficients. The standard ANFIS includes an input layer, a membership layer, a rule layer and a defuzzification layer.

(1) The input layer receives multiple features of the first sub-histogram, and the multiple features may be, for example, the brightness average, the brightness standard deviation, and the pixel percentage of the first sub-histogram. It should be noted that the multiple features are not limited to the aforementioned three, and may also include other features, which are not specifically limited.

(2) The membership level has multiple membership functions for each feature. Calculate the membership degree of each characteristic value in the corresponding gear according to the membership function. The membership function can be triangular, trapezoidal, Gaussian, etc. The membership function between each gear can have overlapping parts to ensure a smooth transition. The membership function of each eigenvalue is different, and the number of gears that can be designed is different according to needs. For example, the designed membership function is trapezoidal with three gears, ranging from 1 to 8 for low gears, 12-18 for middle gears, 22-30 for high gears, 8-12 for low gears and middle gears trapezoidal oblique The overlapping area of the side, 18-22 is the overlapping area of the trapezoidal hypotenuse of the middle gear and the high gear. If the characteristic value is the mean value of brightness, enter 5, then the membership degrees of the three gears are calculated to be 1,0,0, and enter 10, then the membership degrees of the three gears are calculated to be 0.5, 0.5, 0.

(3) The rule layer adopts the fuzzy inference system (Fuzzy Inference System, FIS) standard rule method, covering all possible combinations of input to the membership function. There are many combinations for different gears of different eigenvalues. The rule layer designs the combination of eigenvalues, calculates the combination weight according to the degree of membership, and designs the value of the corresponding system output coefficient of the combination.

(4) The defuzzification layer adopts standard FIS implementation methods, including but not limited to the maximum membership method, the center of gravity method, and the weighted average method. The defuzzification layer calculates the gain curve coefficient that the system needs to output according to the calculation result of the weight and the design scheme of the rule layer. The calculation formula of the gain curve coefficient can be

Among them, γ _i represents the gain curve coefficient of the i-th sub-histogram, b _j (j = 1, 2, ..., k) represents the weight of the combination of the i-th sub-histogram output by the rule layer, and γ _j represents the design of the rule layer Corresponding to the j-th gear value of the required output coefficient (gain curve coefficient) of the combined system.

The following also takes ANFIS as an example to describe the output process of the equalization curve weight.

(3) The rule layer adopts the fuzzy inference system (Fuzzy Inference System, FIS) standard rule method, covering all possible combinations of input to the membership function. The difference between the calculation of the equalization curve weight and the above calculation of the gain curve coefficient lies in the rule layer, including the design of the membership function of the membership layer, the combination of eigenvalues, the calculation of the combination weight, and the value of the system output coefficient corresponding to the combination. .

(4) The defuzzification layer adopts standard FIS implementation methods, including but not limited to the maximum membership method, the center of gravity method, and the weighted average method. The defuzzification layer calculates the gain curve coefficient that the system needs to output according to the calculation result of the weight and the design scheme of the rule layer. The formula for calculating the weight of the equilibrium curve can be

Among them, β _i represents the weight of the equalization curve of the i-th sub-histogram, c _j (j = 1, 2,..., l) represents the weight of the combination of the i-th sub-histogram output by the rule layer, and β _j represents the design of the rule layer Corresponding to the j-th gear value of the required output coefficient (equalization curve weight) of the combined system.

Both the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator are controlled by a series of parameters to control their input and output characteristics. For ANFIS, the series of parameters can include input feature membership function parameters, output membership function parameters, rule functions, and so on. For neural networks, the series of parameters can include multiple features, membership function calculation nodes, and rule functions to design corresponding output values for different combinations. The generation process of these parameters includes: a) Collect image data training set 1, including data set construction, terminal equipment shooting natural scenes under different weather, different lighting conditions, etc., initialize the above parameters, and select typical scenes to observe its sub-histograms The membership function is initially designed. The combined values of the first enhanced coefficient rule layer are initialized to 1, and the combined values of the second enhanced coefficient rule layer are initialized to a smaller value, such as 5; the parameter set 1 is obtained, including each Sub-histogram membership functions, generator output coefficients in different combinations of design values in the rule layer, etc.; b) Use the initial histogram enhancement coefficient generator on training set 1 and parameter set 1 to generate coefficient set 1, including the first straight The gain curve coefficient output by the graph enhancement coefficient generator and the equalization curve weight output by the second histogram enhancement coefficient generator; c) Use coefficient set 1 to generate GTMC curve set 1, and apply GTMC curve set 1 to training set 1, and get Output set 1, and observe the effect; d) According to the brightness and contrast effect of output set 1, correct coefficient set 1 to obtain coefficient label set 1. e) Perform parameter training on ANFIS according to training set 1 and coefficient label set 1. Use and are not limited to general training methods such as gradient descent method and genetic algorithm; use the trained parameter set 2 to generate coefficient set 2; f) replace parameter set 2 and coefficient set 2 with parameter set 1 and coefficient set 1, and repeat b～e Step, until the image effect reaches the target, a trained histogram enhancement coefficient generator is obtained. It can be seen that using ANFIS or neural network as a histogram enhancement coefficient generator for gain curve coefficients and equalization curve weights, on the one hand, the histogram enhancement coefficient generator itself can be optimized by the method of generation-inspection-correction-training-correction-training Its control parameters can be adjusted locally according to the characteristics of histogram distribution in different scenarios, even if the trained histogram enhancement coefficient generator is used to modify control parameters such as membership functions or rules in a certain scenario, It will not affect the strategy of generating curves in other scenes. On the other hand, the gain curve coefficients and equalization curve weights obtained by the histogram enhancement coefficient generator are more in line with the brightness enhancement requirements of the image, and help to improve the image processing efficiency.

Optionally, this application may input multiple features and multiple features of adjacent sub-histograms into the pre-trained first histogram enhancement coefficient generator or the second histogram enhancement coefficient generator when calculating the gain curve coefficients and the equalization curve weights. The histogram enhancement coefficient generator, since the output data in this way refers to the brightness distribution of the adjacent sub-histograms, a smooth transition of brightness can be achieved when the scene changes.

After the gain curve coefficient and the equalization curve weight are obtained, the present application can calculate the mapping curve of the first sub-histogram according to the gain curve coefficient and the equalization curve weight. According to formula (1), the mapping curve of the i-th sub-histogram is calculated:

y _i =α _i ×y _{i_gain} +β _i ×y _{i_hist} (1)

Among them, y _i represents the mapping curve of the i-th sub-histogram, β _i represents the weight of the equalization curve of the i-th sub-histogram, α _i = 1-β _i , and y _{i_gain} represents the gain curve of the i-th sub-histogram, which is used for Adjust the curve of image brightness,

j∈[0,m], γ _i represents the gain curve coefficient of the i-th sub-histogram, pow(j, γ _i )=j∧γ _i , L represents the maximum grayscale value of the image, and y _{i_hist} represents the i-th sub-histogram The equalization curve in the figure is a curve used to adjust the contrast of the image.

It can be seen that the key to obtaining the mapping curve of the sub-histogram in this application is the calculation of the gain curve coefficient and the equalization curve weight, which can parameterize the generation process of the mapping curve and improve the output efficiency and accuracy. In addition, the gain curve coefficient and the equalization curve weight are introduced to each sub-histogram, and the brightness distribution and contrast distribution of each sub-histogram can be adjusted to achieve the purpose of equalization.

Step 304: Synthesize the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram.

In this application, the mapping curves of n sub-histograms can be spliced to obtain a complete GTMC, that is, the mapping curve of the histogram of the image to be processed. The principle of splicing can include: in the extended range (the abscissa of the aforementioned pair of histograms extends to the entire abscissa range of the image to be processed, and the extended range refers to other abscissas except the abscissa range of the sub-histogram before expansion The mapping curve calculated in the range) is only valid in the sub-histogram, and the mean, larger or smaller value can be taken at the splicing of two adjacent sub-histograms to ensure the monotonicity of GTMC and thereby make the image brightness Maintain the original relative relationship, and there will be no reverse.

If two adjacent sub-histograms do not overlap, the endpoint median connection method (that is, the ordinates of the endpoints of the mapping curves of the two adjacent sub-histograms are averaged), and the endpoint maximum connection method (ie The ordinate of the endpoints of the mapping curves of two adjacent sub-histograms is the larger one), the endpoint minimum connection method (that is, the ordinates of the endpoints of the mapping curves of two adjacent sub-histograms are the smaller), etc. The mapping curve splicing of the histogram. If two adjacent sub-histograms overlap, in the overlapping part, only the mapping curve of one of the sub-histograms is taken.

Example 1: The abscissa of the histogram includes 256 coordinate values 0-255. The histogram is divided into 4 sub-histograms that do not overlap each other. The range of the abscissa of the sub-histogram 1 is [0,63], the sub-histogram The range of the abscissa of Fig. 2 is [64,127], the range of the abscissa of the sub-histogram 3 is [128,191], and the range of the abscissa of the sub-histogram 3 is [192,255]. When extracting features, the value range of the parameters in the formula is usually limited to the range of the abscissa of the sub-histogram, that is, _{the value range of x i} in sub-histogram 1 is in [0,63], specifically 0, 1,..., 62, 63, and when counting m, the value is taken on the ordinate corresponding to the abscissas of 0, 1,..., 62, 63; the value range _{of x i in sub-histogram 2 is in [} 64, 127], specifically 64, 65,..., 126, 127, and when counting m, take the value on the ordinate corresponding to these abscissas of 64, 65,..., 126, 127; the value of x _i in sub-histogram 3 The value range is [128,191], which is 128, 129,..., 190, 191, and when counting m, the value is taken on the ordinate corresponding to the abscissas of 128, 129,..., 190, 191; sub-histogram 3 The value range of x _i is [192,255], specifically 192, 193,...254, 255, and when counting m, the value is taken on the ordinate corresponding to the abscissas of 192, 193,..., 254, 255. . It can be seen that in this embodiment, the abscissa range of the sub-histogram and the parameter value range when extracting features are in one-to-one correspondence and are completely consistent.

When splicing, the mapping curve of sub-histogram 1 and the mapping curve of sub-histogram 2 are spliced end to end, the mapping curve of sub-histogram 2 and the mapping curve of sub-histogram 3 are spliced end-to-end, and the mapping curve of sub-histogram 3 and sub-histogram 3 are spliced. The mapping curve is spliced end to end. The splicing place adopts the endpoint median connection method, that is, the average value 1 is calculated by taking the ordinate corresponding to 63 and 64 on the abscissa, and the average value 1 is used as the first and last transition of the mapping curve of sub-histogram 1 and the mapping curve of sub-histogram 2. Value; take the abscissa as 127 and 128 respectively and the ordinate corresponding to the average value 2 is calculated, and the average value 2 is used as the first and last transition value of the mapping curve of sub-histogram 2 and the mapping curve of sub-histogram 3; take the abscissa as 191 and The ordinate corresponding to 192 is calculated to obtain the average value 3, and the average value 3 is used as the first and last transition value of the mapping curve of the sub-histogram 3 and the mapping curve of the sub-histogram 3.

Example 2: The abscissa of the histogram includes 256 coordinate values 0-255. The histogram is divided into 4 overlapping sub-histograms. The range of the abscissa of sub-histogram 1 is [0,69]. The range of the abscissa of the histogram 2 is [61,130], the range of the abscissa of the sub-histogram 3 is [125,194], and the range of the abscissa of the sub-histogram 3 is [186,255]. When extracting features, the value range of the parameters in the formula can only take a subset of the sub-histogram's abscissa that does not overlap, that is, _{the value range of x i} in sub-histogram 1 is in [0,60], Specifically, it is 0, 1,..., 59, 60, and when counting m, the value is taken on the ordinate corresponding to the abscissas of 0, 1,..., 59, 60; the value range _{of x i in the sub-histogram 2 is} In [70,124], specifically 70, 71, ..., 123, 124, and when counting m, take the value on the ordinate corresponding to these abscissas of 70, 71, ..., 123, 124; sub-histogram 3 x _i The value range of is in [131,185], specifically 131, 132, ..., 184, 185, and when statistical m is taken, the value is taken on the ordinate corresponding to the abscissa of 131, 132, ..., 184, 185; sub-histogram _{The value range of x i} in Fig. 3 is [195,255], specifically 195, 196,...254, 255, and when counting m, it is on the ordinate corresponding to the abscissas of 195, 196,..., 254, 255. Value. It can be seen that the parameter value range when extracting features in this embodiment is a subset of the abscissa range of the corresponding sub-histogram.

When splicing, the overlapping part of two adjacent sub-histograms, only the mapping curve of the overlapping part of the right sub-histogram is taken, that is, the mapping curve corresponding to the abscissa [0,61] is taken in the sub-histogram 1, Take the mapping curve corresponding to [61,125] in sub-histogram 2, take the mapping curve corresponding to [125,186] in sub-histogram 3, take the mapping curve corresponding to [186,255] in sub-histogram 3, and divide these four mapping curves Splice end to end in order.

Example 3: The abscissa of the histogram includes 256 coordinate values 0-255. The histogram is divided into 4 overlapping sub-histograms. The range of the abscissa of sub-histogram 1 is [0,69]. The range of the abscissa of the histogram 2 is [61,130], the range of the abscissa of the sub-histogram 3 is [125,194], and the range of the abscissa of the sub-histogram 3 is [186,255]. When extracting features, the value range of the parameters in the formula can also take the subsets that do not overlap in the abscissa of the sub-histogram and the subsets that have overlap in the abscissa of the sub-histogram, that is, the sub-histogram. The value range of x _i in Figure 1 is in [0,64], specifically 0, 1, ..., 63, 64, and when counting m, it is in the vertical direction corresponding to the abscissas of 0, 1, ..., 63, and 64. _{Take the value on the coordinates; the value range of x i} in the sub-histogram 2 is [65,127], specifically 65, 66,...,126,127, and when statistical m is 65,66,...,126,127 these Take the value on the ordinate corresponding to the abscissa; _{the value range of x i} in the sub-histogram 3 is [128,189], specifically 128, 129,...,188,189, and when the statistical m is calculated, it is at 128,129,... , 188, 189 these abscissas correspond to the ordinates; _{the value range of x i} in the sub-histogram 3 is in [190,255], specifically 190, 191,...,254, 255, and when counting m, it is at 190, 191, ..., 254, 255 these abscissas correspond to the ordinates. It can be seen that the parameter value range when extracting features in this embodiment is a subset of the abscissa range of the corresponding sub-histogram.

When splicing, the overlapping part of two adjacent sub-histograms, only the mapping curve of the overlapping part of the left sub-histogram is taken, that is, the sub-histogram 1 takes the mapping curve corresponding to the abscissa [0,69], Take the mapping curve corresponding to the abscissa [69,130] in the sub-histogram 2, take the mapping curve corresponding to the abscissa [130,194] in the sub-histogram 3, and take the mapping curve corresponding to the abscissa [194,255] in the sub-histogram 3 , The four mapping curves are spliced end to end in sequence.

It should be noted that when the above-mentioned mapping curve is synthesized, the endpoint median connection method is used at the splicing of the mapping curves of two adjacent sub-histograms. In addition, the application can also adopt the endpoint maximum connection method and endpoint minimum value. The connection method, etc., is not specifically limited.

Step 305: Perform enhancement processing on the image to be processed according to the mapping curve of the histogram.

In the above step 302, before or after the histogram of the image to be processed is divided, the histogram is subjected to brightness transformation processing, such as gamma transformation, so the mapping curve of the histogram of the image to be processed obtained in step 304 is based on the brightness Calculated by transforming the processed histogram. It is necessary to inversely transform the mapping curve of the histogram of the image to be processed, such as inverse Gamma curve mapping, to obtain the final mapping curve. The final mapping curve is then applied to the pixels of the image to be processed, that is, the brightness of the pixels is mapped pixel by pixel according to the calculated mapping curve to obtain an enhanced image.

This application divides the histogram of the image to be processed into multiple sub-histograms, extracts features separately for each sub-histogram and obtains the gain curve coefficient and equalization curve weight, and then obtains the mapping curve of the sub-histogram according to the aforementioned parameters to realize the image to be processed Since the sub-histograms are processed separately, the brightness distribution and contrast distribution of the sub-histograms can be dynamically adjusted locally for different brightness intervals, which not only achieves the purpose of fine adjustment, but also adjusts the brightness distribution and contrast distribution to achieve The purpose of balance.

FIG. 6 is a schematic structural diagram of an embodiment of an image enhancement processing apparatus of this application. As shown in FIG. 6, the apparatus of this embodiment can be applied to the end device in the embodiment shown in FIGS. 1-3. The image enhancement processing device may include: an acquisition module 601 and a processing module 602, wherein the acquisition module 601 is used to obtain a histogram of the image to be processed; the processing module 602 is used to divide the histogram into n sub-histograms, n is a positive integer; calculate the mapping curve of each sub-histogram in the n sub-histograms; synthesize the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram; according to the mapping curve of the histogram Perform enhancement processing on the to-be-processed image.

In a possible implementation manner, the processing module 602 is specifically configured to extract multiple features of each sub-histogram; and determine the mapping curve of each sub-histogram according to the multiple features.

In a possible implementation manner, the processing module 602 is specifically configured to input the multiple features into the first histogram enhancement coefficient generator obtained by pre-training to obtain the gain curve coefficients, and the first histogram The histogram enhancement coefficient generator is used to calculate the gain curve coefficients; the multiple features are input into the second histogram enhancement coefficient generator obtained by pre-training to obtain the equalization curve weight, and the second histogram enhancement coefficient generator is used To calculate the weight of the equilibrium curve.

In a possible implementation manner, the processing module 602 is specifically configured to input the multiple features and the multiple features of the neighboring sub-histograms into the first histogram enhancement coefficient obtained by pre-training to generate The first histogram enhancement coefficient generator is used to calculate the gain curve coefficient; the multiple features and the multiple features of the adjacent sub-histograms are all input to the pre-trained The second histogram enhancement coefficient generator obtains the equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.

In a possible implementation manner, the processing module 602 is further configured to perform brightness conversion processing on the histogram; divide the histogram after the brightness conversion into n sub-histograms.

In a possible implementation manner, the processing module 602 is further configured to perform brightness transformation processing on each sub-histogram in the n sub-histograms.

In a possible implementation, the processing module 602 is specifically configured to splice the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram, and the mapping curves of the two adjacent sub-histograms The value at the connection of takes the average value, maximum value or minimum value, and the average value, maximum value or minimum value is the average of the two values of the mapping curves of the two adjacent sub-histograms at the connection ends of each other, Maximum or minimum value.

In a possible implementation, the processing module 602 is further configured to train the adaptive neuro-fuzzy system according to a first set parameter, and the first set parameter includes input feature membership function parameters, output Membership function parameter, rule function.

In a possible implementation manner, the processing module 602 is further configured to train the neural network according to a second set parameter, where the second set parameter includes the multiple features and membership function calculation nodes , The corresponding output value of different combination design in the rule function.

In a possible implementation manner, the processing module 602 is specifically configured to calculate the mapping curve of the first sub-histogram according to formula (1):

y _i =α _i ×y _{i_gain} +β _i ×y _{i_hist} (1)

The device in this embodiment can be used to implement the technical solution of the method embodiment shown in FIG. 3, and its implementation principles and technical effects are similar, and will not be repeated here.

In the implementation process, the steps of the foregoing method embodiments may be completed by hardware integrated logic circuits in the processor or instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other Programming logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware encoding processor, or executed and completed by a combination of hardware and software modules in the encoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

The memory mentioned in the above embodiments may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), and electrically available Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), and synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM) ) And direct memory bus random access memory (direct rambus RAM, DR RAM). It should be noted that the memories of the systems and methods described herein are intended to include, but are not limited to, these and any other suitable types of memories.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (personal computer, server, or network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

An image enhancement processing method, characterized in that it comprises:

Obtain the histogram of the image to be processed;

Divide the histogram into n sub-histograms, where n is a positive integer;

Calculating a mapping curve of each sub-histogram in the n sub-histograms;

Synthesize the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram;

The image to be processed is enhanced according to the mapping curve of the histogram.
The method according to claim 1, wherein the calculating the mapping curve of each sub-histogram in the n sub-histograms comprises:

Extract multiple features of each sub-histogram;

The mapping curve of each sub-histogram is determined according to the multiple features.
The method according to claim 2, wherein the multiple features include one or more of the brightness mean value of the sub-histogram, the brightness standard deviation of the sub-histogram, and the pixel percentage of the sub-histogram, and the sub-histogram The pixel percentage of the graph is the ratio of the number of pixels included in the sub-histogram to the number of pixels included in the histogram.
The method according to claim 2 or 3, wherein the determining the mapping curve of each sub-histogram according to the multiple characteristics comprises:

Calculating the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple characteristics;

The mapping curve of each sub-histogram is determined according to the gain curve coefficient and the equalization curve weight.
The method according to claim 4, wherein the calculating the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple characteristics comprises:

Inputting the plurality of features into a pre-trained first histogram enhancement coefficient generator to obtain the gain curve coefficient, and the first histogram enhancement coefficient generator is used to calculate the gain curve coefficient;

The second histogram enhancement coefficient generator obtained by pre-training is input into the plurality of features to obtain the equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.
The method according to claim 4, wherein the calculating the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple characteristics comprises:

The multiple features and the multiple features of the adjacent sub-histograms are all input to the first histogram enhancement coefficient generator obtained by pre-training to obtain the gain curve coefficients, and the first histogram enhancement coefficients are generated The device is used to calculate the coefficient of the gain curve;

The multiple features and the multiple features of the adjacent sub-histograms are all input to a second histogram enhancement coefficient generator obtained in advance to obtain the equalization curve weight, and the second histogram enhancement coefficient generator is used for Calculate the weight of the equilibrium curve.
The method according to claim 5 or 6, wherein the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator comprise an adaptive neuro-fuzzy system or a neural network.
The method according to any one of claims 1-7, wherein each sub-histogram in the n sub-histograms is a histogram processed by low-pass filtering.
The method according to any one of claims 1-8, wherein before dividing the histogram into n sub-histograms, the method further comprises:

Performing brightness conversion processing on the histogram;

The dividing the histogram into n sub-histograms includes:

The histogram after the brightness transformation is divided into n sub-histograms.
The method according to any one of claims 1-8, wherein after the dividing the histogram into n sub-histograms, the method further comprises:

Perform brightness conversion processing on each sub-histogram in the n sub-histograms.
The method according to any one of claims 1-10, wherein the synthesizing the mapping curves of the n sub-histograms to obtain the mapping curve of the histogram comprises:

The mapping curves of the n sub-histograms are spliced to obtain the mapping curve of the histogram, and the values at the junction of the mapping curves of two adjacent sub-histograms are averaged, maximum, or minimum, and the average The value, the maximum value or the minimum value are the average value, the maximum value or the minimum value of the two values of the mapping curves of the two adjacent sub-histograms at the connection ends of each other.
An image enhancement processing device, characterized in that it comprises:

The acquisition module is used to acquire the histogram of the image to be processed;

A processing module for dividing the histogram into n sub-histograms, where n is a positive integer; calculating the mapping curve of each sub-histogram in the n sub-histograms; synthesizing the mapping curves of the n sub-histograms Obtain the mapping curve of the histogram; perform enhancement processing on the image to be processed according to the mapping curve of the histogram.
The device according to claim 12, wherein the processing module is specifically configured to extract multiple features of each sub-histogram; and determine the mapping curve of each sub-histogram according to the multiple features.
The device according to claim 13, wherein the multiple features include one or more of the brightness mean value of the sub-histogram, the brightness standard deviation of the sub-histogram, and the pixel percentage of the sub-histogram, and the sub-histogram The pixel percentage of the graph is the ratio of the number of pixels included in the sub-histogram to the number of pixels included in the histogram.
The device according to claim 13 or 14, wherein the processing module is specifically configured to calculate the gain curve coefficient and the equalization curve weight of each sub-histogram according to the multiple characteristics; according to the gain curve coefficient and The weight of the equalization curve determines the mapping curve of each sub-histogram.
The apparatus according to claim 15, wherein the processing module is specifically configured to input the multiple features into a first histogram enhancement coefficient generator obtained by pre-training to obtain the gain curve coefficients, and The first histogram enhancement coefficient generator is used to calculate the gain curve coefficients; the multiple features are input into the second histogram enhancement coefficient generator obtained by pre-training to obtain the equalization curve weight, and the second histogram is enhanced The coefficient generator is used to calculate the weight of the equalization curve.
The device according to claim 15, wherein the processing module is specifically configured to input the multiple features and the multiple features of the neighboring sub-histograms into a first histogram obtained by pre-training The enhancement coefficient generator obtains the gain curve coefficient, and the first histogram enhancement coefficient generator is used to calculate the gain curve coefficient; inputting the multiple features and the multiple features of adjacent sub-histograms The pre-trained second histogram enhancement coefficient generator obtains the equalization curve weight, and the second histogram enhancement coefficient generator is used to calculate the equalization curve weight.
The device according to claim 16 or 17, wherein the first histogram enhancement coefficient generator and the second histogram enhancement coefficient generator comprise an adaptive neuro-fuzzy system or a neural network.
The device according to any one of claims 12-18, wherein each sub-histogram in the n sub-histograms is a histogram after low-pass filtering.
The device according to any one of claims 12-19, wherein the processing module is further configured to perform brightness conversion processing on the histogram; divide the histogram after the brightness conversion into n sub-histograms .
The device according to any one of claims 12-19, wherein the processing module is further configured to perform brightness transformation processing on each sub-histogram in the n sub-histograms.
The device according to any one of claims 12-21, wherein the processing module is specifically configured to splice the mapping curves of the n sub-histograms to obtain the mapping curves of the histograms, and The value at the connection of the mapping curves of two adjacent sub-histograms takes the average, maximum or minimum value, and the average, maximum or minimum is the connection between the mapping curves of the two adjacent sub-histograms. The average, maximum or minimum of the two values at the end.
An image enhancement processing device, characterized in that it comprises:

One or more processors;

Memory, used to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-11.
A computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are run on a computer, they are used to execute the method according to any one of claims 1-11.