WO2022133749A1

WO2022133749A1 - Image processing method and apparatus, storage medium and electronic device

Info

Publication number: WO2022133749A1
Application number: PCT/CN2020/138407
Authority: WO
Inventors: 罗俊
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-12-22
Filing date: 2020-12-22
Publication date: 2022-06-30
Also published as: CN116457822A

Abstract

An image processing method, an image processing apparatus, a storage medium and an electronic device. The image processing method comprises: acquiring, from among multiple continuous frames of images, an image to be processed (S310); performing luminance mapping processing on the image to be processed and generating an intermediate image (S320); acquiring at least one frame of reference image by using the multiple continuous frames of images (S330); and performing time sequence filtering processing on the reference image and the intermediate image so as to obtain an optimized image corresponding to the image to be processed (S340). By means of the method, luminance problems in images are alleviated, image noise reduction and the restoration of missing local information are achieved, and a high-quality image optimization result is obtained.

Description

Image processing method, device, storage medium and electronic device

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device.

Background technique

Image brightness refers to the brightness of an image, and is an important factor that affects people's visual experience when viewing an image. When the image brightness is inappropriate, such as too high or too low, the content of the image cannot be fully presented, for example, the face and text in the image are difficult to recognize, thus affecting the image quality.

SUMMARY OF THE INVENTION

The present disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium and an electronic device, so as to improve the brightness problem in an image at least to a certain extent.

According to a first aspect of the present disclosure, an image processing method is provided, comprising: acquiring an image to be processed from consecutive multi-frame images; performing luminance mapping processing on the to-be-processed image to generate an intermediate image; using the consecutive multi-frame images Acquire at least one frame of reference image; perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

According to a second aspect of the present disclosure, there is provided an image processing apparatus, including a processor; wherein the processor is configured to execute the following program modules stored in a memory: a to-be-processed image acquisition module, configured to acquire a to-be-processed image from multiple consecutive frames of images processing an image; an intermediate image generation module for performing brightness mapping processing on the to-be-processed image to generate an intermediate image; a reference image acquisition module for acquiring at least one frame of reference image by using the continuous multi-frame images; an image fusion processing module , which is used to perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

According to a third aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the image processing method of the first aspect and possible implementations thereof.

According to a fourth aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to The image processing method of the above-mentioned first aspect and possible implementations thereof are performed.

The technical solution of the present disclosure has the following beneficial effects:

On the one hand, brightness mapping processing of the image to be processed can improve the brightness problems in the image, such as low brightness, high brightness, uneven local brightness, etc. The fusion processing of the reference image and the intermediate image can achieve noise reduction, And repair the missing local information in the image, so as to obtain an optimized image with clearly visible image content. On the other hand, based on the continuous multiple frames of images collected during video shooting or image preview, this solution can realize the optimization of any one of the frame images without external information, and has lower implementation cost and higher practicability.

Description of drawings

Figure 1A shows a face image captured in a low-light environment;

Figure 1B shows a face image captured in a backlit environment;

FIG. 2 shows a schematic structural diagram of an electronic device in this exemplary embodiment;

FIG. 3 shows a flowchart of an image processing method in this exemplary embodiment;

FIG. 4 shows a flowchart of a method for acquiring an image to be processed in this exemplary embodiment;

FIG. 5 shows a schematic diagram of converting a RAW image into a single-channel image in this exemplary embodiment;

FIG. 6 shows a schematic diagram of a mapping curve in this exemplary embodiment;

FIG. 7 shows a flowchart of a luminance mapping processing method in this exemplary embodiment;

FIG. 8 shows a flowchart of a method for obtaining a reference image in this exemplary embodiment;

FIG. 9 shows an example diagram of image processing in this exemplary embodiment;

FIG. 10 shows an example diagram of image processing and face recognition in this exemplary embodiment;

FIG. 11 shows a schematic structural diagram of an image processing apparatus in this exemplary embodiment;

FIG. 12 shows a schematic diagram of an architecture in this exemplary embodiment.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments, however, can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other instances, well-known solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the figures are functional entities that do not necessarily necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The ambient light conditions and the exposure parameters of the shooting equipment when taking the image will affect the brightness of the image. Among them, the influence of ambient light conditions is greater. In relatively extreme environments such as weak light, backlight, and extremely strong light sources, the captured images may lack important part of the information. For example, as shown in FIG. 1A, when shooting a portrait, if the ambient light is very weak, that is, a weak light environment, the overall brightness of the captured image is very low, and it is difficult to recognize the face; as shown in FIG. 1B, if the light source is located at the Behind the person, that is, the backlight environment, the brightness of the face part is low, and it is also difficult to identify. Therefore, there is a need to improve the brightness of images captured in extreme environments.

To this end, the exemplary embodiments of the present disclosure first provide an image processing method, the application scenarios of which include but are not limited to: in the intelligent interaction scenario of a mobile terminal, collecting, monitoring and detecting through an AON (Always ON) camera Face and gesture information to achieve specific interactive functions, such as automatically activating the display when a face is detected, and automatically turning the user interface when a page turning gesture is detected; however, in weak lighting, backlight, extremely strong In an environment such as a light source, the brightness of the face and gesture images collected by the AON camera may be too low or too high, which affects the accuracy of the above detection; through the image processing method of this exemplary embodiment, the image brightness can be improved to increase the The accuracy of detection of faces, gestures, etc. In the scene of target tracking, it is necessary to collect images in real time to identify the target in the image; if it is in the above-mentioned extreme lighting environment, the brightness of the target part in the image is too low or too high, making it difficult to identify; The image processing method of the embodiment is improved to improve the recognition accuracy.

Exemplary embodiments of the present disclosure also provide an electronic device for executing the above-described image processing method. The electronic devices include, but are not limited to, computers, smart phones, tablet computers, game consoles, wearable devices, and the like. Typically, an electronic device includes a processor and a memory. The memory is used to store executable instructions of the processor, and may also store application data, such as image data, game data, etc.; the processor is configured to execute the image processing method in this exemplary embodiment by executing the executable instructions.

The following takes the mobile terminal 200 in FIG. 2 as an example to illustrate the structure of the above electronic device. It will be understood by those skilled in the art that the configuration in Figure 2 can also be applied to stationary type devices, in addition to components specifically for mobile purposes.

As shown in FIG. 2 , the mobile terminal 200 may specifically include: a processor 210 , an internal storage area 221 , an external memory interface 222 , a USB (Universal Serial Bus, Universal Serial Bus) interface 230 , a charging management module 240 , and a power management module 241 , battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, headphone jack 274, sensor module 280, display screen 290, camera module 291, Indicator 292, motor 293, key 294, SIM (Subscriber Identification Module, Subscriber Identification Module) card interface 295 and so on.

The processor 210 may include one or more processing units, for example, the processor 210 may include an AP (Application Processor, application processor), a modem processor, a GPU (Graphics Processing Unit, graphics processor), an ISP (Image Signal Processor, image signal processor), controller, encoder, decoder, DSP (Digital Signal Processor, digital signal processor), baseband processor and/or NPU (Neural-Network Processing Unit, neural network processor), etc.

In some embodiments, the processor 210 may include one or more interfaces through which connections are formed with other components of the mobile terminal 200 .

Internal storage area 221 may be used to store computer executable program code, which includes instructions. The internal storage area 221 may include volatile memory, such as DRAM (Dynamic Random Access Memory, dynamic random access memory), SRAM (Static Random Access Memory, static random access memory), and may also include non-volatile memory, such as at least one disk memory devices, flash memory devices, UFS (Universal Flash Storage, universal flash memory), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by executing instructions stored in the internal storage area 221 and/or instructions stored in a memory provided in the processor.

The external memory interface 222 can be used to connect an external memory, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory communicates with the processor 210 through the external memory interface 222 to implement data storage functions, such as storing music, video and other files.

The USB interface 230 is an interface conforming to the USB standard specification, and can be used to connect a charger to charge the mobile terminal 200, and can also be connected to an earphone or other electronic devices.

The charging management module 240 is used to receive charging input from the charger. While charging the battery 242, the charging management module 240 can also supply power to the device through the power management module 241; the power management module 241 can also monitor the state of the battery.

The wireless communication function of the mobile terminal 200 may be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modulation and demodulation processor, the baseband processor, and the like. Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 250 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the mobile terminal 200 . The wireless communication module 260 can provide applications on the mobile terminal 200 including WLAN (Wireless Local Area Networks, wireless local area network) (such as Wi-Fi (Wireless Fidelity, wireless fidelity) network), BT (Bluetooth, Bluetooth), GNSS (Global Navigation Satellite System, global navigation satellite system), FM (Frequency Modulation, frequency modulation), NFC (Near Field Communication, short-range wireless communication technology), IR (Infrared, infrared technology) and other wireless communication solutions.

The mobile terminal 200 may implement a display function through a GPU, a display screen 290, an AP, and the like.

The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the encoder, the decoder, the GPU, the display screen 290, the AP, and the like. The camera module 291 can include various types of cameras, such as AON cameras, wide-angle cameras, high-definition cameras, etc. The camera can be arranged at any position of the mobile terminal 200, for example, arranged on the side of the display screen 290 to form a front camera, Or set on the opposite side of the display screen 290 to form a rear camera.

The mobile terminal 200 can implement audio functions through an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, an AP, and the like.

The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyro sensor 2803, an air pressure sensor 2804, etc., to realize different sensing detection functions.

The indicator 292 can be an indicator light, which can be used to indicate the charging status, the change of power, and can also be used to indicate messages, missed calls, notifications, and the like. The motor 293 can generate vibration prompts, and can also be used for touch vibration feedback and the like. The keys 294 include a power-on key, a volume key, and the like.

The mobile terminal 200 may support one or more SIM card interfaces 295 for connecting the SIM cards to realize functions such as calling and data communication.

FIG. 3 shows a schematic flow of the image processing method in this exemplary embodiment, which may include:

Step S310, acquiring an image to be processed from multiple consecutive frames of images;

Step S320, performing brightness mapping processing on the image to be processed to generate an intermediate image;

Step S330, obtaining at least one frame of reference image by using the above-mentioned consecutive multi-frame images;

Step S340, performing fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

Through the above method, on the one hand, the brightness mapping processing of the image to be processed can improve the brightness problems in the image, such as low brightness, high brightness, uneven local brightness, etc. It realizes noise reduction and repairs the lack of local information in the image, so as to obtain an optimized image with clearly visible image content, which is conducive to the further realization of face recognition, gesture recognition, target detection and other applications; further, this solution improves image capture and processing. Robustness in extreme lighting environments reduces the dependence on the performance of hardware such as cameras and image sensors. For example, for image sensors with low photosensitive performance, this solution can be used to optimize the captured images to obtain high-quality images. This helps to reduce hardware costs. On the other hand, based on the continuous multi-frame images collected during video shooting or image preview, this solution can realize the optimization of any one of the frame images without external information, and has low implementation cost and high practicability.

Each step in FIG. 3 will be specifically described below.

In step S310, the to-be-processed image is acquired from consecutive multiple frames of images.

The above-mentioned continuous multi-frame images may be images continuously collected by a camera, for example, a camera shoots a video or continuously collects a preview image, and the like. The image to be processed can be any one of the frame images. Generally, when the camera captures an image, the method in FIG. 3 is executed in real time by taking the currently collected frame of image as the image to be processed, so as to realize the processing of each frame of image. .

In an optional implementation manner, referring to FIG. 4 , step S310 may include

Step S410, obtaining the current RAW image from consecutive multiple frames of RAW images;

Step S420, performing channel conversion processing on the current RAW image to obtain an image to be processed.

The RAW image refers to an image stored in a RAW format, and generally refers to an original image collected by an image sensor in a camera. The image sensor collects the light signal through a Bayer filter and converts it into a digital signal to obtain a RAW image. Referring to the left image in Figure 5, each pixel in the RAW image has only one color in RGB and is arranged in a Bayer array. In an optional implementation manner, an AON camera may be set up on the terminal device, and the above-mentioned continuous multiple frames of RAW images may be collected and obtained

In this exemplary embodiment, a currently collected RAW image is acquired, and since the color channels of each pixel point are different, channel conversion processing is performed on the current RAW image to obtain an image to be processed. Channel conversion processing is to unify the color channels of each pixel, including but not limited to the following two methods:

1. Convert each pixel of the current RAW image to any channel in RGB, for example, uniformly convert it to the R channel. Generally, since the human eye is more sensitive to green, as shown in FIG. 5 , the R channel and B channel of the current RAW image can be converted into the G channel to obtain an image with all pixels in the G channel. Specifically, the pixels of the R channel and the B channel can be mapped to convert to the pixel value of the G channel, for example:

P _R *a ₁ =P _G , P _B *a ₂ =P _G (1)

Among them, P _R , P _G , and P _B represent the pixel values of the R channel, the G channel, and the B channel, respectively; a ₁ and a ₂ are the conversion coefficients from the R channel and the B channel to the G channel, which can be empirical coefficients, such as a ₁ = 0.299/0.587 = 0.509 (0.299 and 0.587 are the coefficients when the R channel and G channel are grayscale conversion, respectively), a ₂ =0.114/0.587=0.194 (0.114 is the coefficient when the B channel is grayscale conversion). It can be seen that when converting the R and B channels to the G channel, the consistency of brightness is considered, that is, the brightness level of the same pixel before and after the conversion is basically the same.

2. Convert the current RAW image into a grayscale image. For example, the R, G, and B channels can be converted into grayscale values according to a certain coefficient ratio, or the RAW image can be processed by Demosaic (de-mosaic) to obtain an RGB image, and then The gray value (Gray) of each pixel is calculated by the following formula (2) to obtain a gray image.

Gray＝P _R *0.299+P _G *0.587+P _B *0.114 (2)

In the current RAW image, each pixel needs 10 bits, of which 8 bits record the pixel value and 2 bits record the channel information. A single-channel image is obtained through channel conversion processing (a grayscale image can be regarded as a special single-channel image), and there is no need to record channel information, so each pixel only needs 8 bits. Compared with RGB images, single-channel images have greatly reduced data volume and save the bit width for recording channel information, thereby reducing the data volume of subsequent processing, which is conducive to real-time optimization processing. Moreover, the single-channel image can characterize the brightness of the image to be processed, and the information is relatively sufficient.

Continuing to refer to FIG. 3 , in step S320, luminance mapping processing is performed on the image to be processed to generate an intermediate image.

The brightness mapping process refers to mapping the brightness value of each pixel in the image to be processed into a new value. Among them, the brightness values of all pixels can be adjusted in the same direction, such as uniform increase or uniform decrease, the brightness change values of different pixels can be the same or different; the brightness values of different pixels can also be adjusted in different directions , such as reducing the brightness of the too bright part, increasing the brightness of the too dark part, adjusting the gray scale of the image, etc.

In this exemplary embodiment, a mapping relationship between luminance values before and after mapping can be pre-configured, and the mapping relationship can be related to the luminance level or luminance distribution of the image to be processed itself, and then the luminance mapping process is implemented according to the mapping relationship. Intermediate image.

In an optional implementation manner, step S320 may include:

Perform luminance mapping processing on the image to be processed according to the illumination information of the image to be processed.

Wherein, the illumination information is information representing the ambient illumination condition of the image to be processed, for example, it may include the ratio of illuminance value to backlight. The illuminance value is a measure of the amount of light received per unit area of the captured image. The backlight ratio is the proportion of the backlight part of the image to be processed. The illumination information can reflect whether the image to be processed has defects in illumination, such as the overall illumination is excessive, the illumination is insufficient, and the local illumination distribution is unbalanced, etc., and further, brightness mapping processing can be performed on the image to be processed.

Exemplarily, the luminance value of the image to be processed may be determined according to the exposure parameter of the image to be processed. The exposure parameter includes at least one of exposure time (Exptime), sensitivity (ISO), and aperture value (F value). Generally, exposure parameters are recorded when the image to be processed is captured, and for cameras on smartphones, the aperture value is usually a fixed value. The exposure parameter is used to estimate the illuminance value. The more comprehensive the acquired exposure parameter, the more accurate the estimated illuminance value. For example, you can refer to the following formula (3):

Lumi=a ₃ *F ² /(Exptime*ISO) (3)

Lumi represents the illuminance value, the unit is lux; a ₃ represents the empirical coefficient, such as 12.4, which can be adjusted according to the actual shooting scene or camera performance; F is the aperture value, which is the ratio of the focal length of the lens to the effective diameter of the lens; Exptime is the exposure time, The unit is seconds; ISO is the sensitivity.

The illuminance value of the image to be processed reflects the global illumination of the image to be processed. If the illuminance value is too low, it indicates that the entire image to be processed is underexposed. When the illuminance value is too high, it indicates that the entire image to be processed is overexposed. Therefore, in an optional implementation manner, step S320 may include:

When the luminance value of the image to be processed satisfies the first preset condition, the image to be processed is subjected to global luminance mapping processing.

The first preset condition may include: less than the first threshold, or greater than the second threshold. The first threshold is a threshold for measuring underexposure, and the second threshold is a threshold for measuring overexposure. The two thresholds can be determined based on experience, or can be adjusted according to the actual scene. For example, the first threshold and the second threshold are appropriately increased during the day. , and appropriately reduce the first threshold and the second threshold in the dark. When the illuminance value of the image to be processed satisfies the first preset condition, there is a situation of under-exposure or over-exposure, so it is subjected to a global luminance mapping process (Global Luminance Mapping). Specifically, the following steps can be included:

When the luminance value of the to-be-processed image is less than the first threshold, the to-be-processed image is subjected to a global brightness upward mapping process, that is, global brightening.

When the luminance value of the to-be-processed image is greater than the second threshold, the to-be-processed image is subjected to global brightness down mapping processing, that is, global brightness reduction.

Taking global brightening as an example, the brightness values of different pixel points in the image to be processed can be mapped to higher brightness values according to a preset mapping curve. The mapping curve can be a linear curve, which reflects a linear mapping relationship, or a quadratic curve or other nonlinear curve, which reflects a nonlinear mapping relationship. It should be noted that, in this exemplary embodiment, a plurality of mapping curves may be configured, which correspond to different mapping intensities respectively, and the higher the mapping intensities, the more significant the brightening will be. During specific processing, the mapping intensity and which mapping curve to use can be determined according to the illuminance value of the image to be processed or the actual scene requirements. For example, the lower the illuminance value, the higher the mapping intensity, and the larger the overall slope of the mapping curve.

In an optional implementation manner, step S320 may further include:

When the illuminance value of the image to be processed does not meet the first preset condition, determining the backlight ratio of the image to be processed;

When the backlight ratio of the to-be-processed image satisfies the second preset condition, tone-mapping processing is performed on the to-be-processed image.

Wherein, the illumination value of the image to be processed does not meet the first preset condition, indicating that the global illumination of the image to be processed is relatively suitable, and then the backlight ratio of the image to be processed is calculated to determine whether there is a local inappropriate situation.

In an optional implementation manner, the backlight ratio of the to-be-processed image may be determined according to the brightness histogram of the to-be-processed image, which may specifically include: setting a plurality of brightness levels, counting the pixel ratios in each brightness level, and forming a to-be-processed image The brightness histogram of the image; if there are at least two brightness levels, the brightness difference value reaches the set value (the set value can be an empirical value, or can be determined according to the specific scene, such as when there is a prominent light source in the scene, the set value Generally larger, the setting is generally smaller under natural light or no significant light source), then the lower brightness level is determined as the backlight part; the brightness histogram of all the backlight parts is counted to obtain the proportion of the backlight part in the image to be processed, that is Backlight ratio.

In an optional embodiment, the highest brightness value in the image to be processed can also be obtained, and the brightness threshold value can be determined according to the highest brightness value. For example, the highest brightness value can be multiplied by a fixed coefficient less than 1 to obtain the brightness threshold value; The pixels below the brightness threshold are extracted, and the connected regions are extracted, that is, the isolated pixels are filtered out; the proportion of the connected regions in the image to be processed is taken as the backlight ratio.

The second preset condition may include: the backlight ratio is greater than the third threshold. It should be noted that the above calculation of the backlight ratio is to estimate the possible backlight phenomenon in the image to be processed. When the backlight ratio is greater than the third threshold, it can be considered that the probability of the existence of the backlight phenomenon is high. The third threshold is a threshold for measuring whether there is a backlight phenomenon, which can be determined according to experience or actual scenarios. In this case, tone mapping processing (Tone Mapping) is performed on the image to be processed. The tone mapping process essentially still maps the brightness. Different from the above-mentioned global brightness mapping process, the tone mapping process can change the brightness range or brightness level distribution of the image, and the brightness adjustment direction of each pixel can be different. For example, in the global brightness mapping process, for all pixels in the entire image, the brightness is adjusted in the same direction, such as increasing the brightness as a whole or decreasing the brightness as a whole, to adjust the overall brightness level; in the tone mapping process, For different pixels in the whole image, the brightness can be adjusted in different directions. For example, the brighter parts are mapped downwards, and the parts with lower brightness are mapped upwards to adjust the brightness distribution.

In this exemplary embodiment, the tone mapping process can be implemented by a mapping curve. Figure 6 shows the mapping curves used in tone mapping processing, wherein the abscissa is the luminance value before mapping, the ordinate is the luminance value after mapping, the curves A, B, and C are the mapping curves under different mapping intensities, and the curve C is The mapping strength is the highest, and curve A has the lowest mapping strength. It can be seen that after the brightness value is mapped by the mapping curve, the brightness distribution of the image to be processed can be mapped to a smaller range. Caused by the local invisible problem. Generally, the higher the backlight ratio, or the greater the difference between the backlight part and the high-brightness part, the higher the mapping intensity is. It can be seen that the mapping curve used in the tone mapping processing is different from the mapping curve used in the above-mentioned global brightness mapping processing: the former is generally a non-linear curve, and a section of the slope is greater than 45 degrees (that is, the part of the brightness upward mapping processing, generally the brightness is relatively high. The lower section), the slope of the other section is less than 45 degrees (that is, the part where the brightness is mapped downward, generally the section with higher brightness); the latter can be a linear curve or a non-linear curve, and the slope of the entire curve is greater than 45 degrees or less.

In an optional embodiment, when the backlight ratio of the image to be processed does not meet the second preset condition, it means that the global illumination and local illumination of the image to be processed are relatively good, and the brightness mapping process may not be performed, and the The image to be processed serves as an intermediate image.

FIG. 7 shows a schematic flow of luminance mapping processing, including:

Step S710, determining the luminance value Lumi of the image to be processed;

Step S720, compare the illuminance value with the first threshold value T1 and the second threshold value T2; when Lumi<T1 or Lumi>T2, it is determined that the first preset condition is met, and step S730 is performed; otherwise, step S740 is performed;

Step S730, performing global brightness mapping processing on the image to be processed;

Step S740, determine the backlight ratio (BL ratio) of the image to be processed;

Step S750, compare the backlight ratio with the third threshold value T3; when BL ratio>T3, it is determined that the second preset condition is met, and step S760 is performed; otherwise, the image to be processed is not processed, and jumps to step S770;

Step S760, performing tone mapping processing on the image to be processed;

In step S770, an intermediate image is obtained.

Through the above methods, the brightness of the image to be processed is improved from the global and local levels, and in the obtained intermediate image, the image information missing due to the brightness problem can be recovered to a certain extent.

Continuing to refer to FIG. 3 , in step S330 , at least one frame of reference image is acquired by using the above-mentioned consecutive multiple frames of images.

Wherein, the reference image is one or more frames of images that are continuous in time with the image to be processed, which can form supplementary image information of the image to be processed.

In an optional implementation, referring to FIG. 8 , step S330 may include:

Step S810, acquiring at least one frame of images other than the image to be processed among the above-mentioned consecutive multiple frames of images;

Step S820, performing luminance mapping processing on the at least one frame of image to obtain a reference image;

Step S830, acquiring an optimized image corresponding to the at least one frame of image as a reference image.

Among them, any one or more frames of images other than the image to be processed can be selected from the above-mentioned continuous multiple frames of images. Generally, the more the number of selected images, the closer the time to the image to be processed, the more conducive to improving the optimization effect. . For example, if the image to be processed is the i-th frame image in the above-mentioned consecutive multi-frame images, and i is a positive integer not less than 2, then the i-m-th frame image to the i-1-th frame image can be selected, that is, the image located in the image to be processed The previous m frames of images are used for subsequent optimization, where m is any positive integer. The value of m can be determined by combining experience, actual demand and computing power performance.

In step S820 and step S830, one of them may be selectively executed. In step S820, for the luminance mapping processing of the above-mentioned at least one frame of image, reference may be made to the luminance mapping processing performed on the image to be processed in step S320 and FIG. 7 . In step S830, the optimized image corresponding to the above-mentioned at least one frame of image may be an image obtained by optimizing the image by using the method flow of FIG. The method process is optimized to obtain an optimized image corresponding to each frame of image, and the optimized image can be used as a reference image for optimizing the next frame of image.

In an optional implementation manner, the exposure parameter of the image to be processed may be different from the exposure parameter of the above-mentioned at least one frame of image, for example, any one of exposure time, sensitivity, and aperture value may be different. Thus, the image to be processed or the intermediate image can be complementary to the above-mentioned at least one frame of image or reference image to form information in terms of exposure and brightness. Further, the device can control the camera to collect the above-mentioned continuous multiple frames of images with different exposure parameters. For example, when the image is collected, the exposure time is gradually increased, or the sensitivity is gradually increased, and the exposure parameters between different frame images are different, so that the continuous The information in the multi-frame images is maximized to form a more effective information complementation.

Continuing to refer to FIG. 3 , in step S340, a fusion process is performed on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.

The reference image and the intermediate image have a slight difference in time, so there is a difference in image information, and the fusion of the two images can form a repairing effect on the image information of the detail part. For example, the image frequencies in the two images can be scanned separately and compared, and an area in the reference image whose image frequency is higher than that in the intermediate image is selected, and the area in the reference image is fused with the intermediate image.

During fusion, weights can be performed on the pixels at the same position, for example, the weights are determined according to the image frequencies of the pixels in the two images, and weighted fusion is performed.

In an optional implementation manner, step S340 may include:

Time series filtering is performed on the reference image and the intermediate image.

Specifically, according to the time stamps of the reference image and the intermediate image, they can be arranged into an image sequence, and then the image information in the image sequence can be converted into time-series signals, and the time-series signals can be filtered, such as Gaussian filtering, mean filtering, etc. Various ways to achieve further optimization effects such as noise reduction.

In an optional embodiment, after the reference image and the intermediate image are fused, the brightness of the fused image may also be adjusted according to the above-mentioned continuous multiple frames to obtain an optimized image corresponding to the image to be processed. Since the intermediate image has undergone brightness mapping processing, its brightness may be significantly different from other frame images, resulting in brightness jumps of the video image. Therefore, the brightness of the fused image can be adjusted according to the brightness of other frame images, for example, selecting the same area in the other frame images and the fused image respectively, based on the brightness difference of the area in the two images, to The overall brightness adjustment is performed on the image after fusion processing, thereby ensuring the brightness consistency between consecutive frame images. It should be noted that, unlike the brightness mapping processing in step S320, the brightness adjustment performed here is generally brightness fine-tuning or brightness smoothing processing.

FIG. 9 shows an example diagram of the effect after the image to be processed is optimized, wherein the upper row is the to-be-processed image, and the lower row is the corresponding optimized image. It can be seen that the brightness of faces or gestures in the image to be processed is low, which makes it impossible to see clearly. After optimization, clearer face or gesture information can be obtained.

In an optional implementation manner, after the optimized image corresponding to the image to be processed is obtained, at least one of target detection, face recognition, and gesture recognition may be performed on the optimized image. Thereby, more accurate detection or identification results can be obtained. Figure 10 shows an example of face recognition on the optimized image. It can be seen that after optimization, the face part is clearly visible, so that the area where the face is located and the facial feature points are accurately detected.

Using the image processing method of this exemplary embodiment, a data set containing 845 images of faces and gestures is tested for optimization processing, most of the images are illuminance values of 3 to 15 lux, and the distance between the face or gesture and the camera is 15 ~ 40cm, belonging to low light environment and backlight environment.

In the case that the above data set is not optimized, it is tested for face and gesture recognition, the accuracy rate is 95.2%, the recall rate is 16.7%, and the F1 value (F1 value is a kind of algorithm model in the field of machine learning). Evaluation index, the calculation method is

) is 28%;

In the case of the above-mentioned data set being optimized, it is tested for face and gesture recognition, the accuracy rate is 99.1%, the recall rate is 82.3%, and the F1 value is 90%.

It can be seen that the image processing method of the present exemplary embodiment has a very significant improvement effect on the recognition algorithm in a low-light environment and a backlight environment. Therefore, even in extreme lighting environments, results similar to those in normal lighting environments can be obtained, which increases the robustness of the recognition algorithm and broadens its application scenarios.

Exemplary embodiments of the present disclosure also provide an image processing apparatus. Referring to FIG. 11 , the image processing apparatus 1100 may include a processor 1110 and a memory 1120 . Wherein, the memory 1120 stores the following program modules:

The to-be-processed image acquisition module 1121 is used to acquire the to-be-processed image from consecutive multiple frames of images;

An intermediate image generation module 1122, configured to perform luminance mapping processing on the image to be processed to generate an intermediate image;

A reference image acquisition module 1123, configured to acquire at least one frame of reference image by using the above-mentioned consecutive multiple frames of images;

The image fusion processing module 1124 is used to perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed;

The processor 1110 may be used to execute the above-described program modules.

In an optional implementation manner, the intermediate image generation module 1122 is configured to:

In an optional implementation manner, the illumination information includes an illumination value; the intermediate image generation module 1122 is configured to:

The illuminance value of the image to be processed is determined according to the exposure parameter of the image to be processed; the exposure parameter includes at least one of exposure time, sensitivity, and aperture value.

In an optional implementation manner, the illumination information further includes a backlight ratio; the intermediate image generation module 1122 is configured to:

The backlight ratio of the to-be-processed image is determined according to the brightness histogram of the to-be-processed image.

When the backlight ratio of the image to be processed does not satisfy the second preset condition, the image to be processed is used as an intermediate image.

In an optional implementation manner, the reference image acquisition module 1123 is configured to:

Obtaining at least one frame of images other than the image to be processed in consecutive multiple frames of images;

Perform luminance mapping processing on the above-mentioned at least one frame of image to obtain a reference image; or

An optimized image corresponding to the at least one frame of image is obtained as a reference image.

In an optional implementation manner, the exposure parameter of the image to be processed is different from the exposure parameter of the above-mentioned at least one frame of image.

In an optional implementation manner, the image to be processed is the i-th frame of images in consecutive multiple frames of images; the above-mentioned at least one frame of image includes the i-m-th frame of images to the i-1-th frame of images in the consecutive multi-frame images; i is a positive integer not less than 2, and m is any positive integer.

In an optional implementation manner, the image fusion processing module 1124 is configured to:

After the reference image and the intermediate image are fused, the brightness of the fused image is adjusted according to the above-mentioned continuous multi-frame images, so as to obtain an optimized image corresponding to the image to be processed.

In an optional implementation manner, the to-be-processed image acquisition module 1121 is configured to:

Obtain the current RAW image from consecutive multi-frame RAW images;

Perform channel conversion processing on the current RAW image to obtain an image to be processed.

Convert the R and B channels of the current RAW image to the G channel; or

Convert the current RAW image to a grayscale image.

In an optional implementation manner, the image processing apparatus 1100 is configured in a terminal device, and the terminal device includes an AON camera, which is used to collect the above-mentioned continuous multiple frames of RAW images.

In an optional implementation manner, the memory 1120 further includes the following program modules:

The image recognition application module is used to perform at least one process of target detection, face recognition, and gesture recognition on the above-mentioned optimized image.

The specific details of each part in the above-mentioned apparatus 1100 have been described in detail in the implementation of the method part, and for undisclosed details, please refer to the implementation content of the method part, and thus will not be repeated.

FIG. 12 shows a schematic diagram of the architecture of this exemplary embodiment. As shown in FIG. 12 , the electronic device is configured with an AON camera, which runs the AON camera service, and can realize the underlying processing of the image through the image signal processor. After processing, the corresponding optimized image is obtained, and the optimized image is provided to the AON software service. The AON software service can perform monitoring, recognition and other services through the digital signal processor, such as face recognition and gesture recognition on the optimized image, obtain the corresponding recognition results, and provide the recognition results to the application service. The application service can run related applications through the main processor, and use the above face and gesture recognition results to implement specific interactive instructions, such as screen locking and unlocking, and user interface page turning.

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium that can be implemented in the form of a program product including program code for enabling the electronic device when the program product is run on the electronic device The steps described in the "Example Methods" section above in this specification in accordance with various exemplary embodiments of the present disclosure are performed. In an alternative embodiment, the program product may be implemented as a portable compact disk read only memory (CD-ROM) and include program code, and may be executed on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

A computer readable signal medium may include a propagated data signal in baseband or as part of a carrier wave with readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium can also be any readable medium other than a readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural programming Language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (eg, using an Internet service provider business via an Internet connection).

Those skilled in the art can easily understand from the description of the above embodiments that the exemplary embodiments described herein may be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to cause a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiment of the present disclosure.

Furthermore, the above-mentioned figures are merely schematic illustrations of the processes included in the methods according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures do not indicate or limit the chronological order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, in multiple modules.

It should be noted that although several modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

As will be appreciated by one skilled in the art, various aspects of the present disclosure may be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be embodied in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which may be collectively referred to herein as implementations "circuit", "module" or "system". Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and embodiments are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

An image processing method, comprising:

Obtain the image to be processed from consecutive multi-frame images;

Perform luminance mapping processing on the to-be-processed image to generate an intermediate image;

Obtain at least one frame of reference image by using the consecutive multiple frames of images;

Fusion processing is performed on the reference image and the intermediate image to obtain an optimized image corresponding to the to-be-processed image.
The method according to claim 1, wherein the performing luminance mapping processing on the to-be-processed image comprises:

Perform luminance mapping processing on the to-be-processed image according to the illumination information of the to-be-processed image.
The method according to claim 2, wherein the illumination information includes an illuminance value; and performing luminance mapping processing on the to-be-processed image according to the illumination information of the to-be-processed image comprises:

When the illuminance value of the to-be-processed image satisfies the first preset condition, global luminance mapping processing is performed on the to-be-processed image.
The method according to claim 3, wherein the method further comprises:

Determine the illuminance value of the to-be-processed image according to the exposure parameter of the to-be-processed image;

The exposure parameters include at least one of exposure time, sensitivity, and aperture value.
The method according to claim 3, wherein the illumination information further includes a backlight ratio; the performing luminance mapping processing on the to-be-processed image according to the illumination information of the to-be-processed image further comprises:

When the illuminance value of the to-be-processed image does not satisfy the first preset condition, determining the backlight ratio of the to-be-processed image;

When the backlight ratio of the to-be-processed image satisfies the second preset condition, the tone-mapping process is performed on the to-be-processed image.
The method according to claim 5, wherein the determining the backlight ratio of the to-be-processed image comprises:

The backlight ratio of the to-be-processed image is determined according to the brightness histogram of the to-be-processed image.
The method according to claim 5, wherein the method further comprises:

When the backlight ratio of the to-be-processed image does not satisfy the second preset condition, the to-be-processed image is used as the intermediate image.
The method according to claim 1, wherein the obtaining at least one frame of reference image by using the consecutive multiple frames of images comprises:

Obtain at least one frame of images other than the to-be-processed image from the consecutive multiple frames of images;

Perform luminance mapping processing on the at least one frame of image to obtain the reference image; or

An optimized image corresponding to the at least one frame of image is acquired as the reference image.
The method according to claim 8, wherein the exposure parameter of the image to be processed is different from the exposure parameter of the at least one frame of image.
The method according to claim 8, wherein the to-be-processed image is the i-th frame of images in the consecutive multi-frame images; the at least one frame of image comprises the i-m-th frame in the consecutive multi-frame images Image to the i-1th frame image; i is a positive integer not less than 2, and m is any positive integer.
The method according to claim 1, wherein the performing fusion processing on the reference image and the intermediate image comprises:

A time series filtering process is performed on the reference image and the intermediate image.
The method according to claim 1, wherein after the reference image and the intermediate image are fused, the method further comprises:

According to the continuous multiple frames of images, the brightness of the fused images is adjusted to obtain an optimized image corresponding to the to-be-processed image.
The method according to claim 1, wherein the acquiring the image to be processed from consecutive multiple frames of images comprises:

Obtain the current RAW image from consecutive multi-frame RAW images;

Perform channel conversion processing on the current RAW image to obtain the to-be-processed image.
The method according to claim 13, wherein the performing channel conversion processing on the current RAW image comprises:

Convert the R channel and B channel of the current RAW image to the G channel; or

Convert the current RAW image to a grayscale image.
The method according to claim 13, wherein the method is applied to a terminal device, and the terminal device comprises a normally-on camera, which is used to collect the continuous multiple frames of RAW images.
The method according to claim 1, wherein after obtaining the optimized image corresponding to the to-be-processed image, the method further comprises:

At least one of target detection, face recognition, and gesture recognition is performed on the optimized image.
An image processing device, comprising a processor;

Wherein, the processor is used to execute the following program modules stored in the memory:

a to-be-processed image acquisition module, used to acquire the to-be-processed image from consecutive multiple frames of images;

an intermediate image generation module, configured to perform luminance mapping processing on the to-be-processed image to generate an intermediate image;

a reference image acquisition module, configured to acquire at least one frame of reference image by using the consecutive multi-frame images;

An image fusion processing module, configured to perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the to-be-processed image.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the method of any one of claims 1 to 16 is implemented.
An electronic device, comprising:

processor; and

a memory for storing executable instructions for the processor;

wherein the processor is configured to perform the method of any one of claims 1 to 16 by executing the executable instructions.