WO2022133749A1 - Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique - Google Patents

Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique Download PDF

Info

Publication number
WO2022133749A1
WO2022133749A1 PCT/CN2020/138407 CN2020138407W WO2022133749A1 WO 2022133749 A1 WO2022133749 A1 WO 2022133749A1 CN 2020138407 W CN2020138407 W CN 2020138407W WO 2022133749 A1 WO2022133749 A1 WO 2022133749A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processed
images
frame
processed image
Prior art date
Application number
PCT/CN2020/138407
Other languages
English (en)
Chinese (zh)
Inventor
罗俊
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to PCT/CN2020/138407 priority Critical patent/WO2022133749A1/fr
Priority to CN202080107017.0A priority patent/CN116457822A/zh
Publication of WO2022133749A1 publication Critical patent/WO2022133749A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction

Definitions

  • the present disclosure relates to the technical field of image processing, and in particular, to an image processing method, an image processing apparatus, a computer-readable storage medium, and an electronic device.
  • Image brightness refers to the brightness of an image, and is an important factor that affects people's visual experience when viewing an image.
  • image brightness is inappropriate, such as too high or too low, the content of the image cannot be fully presented, for example, the face and text in the image are difficult to recognize, thus affecting the image quality.
  • the present disclosure provides an image processing method, an image processing apparatus, a computer-readable storage medium and an electronic device, so as to improve the brightness problem in an image at least to a certain extent.
  • an image processing method comprising: acquiring an image to be processed from consecutive multi-frame images; performing luminance mapping processing on the to-be-processed image to generate an intermediate image; using the consecutive multi-frame images Acquire at least one frame of reference image; perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.
  • an image processing apparatus including a processor; wherein the processor is configured to execute the following program modules stored in a memory: a to-be-processed image acquisition module, configured to acquire a to-be-processed image from multiple consecutive frames of images processing an image; an intermediate image generation module for performing brightness mapping processing on the to-be-processed image to generate an intermediate image; a reference image acquisition module for acquiring at least one frame of reference image by using the continuous multi-frame images; an image fusion processing module , which is used to perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.
  • a to-be-processed image acquisition module configured to acquire a to-be-processed image from multiple consecutive frames of images processing an image
  • an intermediate image generation module for performing brightness mapping processing on the to-be-processed image to generate an intermediate image
  • a reference image acquisition module for acquiring at least one frame of reference image by using the continuous multi-frame images
  • an image fusion processing module which is used to perform
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the image processing method of the first aspect and possible implementations thereof.
  • an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to The image processing method of the above-mentioned first aspect and possible implementations thereof are performed.
  • brightness mapping processing of the image to be processed can improve the brightness problems in the image, such as low brightness, high brightness, uneven local brightness, etc.
  • the fusion processing of the reference image and the intermediate image can achieve noise reduction, And repair the missing local information in the image, so as to obtain an optimized image with clearly visible image content.
  • this solution can realize the optimization of any one of the frame images without external information, and has lower implementation cost and higher practicability.
  • Figure 1A shows a face image captured in a low-light environment
  • Figure 1B shows a face image captured in a backlit environment
  • FIG. 2 shows a schematic structural diagram of an electronic device in this exemplary embodiment
  • FIG. 3 shows a flowchart of an image processing method in this exemplary embodiment
  • FIG. 4 shows a flowchart of a method for acquiring an image to be processed in this exemplary embodiment
  • FIG. 5 shows a schematic diagram of converting a RAW image into a single-channel image in this exemplary embodiment
  • FIG. 6 shows a schematic diagram of a mapping curve in this exemplary embodiment
  • FIG. 7 shows a flowchart of a luminance mapping processing method in this exemplary embodiment
  • FIG. 8 shows a flowchart of a method for obtaining a reference image in this exemplary embodiment
  • FIG. 9 shows an example diagram of image processing in this exemplary embodiment
  • FIG. 10 shows an example diagram of image processing and face recognition in this exemplary embodiment
  • FIG. 11 shows a schematic structural diagram of an image processing apparatus in this exemplary embodiment
  • FIG. 12 shows a schematic diagram of an architecture in this exemplary embodiment.
  • Example embodiments will now be described more fully with reference to the accompanying drawings.
  • Example embodiments can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
  • the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
  • numerous specific details are provided in order to give a thorough understanding of the embodiments of the present disclosure.
  • those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed.
  • well-known solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
  • the ambient light conditions and the exposure parameters of the shooting equipment when taking the image will affect the brightness of the image.
  • the influence of ambient light conditions is greater.
  • the captured images may lack important part of the information.
  • FIG. 1A when shooting a portrait, if the ambient light is very weak, that is, a weak light environment, the overall brightness of the captured image is very low, and it is difficult to recognize the face; as shown in FIG. 1B, if the light source is located at the Behind the person, that is, the backlight environment, the brightness of the face part is low, and it is also difficult to identify. Therefore, there is a need to improve the brightness of images captured in extreme environments.
  • the exemplary embodiments of the present disclosure first provide an image processing method, the application scenarios of which include but are not limited to: in the intelligent interaction scenario of a mobile terminal, collecting, monitoring and detecting through an AON (Always ON) camera Face and gesture information to achieve specific interactive functions, such as automatically activating the display when a face is detected, and automatically turning the user interface when a page turning gesture is detected; however, in weak lighting, backlight, extremely strong In an environment such as a light source, the brightness of the face and gesture images collected by the AON camera may be too low or too high, which affects the accuracy of the above detection; through the image processing method of this exemplary embodiment, the image brightness can be improved to increase the The accuracy of detection of faces, gestures, etc.
  • AON Automatic ON
  • Exemplary embodiments of the present disclosure also provide an electronic device for executing the above-described image processing method.
  • the electronic devices include, but are not limited to, computers, smart phones, tablet computers, game consoles, wearable devices, and the like.
  • an electronic device includes a processor and a memory.
  • the memory is used to store executable instructions of the processor, and may also store application data, such as image data, game data, etc.; the processor is configured to execute the image processing method in this exemplary embodiment by executing the executable instructions.
  • the mobile terminal 200 may specifically include: a processor 210 , an internal storage area 221 , an external memory interface 222 , a USB (Universal Serial Bus, Universal Serial Bus) interface 230 , a charging management module 240 , and a power management module 241 , battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, headphone jack 274, sensor module 280, display screen 290, camera module 291, Indicator 292, motor 293, key 294, SIM (Subscriber Identification Module, Subscriber Identification Module) card interface 295 and so on.
  • a processor 210 an internal storage area 221 , an external memory interface 222 , a USB (Universal Serial Bus, Universal Serial Bus) interface 230 , a charging management module 240 , and a power management module 241 , battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, headphone jack 274, sensor module 280
  • the processor 210 may include one or more processing units, for example, the processor 210 may include an AP (Application Processor, application processor), a modem processor, a GPU (Graphics Processing Unit, graphics processor), an ISP (Image Signal Processor, image signal processor), controller, encoder, decoder, DSP (Digital Signal Processor, digital signal processor), baseband processor and/or NPU (Neural-Network Processing Unit, neural network processor), etc.
  • AP Application Processor
  • modem processor e.g., graphics processing circuitry
  • GPU Graphics Processing Unit, graphics processor
  • ISP Image Signal Processor, image signal processor
  • controller encoder, decoder
  • DSP Digital Signal Processor, digital signal processor
  • baseband processor and/or NPU Neural-Network Processing Unit, neural network processor
  • the processor 210 may include one or more interfaces through which connections are formed with other components of the mobile terminal 200 .
  • Internal storage area 221 may be used to store computer executable program code, which includes instructions.
  • the internal storage area 221 may include volatile memory, such as DRAM (Dynamic Random Access Memory, dynamic random access memory), SRAM (Static Random Access Memory, static random access memory), and may also include non-volatile memory, such as at least one disk memory devices, flash memory devices, UFS (Universal Flash Storage, universal flash memory), etc.
  • the processor 210 executes various functional applications and data processing of the mobile terminal 200 by executing instructions stored in the internal storage area 221 and/or instructions stored in a memory provided in the processor.
  • the external memory interface 222 can be used to connect an external memory, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200.
  • the external memory communicates with the processor 210 through the external memory interface 222 to implement data storage functions, such as storing music, video and other files.
  • the USB interface 230 is an interface conforming to the USB standard specification, and can be used to connect a charger to charge the mobile terminal 200, and can also be connected to an earphone or other electronic devices.
  • the charging management module 240 is used to receive charging input from the charger. While charging the battery 242, the charging management module 240 can also supply power to the device through the power management module 241; the power management module 241 can also monitor the state of the battery.
  • the wireless communication function of the mobile terminal 200 may be implemented by the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modulation and demodulation processor, the baseband processor, and the like.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • the mobile communication module 250 may provide wireless communication solutions including 2G/3G/4G/5G etc. applied on the mobile terminal 200 .
  • the wireless communication module 260 can provide applications on the mobile terminal 200 including WLAN (Wireless Local Area Networks, wireless local area network) (such as Wi-Fi (Wireless Fidelity, wireless fidelity) network), BT (Bluetooth, Bluetooth), GNSS (Global Navigation Satellite System, global navigation satellite system), FM (Frequency Modulation, frequency modulation), NFC (Near Field Communication, short-range wireless communication technology), IR (Infrared, infrared technology) and other wireless communication solutions.
  • WLAN Wireless Local Area Networks, wireless local area network
  • Wi-Fi Wireless Fidelity, wireless fidelity
  • BT Bluetooth
  • GNSS Global Navigation Satellite System, global navigation satellite system
  • FM Frequency Modulation, frequency modulation
  • NFC Near Field Communication, short-range wireless communication technology
  • IR Infrared, infrared technology
  • the mobile terminal 200 may implement a display function through a GPU, a display screen 290, an AP, and the like.
  • the mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the encoder, the decoder, the GPU, the display screen 290, the AP, and the like.
  • the camera module 291 can include various types of cameras, such as AON cameras, wide-angle cameras, high-definition cameras, etc.
  • the camera can be arranged at any position of the mobile terminal 200, for example, arranged on the side of the display screen 290 to form a front camera, Or set on the opposite side of the display screen 290 to form a rear camera.
  • the mobile terminal 200 can implement audio functions through an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, an AP, and the like.
  • the sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyro sensor 2803, an air pressure sensor 2804, etc., to realize different sensing detection functions.
  • the indicator 292 can be an indicator light, which can be used to indicate the charging status, the change of power, and can also be used to indicate messages, missed calls, notifications, and the like.
  • the motor 293 can generate vibration prompts, and can also be used for touch vibration feedback and the like.
  • the keys 294 include a power-on key, a volume key, and the like.
  • the mobile terminal 200 may support one or more SIM card interfaces 295 for connecting the SIM cards to realize functions such as calling and data communication.
  • FIG. 3 shows a schematic flow of the image processing method in this exemplary embodiment, which may include:
  • Step S310 acquiring an image to be processed from multiple consecutive frames of images
  • Step S320 performing brightness mapping processing on the image to be processed to generate an intermediate image
  • Step S330 obtaining at least one frame of reference image by using the above-mentioned consecutive multi-frame images
  • Step S340 performing fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.
  • the brightness mapping processing of the image to be processed can improve the brightness problems in the image, such as low brightness, high brightness, uneven local brightness, etc. It realizes noise reduction and repairs the lack of local information in the image, so as to obtain an optimized image with clearly visible image content, which is conducive to the further realization of face recognition, gesture recognition, target detection and other applications; further, this solution improves image capture and processing. Robustness in extreme lighting environments reduces the dependence on the performance of hardware such as cameras and image sensors. For example, for image sensors with low photosensitive performance, this solution can be used to optimize the captured images to obtain high-quality images. This helps to reduce hardware costs. On the other hand, based on the continuous multi-frame images collected during video shooting or image preview, this solution can realize the optimization of any one of the frame images without external information, and has low implementation cost and high practicability.
  • step S310 the to-be-processed image is acquired from consecutive multiple frames of images.
  • the above-mentioned continuous multi-frame images may be images continuously collected by a camera, for example, a camera shoots a video or continuously collects a preview image, and the like.
  • the image to be processed can be any one of the frame images.
  • the method in FIG. 3 is executed in real time by taking the currently collected frame of image as the image to be processed, so as to realize the processing of each frame of image. .
  • step S310 may include
  • Step S410 obtaining the current RAW image from consecutive multiple frames of RAW images
  • Step S420 performing channel conversion processing on the current RAW image to obtain an image to be processed.
  • the RAW image refers to an image stored in a RAW format, and generally refers to an original image collected by an image sensor in a camera.
  • the image sensor collects the light signal through a Bayer filter and converts it into a digital signal to obtain a RAW image.
  • each pixel in the RAW image has only one color in RGB and is arranged in a Bayer array.
  • an AON camera may be set up on the terminal device, and the above-mentioned continuous multiple frames of RAW images may be collected and obtained
  • a currently collected RAW image is acquired, and since the color channels of each pixel point are different, channel conversion processing is performed on the current RAW image to obtain an image to be processed.
  • Channel conversion processing is to unify the color channels of each pixel, including but not limited to the following two methods:
  • each pixel of the current RAW image to any channel in RGB, for example, uniformly convert it to the R channel.
  • the R channel and B channel of the current RAW image can be converted into the G channel to obtain an image with all pixels in the G channel.
  • the pixels of the R channel and the B channel can be mapped to convert to the pixel value of the G channel, for example:
  • the R, G, and B channels can be converted into grayscale values according to a certain coefficient ratio, or the RAW image can be processed by Demosaic (de-mosaic) to obtain an RGB image, and then The gray value (Gray) of each pixel is calculated by the following formula (2) to obtain a gray image.
  • each pixel needs 10 bits, of which 8 bits record the pixel value and 2 bits record the channel information.
  • a single-channel image is obtained through channel conversion processing (a grayscale image can be regarded as a special single-channel image), and there is no need to record channel information, so each pixel only needs 8 bits.
  • RGB images single-channel images have greatly reduced data volume and save the bit width for recording channel information, thereby reducing the data volume of subsequent processing, which is conducive to real-time optimization processing.
  • the single-channel image can characterize the brightness of the image to be processed, and the information is relatively sufficient.
  • step S320 luminance mapping processing is performed on the image to be processed to generate an intermediate image.
  • the brightness mapping process refers to mapping the brightness value of each pixel in the image to be processed into a new value.
  • the brightness values of all pixels can be adjusted in the same direction, such as uniform increase or uniform decrease, the brightness change values of different pixels can be the same or different; the brightness values of different pixels can also be adjusted in different directions , such as reducing the brightness of the too bright part, increasing the brightness of the too dark part, adjusting the gray scale of the image, etc.
  • a mapping relationship between luminance values before and after mapping can be pre-configured, and the mapping relationship can be related to the luminance level or luminance distribution of the image to be processed itself, and then the luminance mapping process is implemented according to the mapping relationship.
  • Intermediate image can be pre-configured, and the mapping relationship can be related to the luminance level or luminance distribution of the image to be processed itself, and then the luminance mapping process is implemented according to the mapping relationship.
  • step S320 may include:
  • the illumination information is information representing the ambient illumination condition of the image to be processed, for example, it may include the ratio of illuminance value to backlight.
  • the illuminance value is a measure of the amount of light received per unit area of the captured image.
  • the backlight ratio is the proportion of the backlight part of the image to be processed.
  • the illumination information can reflect whether the image to be processed has defects in illumination, such as the overall illumination is excessive, the illumination is insufficient, and the local illumination distribution is unbalanced, etc., and further, brightness mapping processing can be performed on the image to be processed.
  • the luminance value of the image to be processed may be determined according to the exposure parameter of the image to be processed.
  • the exposure parameter includes at least one of exposure time (Exptime), sensitivity (ISO), and aperture value (F value).
  • Expotime Expotime
  • ISO sensitivity
  • F value aperture value
  • Exponce parameters are recorded when the image to be processed is captured, and for cameras on smartphones, the aperture value is usually a fixed value.
  • the exposure parameter is used to estimate the illuminance value. The more comprehensive the acquired exposure parameter, the more accurate the estimated illuminance value. For example, you can refer to the following formula (3):
  • Lumi a 3 *F 2 /(Exptime*ISO) (3)
  • Lumi represents the illuminance value, the unit is lux; a 3 represents the empirical coefficient, such as 12.4, which can be adjusted according to the actual shooting scene or camera performance; F is the aperture value, which is the ratio of the focal length of the lens to the effective diameter of the lens; Exptime is the exposure time, The unit is seconds; ISO is the sensitivity.
  • step S320 may include:
  • the image to be processed is subjected to global luminance mapping processing.
  • the first preset condition may include: less than the first threshold, or greater than the second threshold.
  • the first threshold is a threshold for measuring underexposure
  • the second threshold is a threshold for measuring overexposure.
  • the two thresholds can be determined based on experience, or can be adjusted according to the actual scene. For example, the first threshold and the second threshold are appropriately increased during the day. , and appropriately reduce the first threshold and the second threshold in the dark.
  • a global luminance mapping process Global Luminance Mapping
  • the to-be-processed image is subjected to a global brightness upward mapping process, that is, global brightening.
  • the to-be-processed image is subjected to global brightness down mapping processing, that is, global brightness reduction.
  • the brightness values of different pixel points in the image to be processed can be mapped to higher brightness values according to a preset mapping curve.
  • the mapping curve can be a linear curve, which reflects a linear mapping relationship, or a quadratic curve or other nonlinear curve, which reflects a nonlinear mapping relationship.
  • a plurality of mapping curves may be configured, which correspond to different mapping intensities respectively, and the higher the mapping intensities, the more significant the brightening will be.
  • the mapping intensity and which mapping curve to use can be determined according to the illuminance value of the image to be processed or the actual scene requirements. For example, the lower the illuminance value, the higher the mapping intensity, and the larger the overall slope of the mapping curve.
  • step S320 may further include:
  • tone-mapping processing is performed on the to-be-processed image.
  • the illumination value of the image to be processed does not meet the first preset condition, indicating that the global illumination of the image to be processed is relatively suitable, and then the backlight ratio of the image to be processed is calculated to determine whether there is a local inappropriate situation.
  • the backlight ratio of the to-be-processed image may be determined according to the brightness histogram of the to-be-processed image, which may specifically include: setting a plurality of brightness levels, counting the pixel ratios in each brightness level, and forming a to-be-processed image The brightness histogram of the image; if there are at least two brightness levels, the brightness difference value reaches the set value (the set value can be an empirical value, or can be determined according to the specific scene, such as when there is a prominent light source in the scene, the set value Generally larger, the setting is generally smaller under natural light or no significant light source), then the lower brightness level is determined as the backlight part; the brightness histogram of all the backlight parts is counted to obtain the proportion of the backlight part in the image to be processed, that is Backlight ratio.
  • the set value can be an empirical value, or can be determined according to the specific scene, such as when there is a prominent light source in the scene, the set value Generally larger, the setting is generally smaller under
  • the highest brightness value in the image to be processed can also be obtained, and the brightness threshold value can be determined according to the highest brightness value.
  • the highest brightness value can be multiplied by a fixed coefficient less than 1 to obtain the brightness threshold value;
  • the pixels below the brightness threshold are extracted, and the connected regions are extracted, that is, the isolated pixels are filtered out; the proportion of the connected regions in the image to be processed is taken as the backlight ratio.
  • the second preset condition may include: the backlight ratio is greater than the third threshold. It should be noted that the above calculation of the backlight ratio is to estimate the possible backlight phenomenon in the image to be processed. When the backlight ratio is greater than the third threshold, it can be considered that the probability of the existence of the backlight phenomenon is high.
  • the third threshold is a threshold for measuring whether there is a backlight phenomenon, which can be determined according to experience or actual scenarios. In this case, tone mapping processing (Tone Mapping) is performed on the image to be processed. The tone mapping process essentially still maps the brightness. Different from the above-mentioned global brightness mapping process, the tone mapping process can change the brightness range or brightness level distribution of the image, and the brightness adjustment direction of each pixel can be different.
  • the brightness is adjusted in the same direction, such as increasing the brightness as a whole or decreasing the brightness as a whole, to adjust the overall brightness level; in the tone mapping process, For different pixels in the whole image, the brightness can be adjusted in different directions. For example, the brighter parts are mapped downwards, and the parts with lower brightness are mapped upwards to adjust the brightness distribution.
  • the tone mapping process can be implemented by a mapping curve.
  • Figure 6 shows the mapping curves used in tone mapping processing, wherein the abscissa is the luminance value before mapping, the ordinate is the luminance value after mapping, the curves A, B, and C are the mapping curves under different mapping intensities, and the curve C is The mapping strength is the highest, and curve A has the lowest mapping strength. It can be seen that after the brightness value is mapped by the mapping curve, the brightness distribution of the image to be processed can be mapped to a smaller range. Caused by the local invisible problem. Generally, the higher the backlight ratio, or the greater the difference between the backlight part and the high-brightness part, the higher the mapping intensity is.
  • mapping curve used in the tone mapping processing is different from the mapping curve used in the above-mentioned global brightness mapping processing: the former is generally a non-linear curve, and a section of the slope is greater than 45 degrees (that is, the part of the brightness upward mapping processing, generally the brightness is relatively high.
  • the lower section), the slope of the other section is less than 45 degrees (that is, the part where the brightness is mapped downward, generally the section with higher brightness);
  • the latter can be a linear curve or a non-linear curve, and the slope of the entire curve is greater than 45 degrees or less.
  • the backlight ratio of the image to be processed does not meet the second preset condition, it means that the global illumination and local illumination of the image to be processed are relatively good, and the brightness mapping process may not be performed, and the The image to be processed serves as an intermediate image.
  • FIG. 7 shows a schematic flow of luminance mapping processing, including:
  • Step S710 determining the luminance value Lumi of the image to be processed
  • Step S720 compare the illuminance value with the first threshold value T1 and the second threshold value T2; when Lumi ⁇ T1 or Lumi>T2, it is determined that the first preset condition is met, and step S730 is performed; otherwise, step S740 is performed;
  • Step S730 performing global brightness mapping processing on the image to be processed
  • Step S740 determine the backlight ratio (BL ratio) of the image to be processed
  • Step S750 compare the backlight ratio with the third threshold value T3; when BL ratio>T3, it is determined that the second preset condition is met, and step S760 is performed; otherwise, the image to be processed is not processed, and jumps to step S770;
  • Step S760 performing tone mapping processing on the image to be processed
  • step S770 an intermediate image is obtained.
  • the brightness of the image to be processed is improved from the global and local levels, and in the obtained intermediate image, the image information missing due to the brightness problem can be recovered to a certain extent.
  • step S330 at least one frame of reference image is acquired by using the above-mentioned consecutive multiple frames of images.
  • the reference image is one or more frames of images that are continuous in time with the image to be processed, which can form supplementary image information of the image to be processed.
  • step S330 may include:
  • Step S810 acquiring at least one frame of images other than the image to be processed among the above-mentioned consecutive multiple frames of images;
  • Step S820 performing luminance mapping processing on the at least one frame of image to obtain a reference image
  • Step S830 acquiring an optimized image corresponding to the at least one frame of image as a reference image.
  • any one or more frames of images other than the image to be processed can be selected from the above-mentioned continuous multiple frames of images.
  • the more the number of selected images the closer the time to the image to be processed, the more conducive to improving the optimization effect.
  • the image to be processed is the i-th frame image in the above-mentioned consecutive multi-frame images, and i is a positive integer not less than 2
  • the i-m-th frame image to the i-1-th frame image can be selected, that is, the image located in the image to be processed
  • the previous m frames of images are used for subsequent optimization, where m is any positive integer.
  • the value of m can be determined by combining experience, actual demand and computing power performance.
  • step S820 and step S830 one of them may be selectively executed.
  • step S820 for the luminance mapping processing of the above-mentioned at least one frame of image, reference may be made to the luminance mapping processing performed on the image to be processed in step S320 and FIG. 7 .
  • the optimized image corresponding to the above-mentioned at least one frame of image may be an image obtained by optimizing the image by using the method flow of FIG. The method process is optimized to obtain an optimized image corresponding to each frame of image, and the optimized image can be used as a reference image for optimizing the next frame of image.
  • the exposure parameter of the image to be processed may be different from the exposure parameter of the above-mentioned at least one frame of image, for example, any one of exposure time, sensitivity, and aperture value may be different.
  • the image to be processed or the intermediate image can be complementary to the above-mentioned at least one frame of image or reference image to form information in terms of exposure and brightness.
  • the device can control the camera to collect the above-mentioned continuous multiple frames of images with different exposure parameters. For example, when the image is collected, the exposure time is gradually increased, or the sensitivity is gradually increased, and the exposure parameters between different frame images are different, so that the continuous The information in the multi-frame images is maximized to form a more effective information complementation.
  • step S340 a fusion process is performed on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed.
  • the reference image and the intermediate image have a slight difference in time, so there is a difference in image information, and the fusion of the two images can form a repairing effect on the image information of the detail part.
  • the image frequencies in the two images can be scanned separately and compared, and an area in the reference image whose image frequency is higher than that in the intermediate image is selected, and the area in the reference image is fused with the intermediate image.
  • weights can be performed on the pixels at the same position, for example, the weights are determined according to the image frequencies of the pixels in the two images, and weighted fusion is performed.
  • step S340 may include:
  • Time series filtering is performed on the reference image and the intermediate image.
  • time stamps of the reference image and the intermediate image they can be arranged into an image sequence, and then the image information in the image sequence can be converted into time-series signals, and the time-series signals can be filtered, such as Gaussian filtering, mean filtering, etc.
  • time-series signals can be filtered, such as Gaussian filtering, mean filtering, etc.
  • noise reduction Various ways to achieve further optimization effects such as noise reduction.
  • the brightness of the fused image may also be adjusted according to the above-mentioned continuous multiple frames to obtain an optimized image corresponding to the image to be processed. Since the intermediate image has undergone brightness mapping processing, its brightness may be significantly different from other frame images, resulting in brightness jumps of the video image. Therefore, the brightness of the fused image can be adjusted according to the brightness of other frame images, for example, selecting the same area in the other frame images and the fused image respectively, based on the brightness difference of the area in the two images, to The overall brightness adjustment is performed on the image after fusion processing, thereby ensuring the brightness consistency between consecutive frame images. It should be noted that, unlike the brightness mapping processing in step S320, the brightness adjustment performed here is generally brightness fine-tuning or brightness smoothing processing.
  • FIG. 9 shows an example diagram of the effect after the image to be processed is optimized, wherein the upper row is the to-be-processed image, and the lower row is the corresponding optimized image. It can be seen that the brightness of faces or gestures in the image to be processed is low, which makes it impossible to see clearly. After optimization, clearer face or gesture information can be obtained.
  • At least one of target detection, face recognition, and gesture recognition may be performed on the optimized image.
  • target detection, face recognition, and gesture recognition may be performed on the optimized image.
  • Figure 10 shows an example of face recognition on the optimized image. It can be seen that after optimization, the face part is clearly visible, so that the area where the face is located and the facial feature points are accurately detected.
  • a data set containing 845 images of faces and gestures is tested for optimization processing, most of the images are illuminance values of 3 to 15 lux, and the distance between the face or gesture and the camera is 15 ⁇ 40cm, belonging to low light environment and backlight environment.
  • the accuracy rate is 95.2%
  • the recall rate is 16.7%
  • the F1 value (F1 value is a kind of algorithm model in the field of machine learning). Evaluation index, the calculation method is ) is 28%;
  • the accuracy rate is 99.1%
  • the recall rate is 82.3%
  • the F1 value is 90%.
  • the image processing method of the present exemplary embodiment has a very significant improvement effect on the recognition algorithm in a low-light environment and a backlight environment. Therefore, even in extreme lighting environments, results similar to those in normal lighting environments can be obtained, which increases the robustness of the recognition algorithm and broadens its application scenarios.
  • the image processing apparatus 1100 may include a processor 1110 and a memory 1120 .
  • the memory 1120 stores the following program modules:
  • the to-be-processed image acquisition module 1121 is used to acquire the to-be-processed image from consecutive multiple frames of images;
  • An intermediate image generation module 1122 configured to perform luminance mapping processing on the image to be processed to generate an intermediate image
  • a reference image acquisition module 1123 configured to acquire at least one frame of reference image by using the above-mentioned consecutive multiple frames of images
  • the image fusion processing module 1124 is used to perform fusion processing on the reference image and the intermediate image to obtain an optimized image corresponding to the image to be processed;
  • the processor 1110 may be used to execute the above-described program modules.
  • the intermediate image generation module 1122 is configured to:
  • the illumination information includes an illumination value
  • the intermediate image generation module 1122 is configured to:
  • the image to be processed is subjected to global luminance mapping processing.
  • the intermediate image generation module 1122 is configured to:
  • the illuminance value of the image to be processed is determined according to the exposure parameter of the image to be processed; the exposure parameter includes at least one of exposure time, sensitivity, and aperture value.
  • the illumination information further includes a backlight ratio;
  • the intermediate image generation module 1122 is configured to:
  • tone-mapping processing is performed on the to-be-processed image.
  • the intermediate image generation module 1122 is configured to:
  • the backlight ratio of the to-be-processed image is determined according to the brightness histogram of the to-be-processed image.
  • the intermediate image generation module 1122 is configured to:
  • the image to be processed is used as an intermediate image.
  • the reference image acquisition module 1123 is configured to:
  • An optimized image corresponding to the at least one frame of image is obtained as a reference image.
  • the exposure parameter of the image to be processed is different from the exposure parameter of the above-mentioned at least one frame of image.
  • the image to be processed is the i-th frame of images in consecutive multiple frames of images; the above-mentioned at least one frame of image includes the i-m-th frame of images to the i-1-th frame of images in the consecutive multi-frame images; i is a positive integer not less than 2, and m is any positive integer.
  • the image fusion processing module 1124 is configured to:
  • Time series filtering is performed on the reference image and the intermediate image.
  • the image fusion processing module 1124 is configured to:
  • the brightness of the fused image is adjusted according to the above-mentioned continuous multi-frame images, so as to obtain an optimized image corresponding to the image to be processed.
  • the to-be-processed image acquisition module 1121 is configured to:
  • the to-be-processed image acquisition module 1121 is configured to:
  • the image processing apparatus 1100 is configured in a terminal device, and the terminal device includes an AON camera, which is used to collect the above-mentioned continuous multiple frames of RAW images.
  • the memory 1120 further includes the following program modules:
  • the image recognition application module is used to perform at least one process of target detection, face recognition, and gesture recognition on the above-mentioned optimized image.
  • FIG. 12 shows a schematic diagram of the architecture of this exemplary embodiment.
  • the electronic device is configured with an AON camera, which runs the AON camera service, and can realize the underlying processing of the image through the image signal processor. After processing, the corresponding optimized image is obtained, and the optimized image is provided to the AON software service.
  • the AON software service can perform monitoring, recognition and other services through the digital signal processor, such as face recognition and gesture recognition on the optimized image, obtain the corresponding recognition results, and provide the recognition results to the application service.
  • the application service can run related applications through the main processor, and use the above face and gesture recognition results to implement specific interactive instructions, such as screen locking and unlocking, and user interface page turning.
  • Exemplary embodiments of the present disclosure also provide a computer-readable storage medium that can be implemented in the form of a program product including program code for enabling the electronic device when the program product is run on the electronic device.
  • program product may be implemented as a portable compact disk read only memory (CD-ROM) and include program code, and may be executed on an electronic device, such as a personal computer.
  • CD-ROM portable compact disk read only memory
  • the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • the program product may employ any combination of one or more readable media.
  • the readable medium may be a readable signal medium or a readable storage medium.
  • the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • a computer readable signal medium may include a propagated data signal in baseband or as part of a carrier wave with readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a readable signal medium can also be any readable medium other than a readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
  • Program code embodied on a readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural programming Language - such as the "C" language or similar programming language.
  • the program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.
  • the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (eg, using an Internet service provider business via an Internet connection).
  • LAN local area network
  • WAN wide area network
  • an external computing device eg, using an Internet service provider business via an Internet connection
  • the exemplary embodiments described herein may be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to cause a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary embodiment of the present disclosure.
  • a computing device which may be a personal computer, a server, a terminal device, or a network device, etc.
  • modules or units of the apparatus for action performance are mentioned in the above detailed description, this division is not mandatory. Indeed, according to exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above may be further divided into multiple modules or units to be embodied.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique. Le procédé de traitement d'image comprend : l'acquisition, parmi de multiples trames continues d'images, d'une image à traiter (S310) ; la réalisation d'un traitement de mappage de luminance sur l'image à traiter et la génération d'une image intermédiaire (S320) ; l'acquisition d'au moins une trame d'image de référence à l'aide des multiples trames continues d'images (S330) ; et la réalisation d'un traitement de filtrage de séquence temporelle sur l'image de référence et l'image intermédiaire de façon à obtenir une image optimisée correspondant à l'image à traiter (S340). Au moyen du procédé, les problèmes de luminance dans les images sont atténués, la réduction du bruit d'image et la restauration d'informations locales manquantes sont obtenues, et un résultat d'optimisation d'image de haute qualité est obtenu.
PCT/CN2020/138407 2020-12-22 2020-12-22 Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique WO2022133749A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2020/138407 WO2022133749A1 (fr) 2020-12-22 2020-12-22 Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique
CN202080107017.0A CN116457822A (zh) 2020-12-22 2020-12-22 图像处理方法、装置、存储介质与电子设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/138407 WO2022133749A1 (fr) 2020-12-22 2020-12-22 Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique

Publications (1)

Publication Number Publication Date
WO2022133749A1 true WO2022133749A1 (fr) 2022-06-30

Family

ID=82158530

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/138407 WO2022133749A1 (fr) 2020-12-22 2020-12-22 Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique

Country Status (2)

Country Link
CN (1) CN116457822A (fr)
WO (1) WO2022133749A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116095432A (zh) * 2022-12-28 2023-05-09 深圳创维－Rgb电子有限公司 电视设备摄像结构的控制方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102779330A (zh) * 2012-06-13 2012-11-14 京东方科技集团股份有限公司 图像增强方法、图像增强装置和显示装置
US20130314568A1 (en) * 2011-02-18 2013-11-28 DigitalOptics Corporation Europe Limited Dynamic Range Extension by Combining Differently Exposed Hand-Held Device-Acquired Images
CN106570850A (zh) * 2016-10-12 2017-04-19 成都西纬科技有限公司 一种图像融合方法
CN109785423A (zh) * 2018-12-28 2019-05-21 广州华多网络科技有限公司 图像补光方法、装置及计算机设备

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5663093B2 (ja) * 2010-10-01 2015-02-04 ドルビー ラボラトリーズ ライセンシング コーポレイション 参照ピクチャー処理のための最適化されたフィルタ選択
CN110009587B (zh) * 2019-04-09 2021-03-02 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备
CN110033418B (zh) * 2019-04-15 2023-03-24 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130314568A1 (en) * 2011-02-18 2013-11-28 DigitalOptics Corporation Europe Limited Dynamic Range Extension by Combining Differently Exposed Hand-Held Device-Acquired Images
CN102779330A (zh) * 2012-06-13 2012-11-14 京东方科技集团股份有限公司 图像增强方法、图像增强装置和显示装置
CN106570850A (zh) * 2016-10-12 2017-04-19 成都西纬科技有限公司 一种图像融合方法
CN109785423A (zh) * 2018-12-28 2019-05-21 广州华多网络科技有限公司 图像补光方法、装置及计算机设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116095432A (zh) * 2022-12-28 2023-05-09 深圳创维－Rgb电子有限公司 电视设备摄像结构的控制方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN116457822A (zh) 2023-07-18

Similar Documents

Publication Publication Date Title
CN109671106B (zh) 一种图像处理方法、装置与设备
WO2019183813A1 (fr) Dispositif et procédé de capture d'image
CN111179282B (zh) 图像处理方法、图像处理装置、存储介质与电子设备
WO2020057198A1 (fr) Procédé et dispositif de traitement d'images, dispositif électronique et support de stockage
WO2019148978A1 (fr) Procédé et appareil de traitement d'images, support de stockage et dispositif électronique
WO2022179335A1 (fr) Procédé et appareil de traitement vidéo, dispositif électronique et support de stockage
CN110766621B (zh) 图像处理方法、装置、存储介质及电子设备
WO2019120016A1 (fr) Procédé et appareil de traitement d'image, support d'informations et dispositif électronique
KR102566998B1 (ko) 이미지 선명도를 결정하기 위한 장치 및 방법
US20170070718A1 (en) Advanced Multi-Band Noise Reduction
US11715184B2 (en) Backwards-compatible high dynamic range (HDR) images
KR20150099302A (ko) 전자 장치와, 그의 제어 방법
US9380218B2 (en) Highlight exposure metric and its applications
CN112289279B (zh) 屏幕亮度调整方法、装置、存储介质与电子设备
WO2022087973A1 (fr) Procédé et appareil de traitement d'image, support lisible par ordinateur, et dispositif électronique
WO2021237732A1 (fr) Procédé et appareil d'alignement d'image, dispositif électronique et support de stockage
US10187566B2 (en) Method and device for generating images
CN113810603B (zh) 点光源图像检测方法和电子设备
CN110047060B (zh) 图像处理方法、装置、存储介质及电子设备
WO2023160285A9 (fr) Procédé et appareil de traitement vidéo
CN114463191B (zh) 一种图像处理方法及电子设备
CN113452969B (zh) 图像处理方法和装置
CN117274109B (zh) 图像处理方法、降噪模型训练方法及电子设备
CN115086567A (zh) 延时摄影方法和装置
WO2022133749A1 (fr) Procédé et appareil de traitement d'image, support d'enregistrement et dispositif électronique

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20966329

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202080107017.0

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20966329

Country of ref document: EP

Kind code of ref document: A1