WO2021218364A1 - 一种图像增强方法及电子设备 - Google Patents

一种图像增强方法及电子设备 Download PDF

Info

Publication number
WO2021218364A1
WO2021218364A1 PCT/CN2021/078787 CN2021078787W WO2021218364A1 WO 2021218364 A1 WO2021218364 A1 WO 2021218364A1 CN 2021078787 W CN2021078787 W CN 2021078787W WO 2021218364 A1 WO2021218364 A1 WO 2021218364A1
Authority
WO
WIPO (PCT)
Prior art keywords
channel component
component
channel
enhanced
electronic device
Prior art date
Application number
PCT/CN2021/078787
Other languages
English (en)
French (fr)
Inventor
韩骁枫
陈晓亮
张田
王建
吴国星
郭锐
马杰延
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21795665.5A priority Critical patent/EP4120183A4/en
Publication of WO2021218364A1 publication Critical patent/WO2021218364A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/64—Circuits for processing colour signals
    • H04N9/68—Circuits for processing colour signals for controlling the amplitude of colour signals, e.g. automatic chroma control circuits
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20208—High dynamic range [HDR] image processing

Definitions

  • This application relates to the field of image processing technology, and in particular to an image enhancement method and electronic equipment.
  • Image enhancement is the use of a series of image enhancement techniques to improve the quality and visual effects of the image, highlight the features of interest in the image, and obtain valuable information in the image, thereby transforming the image into a more suitable for analysis and analysis by humans or machines.
  • the processing form makes the processed image have better effects for certain specific applications.
  • Image enhancement theory is widely used in the fields of biomedicine, industrial production, public safety, and aerospace.
  • an image enhancement method uses a super-resolution (SR) technology to upscale the input image into a high-resolution output image.
  • Another image enhancement method is to use high dynamic range imaging (HDR) technology to extend the value of each pixel in a standard dynamic range input image to a high dynamic range.
  • HDR high dynamic range imaging
  • the decoding result of the video decoder on the smart phone side is usually in the YUV color space instead of the RGB color space, and the current image enhancement methods are generated by using the video of the RGB color space for model training and inference.
  • the current image enhancement methods are based on deep learning and machine learning super-resolution algorithms, which require a large amount of calculation, which cannot meet the real-time requirements of the mobile terminal for video processing, and the power consumption is also large.
  • the present application provides an image enhancement method and electronic equipment to achieve better image enhancement effects and solve the problem that the image quality algorithm in the prior art does not meet the real-time requirements of the mobile terminal for video processing.
  • an embodiment of the present application provides an image enhancement method, including: an electronic device acquires a color image in the YUV color space, because the electronic device performs image processing on the first U channel component and the first V channel component of the color image The size is enlarged to obtain the processed second U channel component and the second V channel component, and super-resolution processing is performed on the first Y channel component of the color image to obtain the processed second Y channel component; in addition, the electronic device Perform high dynamic range imaging processing on the color image to obtain component coefficients respectively corresponding to the first Y channel component, the first U channel component, and the first V channel component.
  • the electronic device obtains the enhanced channel component corresponding to the first Y channel component according to the component coefficients corresponding to the second Y channel component and the first Y channel component, and the second U channel component
  • the component coefficients corresponding to the first U-channel component to obtain the enhanced channel component corresponding to the first U-channel component, and the component coefficients corresponding to the second V-channel component and the first V-channel component, Obtain the enhanced channel component corresponding to the first V channel component
  • the electronic device corresponds to the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component, and the first V channel component
  • the electronic device adopts different image enhancement methods for different channels of the color image in the YUV color space, and both super-resolution processing and high dynamic range imaging processing are performed on the color image in the YUV color space. Improved image enhancement effect.
  • the CPU of the electronic device amplifies the first U channel component and the first V channel component of the color image to obtain the amplified second U channel component and the second V channel component.
  • the electronic The NPU of the device performs super-resolution processing on the first Y-channel component of the color image to obtain the processed second Y-channel component; and the NPU of the electronic device performs high-dynamic-range imaging processing on the color image to obtain three types of The component coefficient corresponding to the channel. It can be seen that the parallel execution of the CPU and NPU of the electronic device can increase the processing speed of the image to meet the real-time requirements of the mobile terminal for video processing.
  • the enhanced channel component corresponding to the first U channel component can satisfy: Among them, U'represents the enhanced channel component corresponding to the first U channel component, W U represents the component coefficient corresponding to the first U channel component, U represents the second U channel component, Is a fixed parameter, Is a fixed parameter, with The value range of is [0,1];
  • the enhanced channel component corresponding to the first V channel component can satisfy: Among them, V'represents the enhanced channel component corresponding to the first V channel component, W V represents the component coefficient corresponding to the first V channel component, V represents the second V channel component, Is a fixed parameter, Is a fixed parameter, with The value range of is [0,1].
  • the electronic device not only performs super-resolution processing on color images in the YUV color space, but also performs high dynamic range imaging processing, which can improve the image enhancement effect.
  • the component coefficients corresponding to the U channel are the same as the component coefficients corresponding to the V channel, so that color deviation can be avoided.
  • the NPU of the electronic device performs super-resolution processing on the first Y channel component input to the dual-branch multitask neural network model, and outputs the second Y channel component; at the same time, the NPU of the electronic device inputs
  • the color image in the dual-branch multi-task neural network model is subjected to high dynamic range imaging processing, and the component coefficients corresponding to the first Y channel component, the first U channel component and the first V channel component are output respectively. That is, the NPU can simultaneously complete video super-resolution enhancement and high dynamic range imaging enhancement.
  • the electronic device includes a video decoder; the video decoder is used to decode the input video stream to generate a color image, where each frame of the image in the input video stream is a color image in the RGB color space.
  • the image enhancement method provided in the embodiments of the present application can implement the above-mentioned image enhancement processing on the image decoded by the decoder.
  • the electronic device includes a video encoder.
  • the video encoder can also encode the enhanced color image of each frame to generate an image-enhanced video stream.
  • Each frame of image in the video stream after image enhancement processing is a color image in the RGB color space. That is to say, the image enhancement method provided by the embodiments of the present application can implement the encoder encoding of the image in the YUV color space after image enhancement, and the image in the RGB color space after being encoded by the encoder.
  • an embodiment of the present application provides an electronic device including a processor and a memory, where the memory is used to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor,
  • the electronic device can implement any possible design method in any of the above-mentioned aspects.
  • an embodiment of the present application further provides a device, which includes a module/unit that executes any one of the possible design methods in any of the foregoing aspects.
  • modules/units can be realized by hardware, or by hardware executing corresponding software.
  • an embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium includes a computer program.
  • the computer program runs on an electronic device, the electronic device executes any of the above aspects. Any one of the possible design methods.
  • the embodiments of the present application also provide a method that includes a computer program product, which when the computer program product runs on a terminal, causes the electronic device to execute any one of the possible designs in any of the above-mentioned aspects.
  • an embodiment of the present application further provides a chip, which is coupled with a memory and is used to execute a computer program stored in the memory to execute any possible design method of any one of the above aspects.
  • FIG. 1 is a schematic structural diagram of a mobile phone provided by an embodiment of the application
  • FIG. 2 is a schematic structural diagram of an Android operating system provided by an embodiment of the application.
  • FIG. 3 is a schematic diagram of an electronic device provided by an embodiment of the application.
  • FIG. 4 is a schematic diagram of a dual-branch multi-task neural network model provided by an embodiment of the application.
  • FIGS. 5A and 5B are schematic diagrams of an image enhancement method provided by an embodiment of this application.
  • FIG. 6 is a schematic diagram of an example of an image enhancement method provided by an embodiment of this application.
  • FIG. 7 is a schematic diagram of an image enhancement device provided by an embodiment of the application.
  • Color space Color is usually described by three independent attributes. The combined effect of three independent variables naturally constitutes a space coordinate, which is the color space. But the described color object itself is objective, and different color spaces just measure the same object from different angles.
  • the color space can be divided into two major categories: the first type, the primary color space, such as red, green, blue, RGB (red, green, blue, RGB) color space, and the second type, the color and light separation color space, such as the YUV color space, Y "Means brightness (luminance or luma), which is the grayscale value; while "U” and “V” mean chrominance (chrominance or chroma), and hue saturation (hue, saturation, value, HSV) Color space and so on.
  • the three components of the RGB color space can be: red, green, and blue components; the three components of the YUV color space can be YUV components.
  • Y represents brightness (luminance or luma), which is the grayscale value; while "U” and “V” represent chrominance (chrominance or chroma).
  • a mobile terminal For mobile terminals, at present, after acquiring a video stream, a mobile terminal needs to decode it through a video decoder.
  • the color image obtained by decoding is usually a color image in the YUV color space, rather than a color image in the RGB color space.
  • the Y channel reflects the brightness of the image in the video stream
  • the UV channel reflects the color of the image in the video stream.
  • the human eye is more sensitive to brightness and less sensitive to subtle differences in color. This requires that when processing video streams in the YUV color space, different image enhancement methods should be adopted for different channels.
  • an embodiment of the present application provides an image enhancement method, which can implement not only super-resolution processing on the video stream, but also high dynamic range enhancement processing on the video stream in the YUV color space, and super-resolution processing.
  • the output image after the rate processing and the output image after the high dynamic range enhancement processing are synthesized to generate the final enhanced image, thereby better improving the quality of the image displayed by the mobile terminal.
  • the method and the device are based on the same technical concept. Since the principles of the method and the device to solve the problem are similar, the implementation of the device and the method can be referred to each other, and the repetition will not be repeated.
  • This application can be applied to video stream playback scenarios, such as mobile phone video playback, video playback in video applications, or TV multimedia content playback.
  • the image enhancement method provided in this application can be applied to electronic equipment including a central processing unit (CPU) and an embedded neural-network processing unit (NPU), the electronic equipment including but not limited to a camera , Cameras, smart cameras, smart camcorders, smart mobile terminals (such as mobile phones, tablet computers, etc.), smart TVs and other portable terminals.
  • the image enhancement method provided in this application can also be applied to camera playback equipment in a video surveillance system, or implemented by a cloud server in the video surveillance system.
  • Exemplary embodiments of portable terminals include, but are not limited to, carrying Or portable terminals with other operating systems.
  • the above-mentioned portable terminal may also be, for example, a laptop computer (Laptop) having a touch-sensitive surface (such as a touch panel) or the like. It should also be understood that, in some other embodiments, the aforementioned terminal may also be a desktop computer with a touch-sensitive surface (such as a touch panel).
  • a laptop computer having a touch-sensitive surface (such as a touch panel) or the like.
  • the aforementioned terminal may also be a desktop computer with a touch-sensitive surface (such as a touch panel).
  • FIG. 1 shows a schematic structural diagram of the mobile phone 100.
  • the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, mobile communication module 151, wireless communication module 152, Audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, SIM card interface 195 and so on.
  • a processor 110 an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, mobile communication module 151, wireless communication module 152, Audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, SIM card interface 195 and so on.
  • the sensor module 180 may include a gyroscope sensor 180A, an acceleration sensor 180B, a pressure sensor 180C, a proximity light sensor 180G, a fingerprint sensor 180H, and a touch sensor 180K (of course, the mobile phone 100 may also include other sensors, such as a temperature sensor, a distance sensor, and a magnetic sensor. Sensors, ambient light sensors, air pressure sensors, bone conduction sensors, etc., not shown in the figure).
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100.
  • the mobile phone 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (Neural-network Processing Unit, NPU) Wait.
  • AP application processor
  • ISP image signal processor
  • controller memory
  • video codec digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • NPU neural network Processing Unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the controller may be the nerve center and command center of the mobile phone 100.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the processor 110 may run the image enhancement method provided in the embodiments of the present application to achieve a better image enhancement effect and solve the problem that the image quality algorithm in the prior art does not meet the real-time requirements of the mobile terminal for processing video.
  • the processor 110 may include different devices. For example, when a CPU and GPU are integrated, the CPU and GPU can cooperate to execute the image enhancement method provided in the embodiments of the present application. In order to get faster processing efficiency.
  • the display screen 194 is used to display images, videos, and the like.
  • the display screen 194 includes a display panel.
  • the display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the mobile phone 100 may include one or N display screens 194, and N is a positive integer greater than one.
  • the touch sensor and/or pressure sensor on the display screen 194 can collect the user's touch operation, and the touch sensor and/or pressure sensor can transmit the detected sensor data to the processor 110 so that the processor 110 can determine The corresponding state of the sensor unit.
  • the display screen 194 may be an integrated flexible display screen, or a spliced display screen composed of two rigid screens and a flexible screen located between the two rigid screens.
  • the processor 110 may control the image enhancement effect of the display interface on the display screen 194 based on the image enhancement method.
  • the camera 193 (a front camera or a rear camera, or a camera can be used as a front camera or a rear camera) is used to capture still images or videos.
  • the camera 193 may include photosensitive elements such as a lens group and an image sensor, where the lens group includes a plurality of lenses (convex lens or concave lens) for collecting light signals reflected by the object to be photographed and transmitting the collected light signals to the image sensor .
  • the image sensor generates an original image of the object to be photographed according to the light signal.
  • the internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions.
  • the processor 110 executes various functional applications and data processing of the mobile phone 100 by running instructions stored in the internal memory 121.
  • the internal memory 121 may include a storage program area and a storage data area.
  • the storage program area can store operating system, application program (such as camera application, WeChat application, etc.) codes and so on.
  • the data storage area can store data created during the use of the mobile phone 100 (such as data collected by sensors, and preset reference state sequence sets).
  • the internal memory 121 may also store the code of the image enhancement algorithm provided in the embodiment of the present application.
  • the processor 110 may control the display interface on the display screen 194.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • the code of the image enhancement algorithm provided in the embodiment of the present application can also be stored in an external memory.
  • the processor 110 can run the image enhancement algorithm code stored in the external memory through the external memory interface 120, and the processor 110 determines the enhanced image of the electronic device, and then controls the display screen 194 according to the enhanced image. Display interface.
  • the function of the sensor module 180 is described below.
  • the gyroscope sensor 180A can be used to determine the movement posture of the mobile phone 100.
  • the angular velocity of the mobile phone 100 around three axes ie, x, y, and z axes
  • the gyroscope sensor 180A can be used to detect the current motion state of the mobile phone 100, such as shaking or static, such as horizontal or vertical screen.
  • the gyroscope sensor 180A can be used to detect folding or unfolding operations on the display screen 194.
  • the gyroscope sensor 180A may report the detected folding operation or unfolding operation as an event to the processor 110 to determine the folding state or unfolding state of the display screen 194.
  • the acceleration sensor 180B can detect the magnitude of the acceleration of the mobile phone 100 in various directions (generally three axes). When the display screen in the embodiment of the present application is a foldable screen, the acceleration sensor 180B can be used to detect folding or unfolding operations on the display screen 194. The acceleration sensor 180B may report the detected folding operation or unfolding operation as an event to the processor 110 to determine the folding state or unfolding state of the display screen 194.
  • the pressure sensor 180C is used to sense the pressure signal and can convert the pressure signal into an electrical signal.
  • the pressure sensor 180B may be disposed on the display screen 194 or the housing part.
  • the capacitive pressure sensor may include at least two parallel plates with conductive materials. When a force is applied to the pressure sensor 180B, the capacitance between the electrodes changes. The mobile phone 100 determines the intensity of the pressure according to the change of the capacitance. When a touch operation acts on the display screen 194, the mobile phone 100 detects the intensity of the touch operation according to the pressure sensor 180B.
  • the mobile phone 100 may also calculate the touched position based on the detection signal of the pressure sensor 180B.
  • touch operations that act on the same touch position but have different touch operation strengths may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity greater than the first pressure threshold acts on both sides of the housing, an instruction to view unread messages is executed.
  • the proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.
  • the light emitting diode may be an infrared light emitting diode.
  • the mobile phone emits infrared light through light-emitting diodes.
  • Mobile phones use photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the phone. When insufficient reflected light is detected, the mobile phone can determine that there is no object near the mobile phone.
  • the proximity light sensor 180G can be arranged on the upper side of the screen 194, and the proximity light sensor 180G can detect whether a human face is close to the screen according to the optical path difference of the infrared signal.
  • the proximity light sensor 180G can be arranged on the first screen of the foldable display screen 194, and the proximity light sensor 180G can detect the first screen according to the optical path difference of the infrared signal. The size of the folding or unfolding angle between the screen and the second screen.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the mobile phone 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, and so on.
  • Touch sensor 180K also called “touch panel”.
  • the touch sensor 180K may be disposed on the display screen 194 or the housing part, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”.
  • the touch sensor 180K is used to detect touch operations acting on or near it.
  • the touch sensor may transmit the detected sensor data to the processor 110, so that the processor 110 determines the state of the sensor unit according to the sensor data, and then determines the state sequence corresponding to the sensor unit of the electronic device.
  • the visual output related to the touch operation can be provided through the display screen 194.
  • the touch sensor 180K may also be disposed on the surface of the mobile phone 100, which is different from the position of the display screen 194.
  • the display screen 194 of the mobile phone 100 displays a main interface, and the main interface includes icons of multiple applications (such as a camera application, a WeChat application, etc.).
  • the display screen 194 displays an interface of the camera application, such as a viewfinder interface.
  • the wireless communication function of the mobile phone 100 can be realized by the antenna 1, the antenna 1, the mobile communication module 151, the wireless communication module 152, the modem processor, and the baseband processor.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in the mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna can be used in combination with a tuning switch.
  • the mobile communication module 151 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied on the mobile phone 100.
  • the mobile communication module 151 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like.
  • the mobile communication module 151 can receive electromagnetic waves by the antenna 1, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation.
  • the mobile communication module 151 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic wave radiation via the antenna 1.
  • at least part of the functional modules of the mobile communication module 151 may be provided in the processor 110.
  • at least part of the functional modules of the mobile communication module 151 and at least part of the modules of the processor 110 may be provided in the same device.
  • the modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.
  • the demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194.
  • the modem processor may be an independent device.
  • the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 151 or other functional modules.
  • the wireless communication module 152 can provide applications on the mobile phone 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellite systems. (global navigation satellite system, GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • WLAN wireless local area networks
  • BT Bluetooth
  • GNSS global navigation satellite system
  • FM frequency modulation
  • NFC near field communication
  • IR infrared technology
  • the wireless communication module 152 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 152 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module 152 can also receive the signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic waves and radiate it through the antenna 2.
  • the mobile phone 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.
  • the mobile phone 100 can receive the key 190 input, and generate key signal input related to the user settings and function control of the mobile phone 100.
  • the mobile phone 100 can use the motor 191 to generate a vibration notification (for example, an incoming call vibration notification).
  • the indicator 192 in the mobile phone 100 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, and so on.
  • the SIM card interface 195 in the mobile phone 100 is used to connect to the SIM card.
  • the SIM card can be connected to and separated from the mobile phone 100 by inserting into the SIM card interface 195 or pulling out from the SIM card interface 195.
  • the mobile phone 100 may include more or less components than those shown in FIG. 1, which is not limited in the embodiment of the present application.
  • the illustrated mobile phone 100 is only an example, and the mobile phone 100 may have more or fewer parts than shown in the figure, may combine two or more parts, or may have a different part configuration.
  • the various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
  • the embodiment of the present invention takes an Android system with a layered architecture as an example to illustrate the software structure of an electronic device.
  • Fig. 2 is a software structure block diagram of an electronic device according to an embodiment of the present invention.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface.
  • the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
  • the application layer can include a series of application packages.
  • the application package can include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
  • the application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.
  • the window manager is used to manage window programs.
  • the window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take a screenshot, etc.
  • the content provider is used to store and retrieve data and make these data accessible to applications.
  • the data may include videos, images, audios, phone calls made and received, browsing history and bookmarks, phone book, etc.
  • the view system includes visual controls, such as controls that display text, controls that display pictures, and so on.
  • the view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.
  • the phone manager is used to provide the communication function of the electronic device. For example, the management of the call status (including connecting, hanging up, etc.).
  • the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.
  • the notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and it can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, and so on.
  • the notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or a scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text messages are prompted in the status bar, prompt sounds, electronic devices vibrate, and indicator lights flash.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.
  • the application layer and application framework layer run in a virtual machine.
  • the virtual machine executes the java files of the application layer and the application framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • the system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.
  • the surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files.
  • the media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.
  • the 2D graphics engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
  • hardware may refer to various types of sensors, such as acceleration sensors, gyroscope sensors, touch sensors, pressure sensors, etc. involved in the embodiments of the present application.
  • the electronic device involved in the embodiment of the present application may include at least one processor 301, a memory 302, a video encoder 304, and a video decoder 303.
  • the processor 301 may include a CPU301A and an NPU301B.
  • the processor 301 may refer to the processor 110 in FIG. 1
  • the memory 302 may refer to the internal memory 121 in FIG. 1.
  • the image enhancement method provided by this application includes the following processes: 1) First, the video decoder 303 decodes the input video stream in the RGB color space to obtain one or more frames of the input image in the YUV color space; 2) Then the input image of one or more frames of YUV color space is sent to CPU301A for data preprocessing; 3) The image processed by CPU301A is sent to the dual-branch multi-task neural network model in NPU301B for processing; 4 ) After the neural network model is processed, the super resolution (SR) image after super resolution processing and the component coefficients after high dynamic range imaging processing are obtained.
  • SR super resolution
  • the SR image and the component coefficients are sent back to the CPU301A for execution
  • the processing thread is used to synthesize the final enhanced image; 5)
  • the enhanced image is transmitted to the video encoder 304 through the NPU301B, and is encoded by the video encoder 304 into a video stream in the RGB color space and then output.
  • the dual-branch multi-task neural network model in NPU301B is generated through training in advance.
  • the dual-branch multi-task neural network model includes two branches. The first branch corresponds to super-resolution processing tasks, and the second branch corresponds to high dynamic range imaging processing. Tasks, super-resolution processing tasks and high dynamic range imaging processing tasks can be executed in parallel.
  • the first branch of the dual-branch multi-task neural network model may include 4 convolutional layers and 4 relu (activation) layers, and the first branch is used to perform super-resolution processing on the input color image.
  • the fourth convolutional layer is a sub-pixel convolutional layer, which means that the output of this layer will be up-sampled to the target resolution.
  • the first convolutional layer 401a in the first branch expands the feature dimension, for example, from 3 to 32, and the next second convolutional layer 401b and third convolutional layer 401c extract each on the feature map The local characteristics of the pixel.
  • the fourth sub-pixel convolution layer 401d is used to up-sample the feature map and enlarge it to the target resolution.
  • the output result of the fourth sub-pixel convolution layer 401d will be activated by the relu (activation) layer to ensure that the value of each pixel is greater than zero.
  • the second branch of the dual-branch multi-task neural network model may include 4 convolutional layers and 4 relu (activation) layers.
  • the second branch is used to perform high dynamic range imaging processing on the input color image.
  • the fourth convolutional layer is a sub-pixel convolutional layer, and the second branch trains the network to predict the weight mask map instead of directly predicting the value of each pixel.
  • the electronic equipment may also include one or more of the following: general-purpose processors, image signal processors (image signal processors, ISP), microprocessors, digital signal processors (digital signal processors, DSP), Field-programmable gate array (field-programmable gate array, FPGA), etc.
  • general-purpose processors image signal processors (image signal processors, ISP), microprocessors, digital signal processors (digital signal processors, DSP), Field-programmable gate array (field-programmable gate array, FPGA), etc.
  • the memory 302 in the electronic device can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), or can store information and Other types of dynamic storage devices for instructions can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory, CD-ROM or other optical discs Storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures And any other media that can be accessed by the device, but not limited to this.
  • the memory can exist independently and is connected to the processor 301 through a bus (as shown in FIG. 3). The memory may also be integrated with the processor 301.
  • the memory 302 may be used to store application program codes for executing the solutions of the present application, and the processor 301 controls the execution, that is, the processor is used to execute the program codes stored in the memory 302 to implement the present application
  • the image enhancement method in the embodiment may be used to implement the present application.
  • the image enhancement method provided by the embodiment of the application is described in detail below.
  • the image enhancement method can be implemented by an image enhancement device in an electronic device, and the image enhancement device can be integrated into the program code of the electronic device as a plug-in.
  • the specific method is shown in the figure. Shown in Figure 5A and Figure 5B.
  • Step 501 The electronic device acquires a color image in the YUV color space.
  • the color image includes a first Y channel component, a first U channel component, and a first V channel component.
  • the input image of the electronic device is a color image in the RGB color space
  • the video decoder of the electronic device needs to decode the input video stream to generate the color image.
  • the electronic device obtains a video stream from a video server corresponding to the video application, and each frame of color image in the video stream is a color image in an RGB color space.
  • the input color image is a color image in the RGB color space
  • the video decoder of the electronic device decodes the video stream from the network side
  • each frame of the color image after decoding is a color image in the YUV color space.
  • Step 502 the electronic device enlarges the image size of the first U-channel component and the first V-channel component to obtain the processed second U-channel component and the second V-channel component, and the electronic device super-resolves the first Y-channel component Rate processing to obtain the processed second Y channel component.
  • the electronic device can simultaneously scale the first U-channel color component and the first V-channel color component in the decoded color image in the YUV color space through the bilinear interpolation algorithm.
  • the CPU of the electronic device can simultaneously amplify the first U-channel color component and the first V-channel color component in the decoded color image through the bilinear interpolation algorithm to obtain the amplified second U-channel component And the second V channel component.
  • the electronic device performs super-resolution processing on the first Y channel component separately, in conjunction with FIG. 5B.
  • the dual-branch multi-task neural network model in the NPU of the electronic device separately performs super-resolution processing on the Y channel component of the input color image to achieve the purpose of targeted optimization of the image according to the characteristics of the YUV color space.
  • Step 503 The electronic device performs high dynamic range imaging processing on the color image to obtain component coefficients respectively corresponding to the first Y channel component, the first U channel component, and the first V channel component.
  • the component coefficients in the embodiments of the present application are also called weighted mask graphs, and the value range of the component coefficients can be from 0 to 1.
  • the component coefficients of each channel are expressed in the Y channel and the U channel, respectively.
  • the so-called coefficient of change of each pixel value refers to the calculation of the difference between the target component of the pixel and the component of the current image for the pixel of any channel, and then the ratio between the absolute value of the difference and the target component is used as the pixel
  • the coefficient of change of the value is used as the component coefficient.
  • the dual-branch multi-task neural network model in the NPU of the electronic device performs high dynamic range imaging processing on the input color image to obtain the component coefficients corresponding to the first Y channel component and the components corresponding to the first U channel component The coefficient and the component coefficient corresponding to the first V channel component.
  • the above steps 502 and 503 can be performed in parallel, that is, the CPU of the electronic device amplifies the first U channel component and the first V channel component, Obtain the amplified second U-channel component and the second V-channel component; the dual-branch multi-task neural network model in the NPU of the electronic device performs super-resolution processing on the second Y-channel component of the input color image, and the electronic device’s
  • the dual-branch multi-task neural network model in the NPU performs high dynamic range imaging processing on the input color image, and obtains the component coefficients corresponding to the first Y channel component, the component coefficients corresponding to the first U channel component, and the first V channel component.
  • the component coefficient In this way, the image processing efficiency can be further improved to meet the real-time requirements of the mobile terminal for video processing.
  • the dual-branch multi-task neural network model in the NPU of the electronic device has the first Y of the input color image.
  • the channel components are subjected to super-resolution processing.
  • the dual-branch multi-task neural network model in the NPU of the electronic device performs high dynamic range imaging processing on the input color image to obtain the component coefficients corresponding to the first Y channel component, and the first Y channel component.
  • a component coefficient corresponding to the U channel component and a component coefficient corresponding to the first V channel component can be further improved to meet the real-time requirements of the mobile terminal for video processing.
  • Step 504 The electronic device obtains the enhanced channel component corresponding to the first Y channel component according to the component coefficients corresponding to the second Y channel component and the first Y channel component, and the second U channel component Obtaining the enhanced channel component corresponding to the first U channel component by the component coefficient corresponding to the first U channel component, and the component coefficients corresponding to the second V channel component and the first V channel component, Obtain an enhanced channel component corresponding to the first V channel component.
  • the electronic device may multiply the component coefficient corresponding to the first Y channel component by the second Y channel component to obtain the enhanced channel component corresponding to the first Y channel component. That is, the electronic device obtains the Y-channel color value based on the Y-channel upsampling result of the Y-channel weight mask map predicted by the dual-branch multi-task neural network model in the NPU and the pixel values output by the dual-branch multi-task neural network model.
  • the enhanced channel component corresponding to the first U channel component satisfies:
  • U′ represents the enhanced channel component corresponding to the first U channel component
  • W U represents the component coefficient corresponding to the first U channel component
  • U represents the second U channel component
  • Is a fixed parameter Is a fixed parameter, with The value range of is [0,1].
  • the enhanced channel component corresponding to the first V channel component satisfies:
  • V' represents the enhanced channel component corresponding to the first V channel component
  • W V represents the component coefficient corresponding to the first V channel component
  • V represents the second V channel component
  • Is a fixed parameter Is a fixed parameter, with The value range of is [0,1].
  • the values of W U and W V may be set to be the same in the embodiment of the present application.
  • Step 505 The electronic device synthesizes the enhanced color according to the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component, and the enhanced channel component corresponding to the first V channel image.
  • the electronic device calculates the component coefficients obtained by the NPU prediction and the components after the super-resolution processing of each channel, and synthesizes the color image in the enhanced YUV color space.
  • the video encoder in the electronic device can also encode the enhanced color images of each frame to obtain an image-enhanced video stream, and various colors in the image-enhanced video stream
  • the image is a color image in the RGB color space.
  • the electronic device can obtain the video stream from the video server corresponding to the video application, and the electronic device performs the above-mentioned image enhancement processing on the video stream, and then passes through the video encoder
  • the color image in the final output RGB color space can be displayed on the monitor.
  • the above-mentioned enhancement method provided by the embodiments of the present application may adopt a parallel mode of CPU and GPU to improve image processing efficiency. Specifically, as shown in Figure 6.
  • the embodiment of the application sets two threads for preprocessing and post-processing, and for the NPU of the electronic device, the embodiment of the application sets a neural network inference thread. These 3 threads process data in parallel, which speeds up the calculation.
  • the preprocessing thread of the CPU of the electronic device executes: the i-th frame of color image is enlarged as shown in step 502, and then the NPU of the electronic device
  • the neural network inference thread executes: the super-resolution processing shown in step 502 and the high dynamic range imaging processing shown in step 503 are performed on the color image of the i-th frame.
  • the post-processing thread of the electronic CPU executes: performing the calculation process shown in step 504 on the i-th frame of color image, and executing the process of synthesizing the enhanced image shown in step 505.
  • the preprocessing thread of the CPU can simultaneously execute: perform the magnification processing shown in step 502 on the i+1 frame color image, and after the electronic CPU
  • the processing thread performs the above-mentioned processing
  • the neural network inference thread of the NPU of the electronic device can be executed at the same time: super-resolution processing shown in step 502 and high dynamic range imaging shown in step 503 are performed on the i+1th frame of the color image deal with.
  • CPU and NPU parallel computing are realized. Therefore, the present invention can perform real-time video processing on the processor (including CPU and NPU) of the mobile terminal, thereby improving the picture quality of the video without occupying more network bandwidth. quality.
  • the image enhancement device may include an image acquisition unit 701 for acquiring a color image in the YUV color space;
  • the first processing unit 702 is configured to scale the first U channel component and the first V channel component to obtain the amplified second U channel component and the second V channel component.
  • the second processing unit 703 is configured to perform super-resolution processing on the first Y channel component to obtain the processed second Y channel component; Component, the first U-channel component, and the first V-channel component corresponding to the component coefficients; the first processing unit 702 is further configured to calculate the second Y-channel component and the first Y-channel component Corresponding component coefficients, the enhanced channel component corresponding to the first Y channel component is obtained, and the component coefficients corresponding to the second U channel component and the first U channel component are obtained, and the first U channel component is obtained.
  • the enhanced channel component corresponding to the channel component, the enhanced channel component corresponding to the first U channel component, and the enhanced channel component corresponding to the first V channel are combined to synthesize an enhanced color image.
  • the enhanced channel component corresponding to the first U channel component satisfies: Where U′ represents the enhanced channel component corresponding to the first U channel component, W U represents the component coefficient corresponding to the first U channel component, U represents the second U channel component, Is a fixed parameter, Is a fixed parameter, with The value range of is [0,1];
  • V' represents the enhanced channel component corresponding to the first V channel component
  • W V represents the component coefficient corresponding to the first V channel component
  • V represents the second V channel component
  • Is a fixed parameter Is a fixed parameter, with The value range of is [0,1].
  • the component coefficient corresponding to the first U channel component is the same as the component coefficient corresponding to the first V channel.
  • the device further includes a decoding unit 704, specifically configured to decode the input video stream to generate a color image in the YUV color space, wherein each of the input video streams The frame image is a color image in the RGB color space.
  • the device further includes an encoding unit 705, specifically configured to encode the enhanced color image of each frame after the first processing unit 702 synthesizes the enhanced color image to generate an image enhancement After the video stream, each frame of the image in the image-enhanced video stream is a color image in the RGB color space.
  • an encoding unit 705 specifically configured to encode the enhanced color image of each frame after the first processing unit 702 synthesizes the enhanced color image to generate an image enhancement After the video stream, each frame of the image in the image-enhanced video stream is a color image in the RGB color space.
  • the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.
  • this application can be provided as methods, systems, or computer program products. Therefore, this application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
  • Telephone Function (AREA)

Abstract

本申请提供一种图像增强方法及电子设备，用以实现图像增强，解决现有技术中存在的图像质量算法不满足移动端处理视频实时要求的问题。该方法包括：电子设备获取YUV彩色空间下的彩色图像，并且电子设备的CPU对该彩色图像的第一U通道分量和第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量，以及电子设备的NPU对该彩色图像的第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量；另外，电子设备的NPU对彩色图像进行高动态范围成像处理，得到分别与三个通道分量分别对应的分量系数，最终，电子设备的CPU根据三个通道对应的分量系数和处理后的通道分量，合成增强后的彩色图像。

Description

一种图像增强方法及电子设备
相关申请的交叉引用
本申请要求在2020年04月27日提交中国专利局、申请号为202010342912.2、申请名称为“一种图像增强方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像增强方法及电子设备。
背景技术
随着智能手机的普及，当前的移动端视频用户数量迅速增加，用户对于画面质量的要求也一直在提高，因此需要能够对视频中的亮度、对比度等参数进行自适应调整，优化并增强图像的画质。
图像增强是通过采用一系列图像增强技术去改善图像的质量和视觉效果，突出图像中感兴趣的特征，获得图像中有价值的信息，从而将图像转换成一种更适合于人或者机器进行分析和处理的形式，使得处理后的图像对某些特定的应用有更好的效果。图像增强理论广泛应用于生物医学领域、工业生产领域、公共安全领域以及航空航天领域等。
目前相关技术中，一种图像增强方法是使用一种超分辨率(super resolution，SR)的技术，将输入图像升频转化为高分辨率的输出图像。另一种图像增强方法则是使用高动态范围成像(high dynamic range imaging，HDR)技术，将一个标准动态范围的输入图像中的每个像素的取值，扩展到高动态范围下。对于智能手机而言，智能手机侧的视频解码器的解码结果通常都是YUV颜色空间，而不是RGB颜色空间，而目前的图像增强方法都是利用RGB颜色空间的视频进行模型训练和推断生成的，并不适用于智能手机等移动平台使用。而且目前的图像增强方法基于深度学习和机器学习的超分辨率算法，计算量较大，不能满足移动端处理视频的实时要求，功耗也较大。
发明内容
本申请提供一种图像增强方法及电子设备，用以实现更好的图像增强效果，解决现有技术中存在的图像质量算法不满足移动端处理视频的实时要求的问题。
第一方面，本申请实施例提供了一种图像增强方法，包括：电子设备获取YUV彩色空间下的彩色图像，因电子设备对该彩色图像的第一U通道分量和第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量，并对该彩色图像的第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量；另外，电子设备对该彩色图像进行高动态范围成像处理，得到分别与第一Y通道分量、第一U通道分量和第一V通道分量分别对应的分量系数。
进一步地，电子设备根据所述第二Y通道分量和所述第一Y通道分量对应的分量系数，得到与所述第一Y通道分量对应的增强后通道分量，并所述第二U通道分量和所述第一U 通道分量对应的分量系数，得到与所述第一U通道分量对应的增强后通道分量，并所述第二V通道分量和所述第一V通道分量对应的分量系数，得到与所述第一V通道分量对应的增强后通道分量，最终电子设备根据第一Y通道分量对应的增强后通道分量、第一U通道分量对应的增强后通道分量和第一V通道分量对应的增强后通道分量，合成增强后的彩色图像。
本申请实施例中，电子设备针对YUV颜色空间下的彩色图像的不同通道，采取不同的图像增强方法，既对YUV颜色空间下的彩色图像进行超分辨率处理，也进行高动态范围成像处理，提升了图像增强效果。
在一种可能的设计中，电子设备的CPU对该彩色图像的第一U通道分量和第一V通道分量进行放大，得到放大后的第二U通道分量和第二V通道分量，同时，电子设备的NPU对该彩色图像的第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量；且电子设备的NPU对该彩色图像进行高动态范围成像处理，得到分别与三种通道对应的分量系数。可见，电子设备的CPU和NPU并行执行，可以提高图像的处理速度，以满足移动端处理视频的实时要求。
在一种可能的设计中，第一Y通道分量对应的增强后通道分量可以满足：Y‘＝W Y×Y；其中，Y‘表示第一Y通道分量对应的增强后通道分量，W Y表示与第一Y通道分量对应的分量系数，Y表示第二Y通道分量；
第一U通道分量对应的增强后通道分量可以满足：
Figure PCTCN2021078787-appb-000001
其中，U‘表示第一U通道分量对应的增强后通道分量，W U表示与第一U通道分量对应的分量系数，U表示第二U通道分量，
Figure PCTCN2021078787-appb-000002
为固定参数，
Figure PCTCN2021078787-appb-000003
为固定参数，
Figure PCTCN2021078787-appb-000004
和
Figure PCTCN2021078787-appb-000005
的取值范围为[0,1]；
第一V通道分量对应的增强后通道分量可以满足：
Figure PCTCN2021078787-appb-000006
其中，V‘表示第一V通道分量对应的增强后通道分量，W V表示第一V通道分量对应的分量系数，V表示第二V通道分量,
Figure PCTCN2021078787-appb-000007
为固定参数，
Figure PCTCN2021078787-appb-000008
为固定参数，
Figure PCTCN2021078787-appb-000009
和
Figure PCTCN2021078787-appb-000010
的取值范围为[0,1]。
可见，本申请实施例中，电子设备既对YUV颜色空间下的彩色图像进行超分辨率处理，也进行高动态范围成像处理，可提升图像增强效果。
在一种可能的设计中，U通道对应的分量系数和V通道对应的分量系数相同，这样可以避免出现颜色偏差。
在一种可能的设计中，电子设备的NPU对输入至双分支多任务神经网络模型中的第一Y通道分量进行超分辨率处理，输出第二Y通道分量；同时，电子设备的NPU对输入至双分支多任务神经网络模型中的彩色图像进行高动态范围成像处理，输出分别与第一Y通道分量、第一U通道分量和第一V通道分量分别对应的分量系数。即，NPU可以同时完成视频的超分辨率增强和高动态范围成像增强。
在一种可能的设计中，电子设备包括视频解码器；视频解码器用于对输入的视频流进行解码，生成彩色图像，其中，输入的视频流中每帧图像为RGB颜色空间下的彩色图像。也就是说，本申请实施例所提供的图像增强方法可以实现对解码器解码之后的图像进行上述图像增强处理。
在一种可能的设计中，电子设备包括视频编码器，电子设备合成增强后的彩色图像之后，视频编码器还可以对各帧增强后的彩色图像进行编码，生成图像增强处理后的视频流，图像增强处理后的视频流中每帧图像为RGB颜色空间下的彩色图像。也就是说，本申请实施例所提供的图像增强方法可以实现将图像增强之后的YUV颜色空间下的图像进行编 码器编码，经过编码器编码之后为RGB颜色空间下的图像。
第二方面，本申请实施例提供一种电子设备，包括处理器和存储器，其中，存储器用于存储一个或多个计算机程序；当存储器存储的一个或多个计算机程序被处理器执行时，使得该电子设备能够实现上述任一方面的任意一种可能的设计的方法。
第三方面，本申请实施例还提供一种装置，该装置包括执行上述任一方面的任意一种可能的设计的方法的模块/单元。这些模块/单元可以通过硬件实现，也可以通过硬件执行相应的软件实现。
第四方面，本申请实施例中还提供一种计算机可读存储介质，所述计算机可读存储介质包括计算机程序，当计算机程序在电子设备上运行时，使得所述电子设备执行上述任一方面的任意一种可能的设计的方法。
第五方面，本申请实施例还提供一种包含计算机程序产品，当所述计算机程序产品在终端上运行时，使得所述电子设备执行上述任一方面的任意一种可能的设计的方法。
第六方面，本申请实施例还提供一种芯片，所述芯片与存储器耦合，用于执行所述存储器中存储的计算机程序，以执行上述任一方面的任意一种可能的设计的方法。
以上第二方面至第六方面中任一方面中的各种设计可以达到的技术效果，请参照上述第一方面中各个设计分别可以达到的技术效果描述，这里不再重复赘述。
附图说明
图1为本申请实施例提供的一种手机结构示意图；
图2为本申请实施例提供的一种安卓操作系统结构示意图；
图3为本申请实施例提供一种电子设备示意图；
图4为本申请实施例提供一种双分支多任务神经网络模型示意图；
图5A和图5B为本申请实施例提供一种图像增强方法示意图；
图6为本申请实施例提供一种图像增强方法示例示意图；
图7为本申请实施例提供一种图像增强装置示意图。
具体实施方式
为了方便理解本申请实施例，下面首先介绍与本申请实施例相关的术语。
1、颜色空间，颜色通常用三个独立的属性来描述，三个独立变量综合作用，自然就构成一个空间坐标，这就是颜色空间。但被描述的颜色对象本身是客观的，不同颜色空间只是从不同的角度去衡量同一个对象。颜色空间可以分为两大类：第一类，基色颜色空间，比如红绿蓝(red，green，blue，RGB)颜色空间，第二类，色、亮分离颜色空间，比如YUV颜色空间，Y”表示明亮度(luminance或luma)，也就是灰阶值；而“U”和“V”表示的则是色度(chrominance或chroma)，和色调饱和度明度(hue，saturation，value，HSV)颜色空间等等。
2、分量，RGB颜色空间的三种分量可以是：红(red)、绿(green)、蓝(blue)分量；YUV颜色空间的三种分量可以是YUV分量。其中，“Y”表示明亮度(luminance或luma)，也就是灰阶值；而“U”和“V”表示的则是色度(chrominance或chroma)。
对于移动终端而言，目前移动终端在获取视频流之后，需要先通过视频解码器进行解 码，解码得到的彩色图像通常是YUV颜色空间下的彩色图像，而不是RGB颜色空间下的彩色图像。在YUV颜色空间下，Y通道体现的是视频流中图像的亮度，而UV通道体现的是视频流中图像的颜色。通常来说，人眼对亮度更敏感，而对颜色的细微差异不太敏感。这就要求在处理YUV颜色空间下的视频流的时候，应该针对不同的通道，采取不同的图像增强方法。基于这一分析，本申请实施例提供一种图像增强方法，该方法可以实现在YUV颜色空间下既对视频流进行超分辨率处理，还对视频流进行高动态范围增强处理，并将超分辨率处理后的输出图像和高动态范围增强处理后的输出图像进行合成，生成最终的增强图像，从而更好的提高移动终端显示图像的质量。
其中，方法和装置是基于同一技术构思的，由于方法及装置解决问题的原理相似，因此装置与方法的实施可以相互参见，重复之处不再赘述。
本申请可以应用于视频流播放场景，如手机视频播放、视频应用中的视频播放或电视多媒体内容的播放等。本申请提供的图像增强方法可以应用于包括具有中央处理器(central processing unit，CPU)和嵌入式神经网络处理器(neural-network processing unit，NPU)的电子设备，该电子设备包括但不限于相机、摄像机、智能摄像机、智能摄录机、智能移动终端(比如移动电话(mobile phone)、平板电脑、等等)、智能电视等便携式终端。本申请提供的图像增强方法还可以应用到视频监控系统中的摄像播放设备，或者由视频监控系统中的云端服务器来实现。便携式终端的示例性实施例包括但不限于搭载
Figure PCTCN2021078787-appb-000011
Figure PCTCN2021078787-appb-000012
或者其它操作系统的便携式终端。上述便携式终端也可以是诸如具有触敏表面(例如触控面板)的膝上型计算机(Laptop)等。还应当理解的是，在其他一些实施例中，上述终端也可以是具有触敏表面(例如触控面板)的台式计算机。
下文以电子设备是手机为例，图1示出了手机100的结构示意图。
手机100可以包括处理器110，外部存储器接口120，内部存储器121，USB接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块151，无线通信模块152，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及SIM卡接口195等。其中传感器模块180可以包括陀螺仪传感器180A，加速度传感器180B，压力传感器180C、接近光传感器180G、指纹传感器180H，触摸传感器180K(当然，手机100还可以包括其它传感器，比如温度传感器，距离传感器、磁传感器、环境光传感器、气压传感器、骨传导传感器等，图中未示出)。
可以理解的是，本发明实施例示意的结构并不构成对手机100的具体限定。在本申请另一些实施例中，手机100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor,ISP)，控制器，存储器，视频编解码器，数字信号处理器(digital signal processor,DSP)，基带处理器，和/或神经网络处理器(Neural-network Processing Unit,NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。其中，控制器可以是手机100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
处理器110可以运行本申请实施例提供的图像增强方法，用以实现更好的图像增强效果，解决现有技术中存在的图像质量算法不满足移动端处理视频的实时要求的问题。处理器110可以包括不同的器件，比如集成CPU和GPU时，CPU和GPU可以配合执行本申请实施例提供的图像增强方法，比如图像增强方法中部分算法由CPU执行，另一部分算法由GPU执行，以得到较快的处理效率。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，手机100可以包括1个或N个显示屏194，N为大于1的正整数。本申请实施例中，显示屏194上的触摸传感器和/或压力传感器可以采集用户的触摸操作，触摸传感器和/或压力传感器可以将检测到的传感器数据传递给处理器110，以便处理器110确定传感器单元对应的状态。
在本申请实施例中，显示屏194可以是一个一体的柔性显示屏，也可以采用两个刚性屏以及位于两个刚性屏之间的一个柔性屏组成的拼接显示屏。当处理器110运行本申请实施例提供的图像增强方法后，处理器110可以基于图像增强方法控制显示屏194上的显示界面的图像增强效果。
摄像头193(前置摄像头或者后置摄像头，或者一个摄像头既可作为前置摄像头，也可作为后置摄像头)用于捕获静态图像或视频。通常，摄像头193可以包括感光元件比如镜头组和图像传感器，其中，镜头组包括多个透镜(凸透镜或凹透镜)，用于采集待拍摄物体反射的光信号，并将采集的光信号传递给图像传感器。图像传感器根据所述光信号生成待拍摄物体的原始图像。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。处理器110通过运行存储在内部存储器121的指令，从而执行手机100的各种功能应用以及数据处理。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，应用程序(比如相机应用，微信应用等)的代码等。存储数据区可存储手机100使用过程中所创建的数据(比如传感器采集的数据，以及预设的参考状态序列集合)等。
内部存储器121还可以存储本申请实施例提供的图像增强算法的代码。当内部存储器121中存储的图像增强算法的代码被处理器110运行时，处理器110可以控制显示屏194上的显示界面。
此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。
当然，本申请实施例提供的图像增强算法的代码还可以存储在外部存储器中。这种情况下，处理器110可以通过外部存储器接口120运行存储在外部存储器中的图像增强算法的代码，处理器110确定电子设备的增强后图像，进而根据该增强后的图像控制显示屏194 上的显示界面。
下面介绍传感器模块180的功能。
陀螺仪传感器180A，可以用于确定手机100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180A确定手机100围绕三个轴(即，x，y和z轴)的角速度。即陀螺仪传感器180A可以用于检测手机100当前的运动状态，比如抖动还是静止，比如横屏还是竖屏。
当本申请实施例中的显示屏为可折叠屏时，陀螺仪传感器180A可用于检测作用于显示屏194上的折叠或者展开操作。陀螺仪传感器180A可以将检测到的折叠操作或者展开操作作为事件上报给处理器110，以确定显示屏194的折叠状态或展开状态。
加速度传感器180B可检测手机100在各个方向上(一般为三轴)加速度的大小。当本申请实施例中的显示屏为可折叠屏时，加速度传感器180B可用于检测作用于显示屏194上的折叠或者展开操作。加速度传感器180B可以将检测到的折叠操作或者展开操作作为事件上报给处理器110，以确定显示屏194的折叠状态或展开状态。
压力传感器180C用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180B可以设置于显示屏194或者壳体部分。压力传感器180B的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180B，电极之间的电容改变。手机100根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，手机100根据压力传感器180B检测所述触摸操作强度。手机100也可以根据压力传感器180B的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度大于第一压力阈值的触摸操作作用于壳体两侧时，执行查看未读消息的指令。
接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。手机通过发光二极管向外发射红外光。手机使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定手机附近有物体。当检测到不充分的反射光时，手机可以确定手机附近没有物体。当本申请实施例中的显示屏为不可折叠屏时，接近光传感器180G可以设置在显示屏194的屏幕上侧，接近光传感器180G可根据红外信号的光程差来检测是否有人脸靠近屏幕。当本申请实施例中的显示屏为可折叠屏时，接近光传感器180G可以设置在可折叠的显示屏194的第一屏上，接近光传感器180G可根据红外信号的光程差来检测第一屏与第二屏的折叠角度或者展开角度的大小。
指纹传感器180H用于采集指纹。手机100可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。
触摸传感器180K，也称“触控面板”。触摸传感器180K可以设置于显示屏194或者壳体部分，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的传感器数据传递给处理器110，以便处理器110根据传感器数据确定传感器单元的状态，继而确定出电子设备的传感器单元对应的状态序列。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于手机100的表面，与显示屏194所处的位置不同。
示例性的，手机100的显示屏194显示主界面，主界面中包括多个应用(比如相机应 用、微信应用等)的图标。用户通过触摸传感器180K点击主界面中相机应用的图标，触发处理器110启动相机应用，打开摄像头193。显示屏194显示相机应用的界面，例如取景界面。
手机100的无线通信功能可以通过天线1，天线1，移动通信模块151，无线通信模块152，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。手机100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块151可以提供应用在手机100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块151可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块151可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块151还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块151的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块151的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块151或其他功能模块设置在同一个器件中。
无线通信模块152可以提供应用在手机100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块152可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块152经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块152还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
另外，手机100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。手机100可以接收按键190输入，产生与手机100的用户设置以及功能控制有关的键信号输入。手机100可以利用马达191产生振动提示(比如来电振动提示)。手机100中的指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。手机100中的SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195，或从SIM卡接口195拔出，实现和手机100的接触和分离。
应理解，在实际应用中，手机100可以包括比图1所示的更多或更少的部件，本申请实施例不作限定。图示手机100仅是一个范例，并且手机100可以具有比图中所示出的更 多的或者更少的部件，可以组合两个或更多的部件，或者可以具有不同的部件配置。图中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。
电子设备的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本发明实施例以分层架构的Android系统为例，示例性说明电子设备的软件结构。图2是本发明实施例的电子设备的软件结构框图。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。
如图2所示，应用程序包可以包括电话、相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等应用程序。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
如图2所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供电子设备的通信功能。例如通话状态的管理(包括接通，挂断等)。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的 融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如：MPEG4,H.264,MP3,AAC,AMR,JPG,PNG等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。
2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动。其中，硬件可以指的是各类传感器，例如本申请实施例中涉及的加速度传感器、陀螺仪传感器、触摸传感器、压力传感器等。
参见图3所示，本申请实施例中涉及的电子设备中可以包括至少一个处理器301、以及存储器302、视频编码器304以及视频解码器303。其中，处理器301可以包括CPU301A和NPU301B。其中，该处理器301可以指图1中的处理器110，存储器302可以指图1中的内部存储器121。
结合图3来说，本申请提供的图像增强方法包括如下过程：1)首先由视频解码器303对输入的RGB颜色空间下的视频流解码，得到一帧或多帧YUV颜色空间的输入图像；2)随后该一帧或多帧YUV颜色空间的输入图像被传送至CPU301A中进行数据预处理；3)经过CPU301A处理后的图像被传送至NPU301B中双分支多任务神经网络模型中进行处理；4)经过神经网络模型处理之后得到经过超分辨率处理后的超分辨率(super resolution，SR)图像，以及经过高动态范围成像处理后的分量系数，SR图像和分量系数被传送回CPU301A中执行后处理线程，合成最终的增强图像；5)增强图像通过NPU301B传送到视频编码器304，被视频编码器304编成RGB颜色空间下的视频流后输出。
其中，NPU301B中的双分支多任务神经网络模型是预先通过训练生成，该双分支多任务神经网络模型包括两个分支，第一分支对应超分辨率处理任务，第二分支对应高动态范围成像处理任务，超分辨率处理任务和高动态范围成像处理任务可以并行执行。
如图4所示，该双分支多任务神经网络模型的第一分支可以包含4个卷积层和4个relu(激活)层，第一分支用于对输入的彩色图像进行超分辨率处理。其中，第四个卷积层是子像素卷积层，这意味着该层的输出将被上采样到目标分辨率。第一分支中的第一个卷积层401a将特征维度进行扩展，例如从3扩展到32，接下来的第二个卷积层401b和第三个卷积层401c在特征图上提取每个像素的局部特征。第四个子像素卷积层401d用于对特征图进行上采样，将其放大到目标分辨率。第四个子像素卷积层401d的输出结果将被relu(激活)层激活，以确保每个像素的值都大于零。
另外，该双分支多任务神经网络模型的第二分支可以包含4个卷积层和4个relu(激活)层。第二分支用于对输入的彩色图像进行高动态范围成像处理。其中，第四个卷积层是子像素卷积层，第二分支训练网络来预测权重掩码图，而不是直接预测每个像素的值。
除此之外，该电子设备还可以包括以下一项或者多项：通用处理器、图像信号处理器(image signal processor，ISP)、微处理器、数字信号处理器(digital signal processor，DSP)、现场可编程门阵列(field－programmable gate array，FPGA)等。
电子设备中的存储器302可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact  Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由该装置存取的任何其他介质，但不限于此。存储器可以是独立存在，通过总线与处理器301相连接(如图3所示)。存储器也可以和处理器301集成在一起。
其中，所述存储器302可以用于存储执行本申请方案的应用程序代码，并由处理器301来控制执行，也就是说，所述处理器用于执行所述存储器302中存储的程序代码实现本申请实施例中的图像增强方法。
下面先对本申请实施例提供的图像增强方法进行详细说明，图像增强方法可以由电子设备中图像增强装置来实现，该图像增强装置可以作为一个插件集成在电子设备的程序代码中，具体方法参见图5A和图5B所示。
步骤501，电子设备获取YUV颜色空间下的彩色图像。
其中，该彩色图像包括第一Y通道分量、第一U通道分量和第一V通道分量。
一种可能的示例中，电子设备的输入图像为RGB颜色空间下的彩色图像，电子设备的视频解码器需要对输入的视频流进行解码，生成该彩色图像。示例性地，当电子设备的视频应用接收到用户的播放请求时，电子设备从视频应用对应的视频服务器获取视频流，该视频流中的各帧彩色图像是RGB颜色空间下的彩色图像。如图5B所示，输入彩色图像为RGB颜色空间下的彩色图像，电子设备的视频解码器对来自网络侧的视频流进行视频解码，解码之后的各帧彩色图像为YUV颜色空间下的彩色图像。
步骤502，电子设备对第一U通道分量和第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量，并且电子设备对第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量。
在该步骤中，电子设备可以通过双线性插值算法同时对解码之后的YUV颜色空间下的彩色图像中第一U通道彩色分量和第一V通道彩色分量进行缩放。结合图5B来说，电子设备的CPU可以通过双线性插值算法对解码之后的彩色图像中第一U通道彩色分量和第一V通道彩色分量同时进行放大，得到放大后的第二U通道分量和第二V通道分量。
基于人眼对亮度的变化比对颜色的变化更敏感的考虑，本申请实施例中，在对图像增强处理过程中，电子设备将第一Y通道分量单独进行超分辨率处理，结合图5B来说，电子设备的NPU中的双分支多任务神经网络模型单独对输入的彩色图像的Y通道分量进行超分辨率处理，以达到针对YUV颜色空间的特点，对图像进行针对性的优化的目的。
步骤503，电子设备对彩色图像进行高动态范围成像处理，得到分别与第一Y通道分量、第一U通道分量和第一V通道分量分别对应的分量系数。
需要说明的是，本申请实施例中的分量系数又称为权重掩码图，分量系数的取值范围可以是从0至1，各通道下的分量系数分别表示在Y通道下和U通道、V通道下各个像素值的变化系数。所谓各像素值的变化系数，指的是针对任意一个通道的像素，计算该像素的目标分量与当前图像的分量的差值，再将该差值的绝对值与目标分量之间的比值作为像素值的变化系数，即作为分量系数。
结合图5B来说，电子设备的NPU中的双分支多任务神经网络模型对输入的彩色图像进行高动态范围成像处理，得到第一Y通道分量对应的分量系数、第一U通道分量对应的分量系数和第一V通道分量对应的分量系数。
需要说明的是，在一种可能的实施例中，针对同一帧彩色图像，上述步骤502和步骤503可以并行执行，即电子设备的CPU对第一U通道分量和第一V通道分量进行放大，得到放大后的第二U通道分量和第二V通道分量；电子设备的NPU中的双分支多任务神经网络模型对输入的彩色图像的第二Y通道分量进行超分辨率处理，以及电子设备的NPU中的双分支多任务神经网络模型对输入的彩色图像进行高动态范围成像处理，得到第一Y通道分量对应的分量系数、与第一U通道分量对应的分量系数和第一V通道分量对应的分量系数。这样，可以进一步地提高图像处理效率，以满足移动端对视频处理的实时要求。
基于NPU中的双分支多任务神经网络模型中的第一分支和第二分支具有并行处理的能力，因此，电子设备的NPU中的双分支多任务神经网络模型对输入的彩色图像的第一Y通道分量进行超分辨率处理，同时，及电子设备的NPU中的双分支多任务神经网络模型对输入的彩色图像进行高动态范围成像处理，得到与第一Y通道分量对应的分量系数、与第一U通道分量对应的分量系数和与第一V通道分量对应的分量系数。这样，可以在CPU和NPU并行处理的基础上，进一步地提高图像处理效率，以满足移动端对视频处理的实时要求。
步骤504，电子设备根据所述第二Y通道分量和所述第一Y通道分量对应的分量系数，得到与所述第一Y通道分量对应的增强后通道分量，并所述第二U通道分量和所述第一U通道分量对应的分量系数，得到与所述第一U通道分量对应的增强后通道分量，并所述第二V通道分量和所述第一V通道分量对应的分量系数，得到与所述第一V通道分量对应的增强后通道分量。
具体来说，针对Y通道，电子设备可以将第一Y通道分量对应的分量系数和该第二Y通道分量相乘，得到该第一Y通道分量对应的增强后通道分量。即电子设备基于NPU中的双分支多任务神经网络模型所预测的Y通道的权重掩码图与双分支多任务神经网络模型输出的像素值的Y通道上采样结果，得到Y通道的颜色值。结合图5B来说，第一Y通道分量对应的增强后通道分量满足：Y‘＝W Y×Y；其中，Y‘表示所述第一Y通道分量对应的增强后通道分量，W Y表示与所述第一Y通道分量对应的分量系数，Y表示所述第二Y通道分量。
另外，针对U通道，第一U通道分量对应的增强后通道分量满足：
Figure PCTCN2021078787-appb-000013
Figure PCTCN2021078787-appb-000014
其中，U‘表示所述第一U通道分量对应的增强后通道分量，W U表示与所述第一U通道分量对应的分量系数，U表示所述第二U通道分量，
Figure PCTCN2021078787-appb-000015
为固定参数，
Figure PCTCN2021078787-appb-000016
为固定参数，
Figure PCTCN2021078787-appb-000017
和
Figure PCTCN2021078787-appb-000018
的取值范围为[0,1]。
另外，针对V通道，第一V通道分量对应的增强后通道分量满足：
Figure PCTCN2021078787-appb-000019
Figure PCTCN2021078787-appb-000020
其中，V‘表示所述第一V通道分量对应的增强后通道分量，W V表示所述第一V通道分量对应的分量系数，V表示所述第二V通道分量,
Figure PCTCN2021078787-appb-000021
为固定参数，
Figure PCTCN2021078787-appb-000022
为固定参数，
Figure PCTCN2021078787-appb-000023
和
Figure PCTCN2021078787-appb-000024
的取值范围为[0,1]。
在本申请实施例中，为了避免U通道增强后的分量和V通道增强后的分量出现颜色偏差，本申请实施例中可以设定W U和W V的值相同。
步骤505，电子设备根据第一Y通道分量对应的增强后通道分量、所述第一U通道分量对应的增强后通道分量和所述第一V通道对应的增强后通道分量，合成增强后的彩色图像。
也就是说，电子设备将NPU预测得到的分量系数与各通道经过超分辨率处理后的分 量，进行计算，合成增强过后的YUV颜色空间下的彩色图像。
在一种可能的示例中，电子设备中的视频编码器还可以将各帧增强后的彩色图像进行编码，得到图像增强处理后的视频流，该图像增强处理后的视频流中的各种彩色图像为RGB颜色空间下的彩色图像。示例性地，当电子设备的视频应用接收到用户的播放请求时，电子设备可以从视频应用对应的视频服务器获取视频流，电子设备对该视频流进行上述图像增强处理后，再经过视频编码器的编码，最终输出的RGB颜色空间下的彩色图像可以在显示器中显示。
本申请实施例所提供的上述增强方法可以采用CPU和GPU并行的方式，提高图像处理效率。具体地，如图6所示。针对电子设备的CPU，本申请实施例设置了预处理和后处理两个线程，而针对电子设备的NPU，本申请实施例设置了神经网络推断线程。这3个线程并行处理数据，从而加速了计算。结合图6来说，针对视频流解码后得到的第i帧彩色图像，电子设备的CPU的预处理线程执行：对第i帧彩色图像进行如步骤502所示的放大处理，之后电子设备的NPU的神经网络推断线程执行：对第i帧彩色图像进行如步骤502所示的超分辨率处理和步骤503所示的高动态范围成像处理。之后，电子的CPU的后处理线程执行：对第i帧彩色图像进行如步骤504所示的计算过程，以及执行如步骤505所示的合成增强后图像过程。其中，如图6所示，在NPU执行上述处理过程中，CPU的预处理线程可以同时执行：对第i+1帧彩色图像进行如步骤502所示的放大处理，以及在电子的CPU的后处理线程执行上述处理过程中，电子设备的NPU的神经网络推断线程可以同时执行：对第i+1帧彩色图像进行如步骤502所示的超分辨率处理和步骤503所示的高动态范围成像处理。依此类推，从而实现CPU和NPU并行计算，因此本发明可以在移动端的处理器(包含CPU和NPU)上进行实时的视频处理，从而在不占用更多网络带宽的情况下，提高视频的画质。
基于与上述方法同样的技术构思，本申请实施例还提供了一种图像增强装置，参见图7所示，该图像增强装置可以包括图像获取单元701，用于获取YUV颜色空间下的彩色图像；第一处理单元702，用于对第一U通道分量和第一V通道分量进行缩放，得到放大后的第二U通道分量和第二V通道分量。第二处理单元703，用于对第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量；以及对彩色图像进行高动态范围成像处理，得到分别与所述第一Y通道分量、所述第一U通道分量和所述第一V通道分量分别对应的分量系数；所述第一处理单元702，还用于根据所述第二Y通道分量和所述第一Y通道分量对应的分量系数，得到与所述第一Y通道分量对应的增强后通道分量，并所述第二U通道分量和所述第一U通道分量对应的分量系数，得到与所述第一U通道分量对应的增强后通道分量，并所述第二V通道分量和所述第一V通道分量对应的分量系数，得到与所述第一V通道分量对应的增强后通道分量；所述第一Y通道分量对应的增强后通道分量、所述第一U通道分量对应的增强后通道分量和所述第一V通道对应的增强后通道分量，合成增强后的彩色图像。
其中，在一种可能的实施例中，所述第一Y通道分量对应的增强后通道分量满足：Y‘＝W Y×Y；其中，Y‘表示所述第一Y通道分量对应的增强后通道分量，W Y表示与所述第一Y通道分量对应的分量系数，Y表示所述第二Y通道分量；
所述第一U通道分量对应的增强后通道分量满足：
Figure PCTCN2021078787-appb-000025
其中， U‘表示所述第一U通道分量对应的增强后通道分量，W U表示与所述第一U通道分量对应的分量系数，U表示所述第二U通道分量，
Figure PCTCN2021078787-appb-000026
为固定参数，
Figure PCTCN2021078787-appb-000027
为固定参数，
Figure PCTCN2021078787-appb-000028
和
Figure PCTCN2021078787-appb-000029
的取值范围为[0,1]；
所述第一V通道分量对应的增强后通道分量满足：
Figure PCTCN2021078787-appb-000030
其中，V‘表示所述第一V通道分量对应的增强后通道分量，W V表示所述第一V通道分量对应的分量系数，V表示所述第二V通道分量,
Figure PCTCN2021078787-appb-000031
为固定参数，
Figure PCTCN2021078787-appb-000032
为固定参数，
Figure PCTCN2021078787-appb-000033
和
Figure PCTCN2021078787-appb-000034
的取值范围为[0,1]。
在一种可能的实施例中，第一U通道分量对应的分量系数和所述第一V通道对应的分量系数相同。
在一种可能的实施例中，该装置还包括解码单元704，具体用于对输入的视频流进行解码，从而生成所述YUV颜色空间下的彩色图像，其中，所述输入的视频流中每帧图像为RGB颜色空间下的彩色图像。
在一种可能的实施例中，该装置还包括编码单元705，具体用于在第一处理单元702合成增强后的彩色图像之后，对所述各帧增强后的彩色图像进行编码，生成图像增强后的视频流，所述图像增强后的视频流中每帧图像为RGB颜色空间下的彩色图像。
需要说明的是，本申请实施例中对单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。在本申请的实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的范围。 这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。

Claims (13)

  1. 一种图像增强方法，其特征在于，包括：
    电子设备获取彩色图像，所述彩色图像包括第一Y通道分量、第一U通道分量和第一V通道分量；
    所述电子设备对所述第一U通道分量和所述第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量，并对所述第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量；
    所述电子设备对所述彩色图像进行高动态范围成像处理，得到分别与所述第一Y通道分量、所述第一U通道分量和所述第一V通道分量分别对应的分量系数；
    所述电子设备根据所述第二Y通道分量和所述第一Y通道分量对应的分量系数，得到与所述第一Y通道分量对应的增强后通道分量，并所述第二U通道分量和所述第一U通道分量对应的分量系数，得到与所述第一U通道分量对应的增强后通道分量，并所述第二V通道分量和所述第一V通道分量对应的分量系数，得到与所述第一V通道分量对应的增强后通道分量；
    所述电子设备根据所述第一Y通道分量对应的增强后通道分量、所述第一U通道分量对应的增强后通道分量和所述第一V通道分量对应的增强后通道分量，合成增强后的彩色图像。
  2. 根据权利要求1所述的方法，其特征在于，
    所述第一Y通道分量对应的增强后通道分量满足：Y‘＝W Y×Y；其中，Y‘表示所述第一Y通道分量对应的增强后通道分量，W Y表示与所述第一Y通道分量对应的分量系数，Y表示所述第二Y通道分量；
    所述第一U通道分量对应的增强后通道分量满足：
    Figure PCTCN2021078787-appb-100001
    其中，U‘表示所述第一U通道分量对应的增强后通道分量，W U表示与所述第一U通道分量对应的分量系数，U表示所述第二U通道分量，
    Figure PCTCN2021078787-appb-100002
    为固定参数，
    Figure PCTCN2021078787-appb-100003
    为固定参数，
    Figure PCTCN2021078787-appb-100004
    和
    Figure PCTCN2021078787-appb-100005
    的取值范围为[0,1]；
    所述第一V通道分量对应的增强后通道分量满足：
    Figure PCTCN2021078787-appb-100006
    其中，V‘表示所述第一V通道分量对应的增强后通道分量，W V表示所述第一V通道分量对应的分量系数，V表示所述第二V通道分量,
    Figure PCTCN2021078787-appb-100007
    为固定参数，
    Figure PCTCN2021078787-appb-100008
    为固定参数，
    Figure PCTCN2021078787-appb-100009
    和
    Figure PCTCN2021078787-appb-100010
    的取值范围为[0,1]。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一U通道分量对应的分量系数和所述第一V通道对应的分量系数相同。
  4. 根据权利要求1至3任一项所述的方法，其特征在于，所述电子设备包括嵌入式神经网络处理器NPU和中央处理器CPU，所述NPU集成有双分支多任务神经网络模型；
    所述电子设备的NPU对输入至所述双分支多任务神经网络模型中的所述第一Y通道分量进行超分辨率处理，输出所述第二Y通道分量，并且所述电子设备的CPU对所述第一U通道分量和所述第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量；
    所述电子设备的NPU对输入至所述双分支多任务神经网络模型中的所述彩色图像进行高动态范围成像处理，输出分别与所述第一Y通道分量、所述第一U通道分量和所述第 一V通道分量分别对应的分量系数。
  5. 根据权利要求1至4任一项所述的方法，其特征在于，所述电子设备获取彩色图像，包括：
    所述电子设备的视频解码器对输入的视频流进行解码，生成所述彩色图像，其中，所述输入的视频流中每帧图像为RGB颜色空间下的彩色图像。
  6. 根据权利要求1至4任一项所述的方法，其特征在于，所述电子设备合成增强后的彩色图像之后，还包括：
    所述电子设备的视频编码器对各帧所述增强后的彩色图像进行编码，生成图像增强处理后的视频流，所述图像增强处理后的视频流中每帧图像为RGB颜色空间下的彩色图像。
  7. 一种电子设备，其特征在于，所述电子设备包括处理器和存储器；
    所述存储器存储有程序指令；
    所述处理器用于运行所述存储器存储的所述程序指令，使得所述电子设备执行：
    获取彩色图像，所述彩色图像包括第一Y通道分量、第一U通道分量和第一V通道分量；
    对所述第一U通道分量和所述第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量，并对所述第一Y通道分量进行超分辨率处理，得到处理后的第二Y通道分量；
    对所述彩色图像进行高动态范围成像处理，得到分别与所述第一Y通道分量、所述第一U通道分量和所述第一V通道分量分别对应的分量系数；
    根据所述第二Y通道分量和所述第一Y通道分量对应的分量系数，得到与所述第一Y通道分量对应的增强后通道分量，并所述第二U通道分量和所述第一U通道分量对应的分量系数，得到与所述第一U通道分量对应的增强后通道分量，并所述第二V通道分量和所述第一V通道分量对应的分量系数，得到与所述第一V通道分量对应的增强后通道分量；
    根据所述第一Y通道分量对应的增强后通道分量、所述第一U通道分量对应的增强后通道分量和所述第一V通道分量对应的增强后通道分量，合成增强后的彩色图像。
  8. 根据权利要求7所述的电子设备，其特征在于，所述第一Y通道分量对应的增强后通道分量满足：Y‘＝W Y×Y；其中，Y‘表示所述第一Y通道分量对应的增强后通道分量，W Y表示与所述第一Y通道分量对应的分量系数，Y表示所述第二Y通道分量；
    所述第一U通道分量对应的增强后通道分量满足：
    Figure PCTCN2021078787-appb-100011
    其中，U‘表示所述第一U通道分量对应的增强后通道分量，W U表示与所述第一U通道分量对应的分量系数，U表示所述第二U通道分量，
    Figure PCTCN2021078787-appb-100012
    为固定参数，
    Figure PCTCN2021078787-appb-100013
    为固定参数，
    Figure PCTCN2021078787-appb-100014
    和
    Figure PCTCN2021078787-appb-100015
    的取值范围为[0,1]；
    所述第一V通道分量对应的增强后通道分量满足：
    Figure PCTCN2021078787-appb-100016
    其中，V‘表示所述第一V通道分量对应的增强后通道分量，W V表示所述第一V通道分量对应的分量系数，V表示所述第二V通道分量,
    Figure PCTCN2021078787-appb-100017
    为固定参数，
    Figure PCTCN2021078787-appb-100018
    为固定参数，
    Figure PCTCN2021078787-appb-100019
    和
    Figure PCTCN2021078787-appb-100020
    的取值范围为[0,1]。
  9. 根据权利要求7或8所述的电子设备，其特征在于，所述第一U通道分量对应的分量系数和所述第一V通道分量对应的分量系数相同。
  10. 根据权利要求7至9任一项所述的电子设备，其特征在于，所述处理器包括嵌入式神经网络处理器NPU和中央处理器CPU，所述NPU集成有双分支多任务神经网络模型；
    所述NPU，用于运行所述存储器存储的所述程序指令，使得所述NPU具体执行：
    对输入至所述双分支多任务神经网络模型中的所述第一Y通道分量进行超分辨率处理，输出所述第二Y通道分量；
    所述CPU，用于运行所述存储器存储的所述程序指令，使得所述CPU具体执行：
    对所述第一U通道分量和所述第一V通道分量进行图像尺寸放大，得到处理后的第二U通道分量和第二V通道分量；
    所述NPU，还用于运行所述存储器存储的所述程序指令，执行对输入至所述双分支多任务神经网络模型中的所述彩色图像进行高动态范围成像处理，输出分别与所述第一Y通道分量、所述第一U通道分量和所述第一V通道分量分别对应的分量系数。
  11. 根据权利要求7至10任一项所述的电子设备，其特征在于，所述电子设备还包括视频解码器；
    所述视频解码器，用于对输入的视频流进行解码，生成所述彩色图像并发送给所述处理器，其中，所述输入的视频流中每帧图像为RGB颜色空间下的彩色图像。
  12. 根据权利要求7至10任一项所述的电子设备，其特征在于，所述电子设备还包括视频编码器；
    所述视频编码器，用于对各帧所述增强后的彩色图像进行编码，生成图像增强处理后的视频流，所述图像增强处理后的视频流中每帧图像为RGB颜色空间下的彩色图像。
  13. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储程序或指令，所述程序或所述指令在被一个或多个处理器读取并执行时可实现权利要求1至6中任一项所述的图像增强方法。
PCT/CN2021/078787 2020-04-27 2021-03-02 一种图像增强方法及电子设备 WO2021218364A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP21795665.5A EP4120183A4 (en) 2020-04-27 2021-03-02 IMAGE ENHANCEMENT METHOD AND ELECTRONIC DEVICE

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010342912.2 2020-04-27
CN202010342912.2A CN113643186A (zh) 2020-04-27 2020-04-27 一种图像增强方法及电子设备

Publications (1)

Publication Number Publication Date
WO2021218364A1 true WO2021218364A1 (zh) 2021-11-04

Family

ID=78331748

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/078787 WO2021218364A1 (zh) 2020-04-27 2021-03-02 一种图像增强方法及电子设备

Country Status (3)

Country Link
EP (1) EP4120183A4 (zh)
CN (1) CN113643186A (zh)
WO (1) WO2021218364A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113947553A (zh) * 2021-12-20 2022-01-18 山东信通电子股份有限公司 一种图像亮度增强方法及设备
CN116528060A (zh) * 2023-07-04 2023-08-01 长春希达电子技术有限公司 一种暗光图像增强器件、方法和装置以及电子设备
CN116668773A (zh) * 2022-11-22 2023-08-29 荣耀终端有限公司 增强视频画质的方法与电子设备

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114913063A (zh) * 2021-02-10 2022-08-16 京东方科技集团股份有限公司 一种图像处理方法和装置
CN116055778B (zh) * 2022-05-30 2023-11-21 荣耀终端有限公司 视频数据的处理方法、电子设备及可读存储介质
CN117059047B (zh) * 2023-10-12 2023-12-22 深圳市柯达科电子科技有限公司 一种用于lcd显示图像色彩智能调整方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160379340A1 (en) * 2015-06-23 2016-12-29 Hong Kong Applied Science and Technology Research Institute Company Limited Wavelet-based Image Decolorization and Enhancement
CN107527332A (zh) * 2017-10-12 2017-12-29 长春理工大学 基于改进Retinex的低照度图像色彩保持增强方法
CN107851309A (zh) * 2016-04-05 2018-03-27 华为技术有限公司 一种图像增强方法及装置
CN110378859A (zh) * 2019-07-29 2019-10-25 西南科技大学 一种新的高动态范围图像生成方法
CN110428382A (zh) * 2019-08-07 2019-11-08 杭州微帧信息科技有限公司 一种用于移动终端的高效视频增强方法、装置和存储介质
WO2020049567A1 (en) * 2018-09-06 2020-03-12 Carmel Haifa University Economic Corporation Ltd. Model-free physics-based reconstruction of images acquired in scattering media

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107302657B (zh) * 2017-05-16 2019-07-23 苏州大学 适用于物联网的图像采集系统
WO2019112085A1 (en) * 2017-12-06 2019-06-13 Korea Advanced Institute Of Science And Technology Method and apparatus for inverse tone mapping
US20190294931A1 (en) * 2018-03-26 2019-09-26 Artomatix Ltd. Systems and Methods for Generative Ensemble Networks

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160379340A1 (en) * 2015-06-23 2016-12-29 Hong Kong Applied Science and Technology Research Institute Company Limited Wavelet-based Image Decolorization and Enhancement
CN107851309A (zh) * 2016-04-05 2018-03-27 华为技术有限公司 一种图像增强方法及装置
CN107527332A (zh) * 2017-10-12 2017-12-29 长春理工大学 基于改进Retinex的低照度图像色彩保持增强方法
WO2020049567A1 (en) * 2018-09-06 2020-03-12 Carmel Haifa University Economic Corporation Ltd. Model-free physics-based reconstruction of images acquired in scattering media
CN110378859A (zh) * 2019-07-29 2019-10-25 西南科技大学 一种新的高动态范围图像生成方法
CN110428382A (zh) * 2019-08-07 2019-11-08 杭州微帧信息科技有限公司 一种用于移动终端的高效视频增强方法、装置和存储介质

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113947553A (zh) * 2021-12-20 2022-01-18 山东信通电子股份有限公司 一种图像亮度增强方法及设备
CN113947553B (zh) * 2021-12-20 2022-03-18 山东信通电子股份有限公司 一种图像亮度增强方法及设备
CN116668773A (zh) * 2022-11-22 2023-08-29 荣耀终端有限公司 增强视频画质的方法与电子设备
CN116668773B (zh) * 2022-11-22 2023-12-22 荣耀终端有限公司 增强视频画质的方法与电子设备
CN116528060A (zh) * 2023-07-04 2023-08-01 长春希达电子技术有限公司 一种暗光图像增强器件、方法和装置以及电子设备
CN116528060B (zh) * 2023-07-04 2023-09-19 长春希达电子技术有限公司 一种暗光图像增强器件、方法和装置以及电子设备

Also Published As

Publication number Publication date
CN113643186A (zh) 2021-11-12
EP4120183A1 (en) 2023-01-18
EP4120183A4 (en) 2023-06-21

Similar Documents

Publication Publication Date Title
WO2021218364A1 (zh) 一种图像增强方法及电子设备
WO2021249053A1 (zh) 图像处理的方法及相关装置
CN113726950B (zh) 一种图像处理方法和电子设备
CN111782879B (zh) 模型训练方法及装置
US20230043815A1 (en) Image Processing Method and Electronic Device
WO2022007862A1 (zh) 图像处理方法、系统、电子设备及计算机可读存储介质
CN113099146B (zh) 一种视频生成方法、装置及相关设备
US20230269324A1 (en) Display method applied to electronic device, graphical user interface, and electronic device
CN113705665B (zh) 图像变换网络模型的训练方法和电子设备
CN111612723B (zh) 图像修复方法及装置
EP4325877A1 (en) Photographing method and related device
WO2023093169A1 (zh) 拍摄的方法和电子设备
US20230377306A1 (en) Video Shooting Method and Electronic Device
WO2022156473A1 (zh) 一种播放视频的方法及电子设备
CN116916151B (zh) 拍摄方法、电子设备和存储介质
CN113538227B (zh) 一种基于语义分割的图像处理方法及相关设备
WO2022143314A1 (zh) 一种对象注册方法及装置
CN117957852A (zh) 用于减少的时延的相机初始化
CN114793283A (zh) 图像编码方法、图像解码方法、终端设备及可读存储介质
US20240184504A1 (en) Screen projection method and system, and related apparatus
EP4296845A1 (en) Screen projection method and system, and related apparatus
CN116700555B (zh) 动效处理方法及电子设备
WO2023241544A1 (zh) 组件预览的方法和电子设备
US20240064397A1 (en) Video Shooting Method and Electronic Device
CN115460343B (zh) 图像处理方法、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21795665

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021795665

Country of ref document: EP

Effective date: 20221013

NENP Non-entry into the national phase

Ref country code: DE