WO2021164731A1 - 图像增强方法以及图像增强装置 - Google Patents

图像增强方法以及图像增强装置 Download PDF

Info

Publication number
WO2021164731A1
WO2021164731A1 PCT/CN2021/076859 CN2021076859W WO2021164731A1 WO 2021164731 A1 WO2021164731 A1 WO 2021164731A1 CN 2021076859 W CN2021076859 W CN 2021076859W WO 2021164731 A1 WO2021164731 A1 WO 2021164731A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
resolution
processed
hdr
scale
Prior art date
Application number
PCT/CN2021/076859
Other languages
English (en)
French (fr)
Inventor
马翼鹏
汪涛
彭竞阳
宋风龙
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21757305.4A priority Critical patent/EP4105877A4/en
Publication of WO2021164731A1 publication Critical patent/WO2021164731A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G06T7/41—Analysis of texture based on statistical description of texture
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20208—High dynamic range [HDR] image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • This application relates to the field of artificial intelligence, and more specifically, to an image enhancement method and an image enhancement device in the computer vision field.
  • Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. To put it vividly, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data.
  • computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information.
  • the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.
  • Image enhancement can also be referred to as image quality enhancement. It is an important branch in the field of image processing. Image enhancement technology can improve image quality without re-collecting data to meet more practical application requirements.
  • image processing based on convolutional neural networks Convolutional Neural Networks, CNN
  • CNN convolutional Neural Networks
  • the deep learning-based super-division method usually only performs a complex super-division process on the brightness channel of the original resolution input image, and the color channel is usually simple up-sampling, which leads to the existence of the image enhancement method before and after the super-resolution processing. Inconsistent issues in color, brightness, contrast, saturation, etc. Therefore, in the case of meeting the real-time requirements of image enhancement, how to improve the image quality after enhanced image processing becomes an urgent problem to be solved.
  • the present application provides an image enhancement method and an image enhancement device, which can enhance the performance of the enhanced image in terms of color, brightness, contrast, saturation, etc., while meeting the real-time requirements of image enhancement processing, so as to improve The effect of image enhancement processing.
  • an image enhancement method including: acquiring a first high dynamic range HDR image corresponding to an image to be processed and a color image feature of the image to be processed, wherein the color image feature is used to indicate the Different brightness areas or different color change areas in the image to be processed, the image to be processed is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the Second resolution; input the first HDR image into a neural network model for super-resolution processing; image the first HDR image and the color image characteristics after the super-resolution processing through the neural network model The enhancement processing is performed to obtain a second HDR image corresponding to the image to be processed, where the second HDR image refers to an HDR image with a resolution of the first resolution.
  • the second HDR image refers to an image with the same resolution size as the image to be processed; where the same resolution size can mean that the image has the same number of pixels, and the image to be processed can be enhanced by image enhancement.
  • the pixel values of at least part of the pixels in the processed image are adjusted, so that the visual effect of the second HDR image is better than that of the image to be processed.
  • the above-mentioned image to be processed may refer to the image to be processed with image enhancement requirements; for example, the image to be processed may refer to the original image with higher resolution and poor image quality; for example, it may refer to the original image subject to weather, distance, shooting Due to factors such as environment, the acquired image to be processed has the problem of low image quality; low image quality includes but is not limited to: image blur, or image color, brightness, saturation, contrast, and dynamic range are poor.
  • the first HDR image may be an image obtained by down-sampling and HDR enhancement of the image to be processed; among them, the high dynamic range (HDR) image is compared with the standard dynamic range (standard dynamic range) image.
  • HDR high dynamic range
  • SDR standard dynamic range
  • color image features can also be referred to as color guide maps.
  • the color image features can provide higher guidance for difficult areas (for example, bright areas or dark areas) in the image to be processed in the input neural network model. Information, so that the neural network model can pay more attention to the enhancement effect of difficult areas in the learning process.
  • the first HDR image is the low-resolution enhanced image corresponding to the image to be processed, and the image to be processed is passed in the neural network model.
  • the color image characteristics of the first HDR image are super-resolution processed, so as to obtain an enhanced image with the same resolution size as the image to be processed, that is, the second HDR image; it can be introduced when super-resolution processing is performed on the first HDR image
  • the color attention mechanism that is, the color image characteristics, enables the neural network model to improve the color, brightness, contrast, and saturation of the difficult areas (too bright and dark areas) in the image to be processed during the image enhancement process. The recovery effect of the aspect, thereby improving the effect of image enhancement.
  • some implementations of the first aspect further include: acquiring a texture image feature of the image to be processed, wherein the texture image feature is used to indicate an edge area or a texture area of the image to be processed ;
  • the inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image according to the texture image feature through the neural network model.
  • the introduction of the texture attention mechanism in the neural network model is to enable the neural network model to learn details such as edges and textures in the image; the neural network model can be improved by using texture image features, also known as texture guide maps. For the learning of regions with higher weights in the texture guide map; thus, the process of super-division processing of the first HDR image can improve the ability of the super-division algorithm to recover image texture details, and avoid image blur or other sensory differences introduced after super-division processing.
  • the super-resolution reduction processing can be performed based on the dual attention mechanism, where the dual attention mechanism can include a texture attention mechanism and a color attention mechanism. ; That is, by using color image features and texture image features in the super-division processing of the first HDR image, the resulting enhanced image is the second HDR image of the image to be processed in terms of color, brightness, saturation, contrast, and texture
  • the dual attention mechanism can include a texture attention mechanism and a color attention mechanism.
  • some implementations of the first aspect further include: acquiring a multi-scale image feature of the image to be processed, wherein the multi-scale image feature is used to indicate the to-be-processed image at different scales. Processing image information of the image, any one of the multi-scale image features has a different scale size;
  • the inputting the first HDR image into a neural network model for super-resolution processing includes:
  • multi-scale image features can refer to image features of different resolutions.
  • the neural network model can introduce more image detail information, which is beneficial to ensure the restoration of the first HDR image in terms of details. .
  • multi-scale image features can be used to indicate the image information of the image to be processed in different scales, where different scales can refer to different resolutions, and image information can refer to high-frequency information in the image.
  • image information can refer to high-frequency information in the image.
  • the high-frequency information may include one or more of edge information, detail information, and texture information in the image.
  • the multi-scale image features of the image to be processed can be obtained, that is, the multi-scale feature information of the image to be processed can be extracted from the original resolution of the image to be processed by feature reduction; It can perform multi-scale super-division processing restoration based on the multi-scale features of the first HDR image, ie the low-resolution HDR image and the original input image to be processed, and the low-resolution HDR image and the extracted feature information of the original input image at the corresponding scale
  • the fusion may be performed to obtain the second HDR image of the image to be processed, that is, the enhanced image after the image to be processed is enhanced.
  • some implementations of the first aspect further include: acquiring a texture image feature of the image to be processed, wherein the texture image feature is used to indicate an edge area or a texture area of the image to be processed ;
  • the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;
  • the inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image according to the texture image feature and the multi-scale feature through the neural network model. Resolution processing.
  • both the dual attention mechanism and the multi-scale image feature can be introduced during the super-resolution processing of the first HDR image; among them, the dual attention mechanism can include the texture attention mechanism and the color attention mechanism.
  • Force mechanism namely color image features and texture features
  • the resulting enhanced image is the second HDR image of the image to be processed in terms of color, brightness, saturation, contrast, and texture details. It is the same as or close to the true value image, thereby improving the effect of image enhancement, that is, improving the image quality after image enhancement processing.
  • the performing super-resolution processing on the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model includes :
  • the image enhancement processing is performed on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain
  • the second HDR image corresponding to the image to be processed includes:
  • an image enhancement method is provided, which is applied to a smart screen (or an artificial intelligence screen), including: detecting a first operation that a user instructs to open a display mode interface of the smart screen; responding to the first operation , Displaying the display mode selection interface on the smart screen; detecting a second operation indicating the first display mode by the user; in response to the second operation, displaying an output image on the smart screen, the The output image is the second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed,
  • the neural network model is applied in the image enhancement processing process, and the acquired first high dynamic range HDR image of the image to be processed is input into the neural network model for super-division processing; the second HDR image passes through all
  • the neural network model is obtained by performing image enhancement processing on the color image features of the super-resolution processed first HDR image and the image to be processed; the image to be processed is an image of the first resolution, so The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution;
  • the color image feature is used to indicate areas of different brightness or areas of different color changes in the image to be processed.
  • the above-mentioned display mode interface may include, but is not limited to, the following display modes: HDR mode, Dolby mode, Ultra HD mode, Blu-ray mode, and Smart mode.
  • the aforementioned output image is a second HDR image obtained after image enhancement processing is performed on the image to be processed, that is, an HDR image with the same resolution and size as the image to be processed; wherein, the image to be processed may mean that the electronic device passes through
  • the image/video captured by the camera, or the above-mentioned image to be processed may also be an image obtained from the inside of the electronic device (for example, an image stored in an album of the electronic device, or a picture obtained by the electronic device from the cloud).
  • the above-mentioned image to be processed may be an original film source stored in a database, or may also refer to a film source that is played in real time.
  • the image enhancement method of the image to be processed may be an offline method executed in the cloud.
  • the cloud server may perform image enhancement of the image to be processed to obtain an output image, and display the output image on a smart screen.
  • the above-mentioned image enhancement method may be executed by a local device, that is, it may refer to an output image obtained by performing image enhancement of an image to be processed on a smart screen.
  • the first operation of the user instructing to open the display mode interface of the smart screen may include but is not limited to: instructing the smart screen to open the display mode interface through the control device, or may include the user instructing the smart screen through voice
  • the behavior of the screen opening the display mode interface may also include other behaviors of the user instructing the smart screen to open the display mode interface; the above is an example and does not limit the application in any way.
  • the above-mentioned second operation of the user instructing the first display mode may include but is not limited to: instructing the first display model in the display mode interface through the control device, or may include the user instructing the first display model in the display mode interface through voice Alternatively, it may also include other actions of the user instructing to select the first display mode in the display mode interface; the foregoing is an example and does not limit the application in any way.
  • the above-mentioned first display mode may refer to the HDR mode or other professional modes.
  • the image enhancement method provided in the embodiments of the present application can be implemented through the first display mode, thereby improving the image quality of the output image, so that users who use smart screens can get better Good visual experience.
  • the method further includes: acquiring a texture image feature of the image to be processed, wherein the texture image feature is used to indicate an edge area or a texture area of the image to be processed ;
  • the inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image according to the texture image feature through the neural network model.
  • the method further includes: acquiring a multi-scale image feature of the image to be processed, wherein the multi-scale image feature is used to indicate the to-be-processed image at different scales. Processing image information of the image, any one of the multi-scale image features has a different scale size;
  • the method further includes: acquiring a texture image feature of the image to be processed, wherein the texture image feature is used to indicate an edge area or a texture area of the image to be processed ;
  • the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;
  • the inputting the first HDR image into the neural network model for super-resolution processing includes:
  • multi-scale image features can be used to indicate image information of an image to be processed at different scales, where different scales can refer to different resolutions, and image information can refer to high-frequency information in the image; for example, The high-frequency information may include one or more of edge information, detail information, and texture information in the image.
  • the super-resolution processing of the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model includes :
  • the image enhancement processing is performed on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain
  • the second HDR image corresponding to the image to be processed includes:
  • an image enhancement method is provided, which is applied to an electronic device with a display screen and a camera, including: detecting a second operation of the user instructing the camera; in response to the second operation, displaying on the display screen Output an image, or save an output image in the electronic device, the output image is the second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed, wherein the neural network model is applied to the image enhancement processing
  • the acquired first high dynamic range HDR image of the image to be processed is input into the neural network model for super-resolution processing
  • the second HDR image is processed by the neural network model after the super-resolution processing
  • the first HDR image and the color image characteristics of the image to be processed are obtained by image enhancement processing
  • the image to be processed is an image with a first resolution
  • the first HDR image is an image with a second resolution
  • the first resolution is greater than the second resolution
  • the second HDR image refers to an HDR image with a resolution of the first resolution
  • the color image feature is used
  • it may further include: detecting a first operation for turning on the camera by the user; in response to the first operation, displaying a shooting interface on the display screen, the shooting interface including a viewfinder, so The frame includes the image to be processed.
  • the foregoing specific process of performing feature enhancement processing on the image to be processed may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.
  • the image enhancement method provided by the embodiment of this application can be applied to the camera field of smart terminals.
  • the image enhancement method of the embodiment of this application can perform image enhancement processing on the output image in terms of color, brightness, and brightness. Contrast, saturation and other aspects of performance can be improved.
  • image enhancement processing is performed on the acquired original image, and the output image after the image enhancement processing is displayed on the screen of the smart terminal, or it can also be performed by performing image processing on the acquired original image. Enhancement processing, save the output image after image enhancement processing to the album of the smart terminal.
  • an image enhancement method including: acquiring a road image to be processed and a first high dynamic range HDR image corresponding to the road image, wherein the road image is an image with a first resolution, and The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the road image is input into a neural network model to obtain the color image characteristics of the road screen, wherein the The color image feature is used to indicate different brightness areas or different color change areas in the road image; the first HDR image is input to the neural network model for super-resolution processing; and the super-resolution processing is performed by the neural network model.
  • the first HDR image after resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the road image, where the second HDR image is an HDR image of the first resolution ; According to the second HDR image, identify road information in the second HDR image.
  • the above-mentioned specific process of performing feature enhancement processing on the road image can be obtained according to the above-mentioned first aspect and any one of the implementation manners of the first aspect.
  • the image enhancement method provided in the embodiment of the present application may be applied to the field of automatic driving.
  • it can be applied to the navigation system of an autonomous vehicle.
  • the image enhancement method in this application can enable the autonomous vehicle to perform image enhancement processing through the acquired original road images with lower image quality during the navigation process of the autonomous vehicle while driving on the road. Obtain the enhanced road image, so as to realize the safety of self-driving vehicles.
  • an image enhancement method including: acquiring a street view image to be processed and a first high dynamic range HDR image corresponding to the street view image, wherein the street view image is an image with a first resolution, so The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the street view image is input into a neural network model to obtain the color image characteristics of the street view image, wherein the The color image features are used to indicate areas of different brightness or areas of different color changes in the street view image; input the first HDR image into the neural network model for super-resolution processing; use the neural network model to perform super-resolution processing; The first HDR image after resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the street view image, where the second HDR image is an HDR image of the first resolution ; According to the second HDR image, identify street view information in the second HDR image.
  • the foregoing specific process of performing feature enhancement processing on the street view image may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.
  • the image enhancement method provided in the embodiment of the present application can be applied to the security field.
  • the image enhancement method of the embodiment of the present application can be applied to the surveillance image enhancement of a safe city.
  • the image (or video) collected by the surveillance equipment in public places is often affected by factors such as weather and distance, and the image is blurred. Problems such as low image quality.
  • the image enhancement method of the present application can perform image enhancement on the collected original images, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information can be provided for case detection.
  • an image enhancement device which includes a module/unit for executing the image enhancement method in any one of the foregoing first to fifth aspects and the first to fifth aspects.
  • an image enhancement device including: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute: The first high dynamic range HDR image corresponding to the processed image and the color image feature of the image to be processed, wherein the color image feature is used to indicate areas of different brightness or areas of different color changes in the image to be processed, and
  • the image to be processed is an image of a first resolution
  • the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution
  • the first HDR image is input to the neural network model Performing super-resolution processing; performing image enhancement processing on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain a second HDR image corresponding to the image to be processed,
  • the second HDR image refers to an HDR image with a resolution of the first resolution.
  • the processor included in the foregoing image enhancement device is also used in the image enhancement method in any one of the foregoing first aspect to the fifth aspect and the first aspect to the fifth aspect.
  • a computer-readable medium stores program code for device execution, and the program code includes the program code for executing the first aspect to the fifth aspect and the first aspect to the fifth aspect.
  • the image enhancement method in any one of the implementations.
  • a computer program product containing instructions is provided.
  • the computer program product runs on a computer, the computer executes any one of the first aspect to the fifth aspect and the first aspect to the fifth aspect.
  • the image enhancement method in the implementation mode.
  • a chip in a tenth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes the first to fifth aspects and the first aspect. To the image enhancement method in any one of the implementation manners of the fifth aspect.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the image enhancement method in any one of the foregoing first aspect to the fifth aspect and the first aspect to the fifth aspect.
  • FIG. 1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of yet another application scenario provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a convolutional neural network structure provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 10 is a schematic diagram of image enhancement processing provided by an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a system architecture of an image enhancement method provided by an embodiment of the present application.
  • FIG. 12 is a schematic flowchart of HDR correction through a color guide diagram provided by an embodiment of the present application.
  • FIG. 13 is a schematic diagram of the system architecture of another image enhancement method provided by an embodiment of the present application.
  • FIG. 14 is a schematic diagram of an extraction process of a texture guide map provided by an embodiment of the present application.
  • 15 is a schematic diagram of the system architecture of still another image enhancement method provided by an embodiment of the present application.
  • FIG. 16 is a schematic flowchart of a self-guided multi-level super-division processing provided by an embodiment of the present application.
  • FIG. 17 is a schematic diagram of HDR correction through a color guide map after introducing multi-scale image features according to an embodiment of the present application.
  • FIG. 18 is a schematic diagram of an evaluation result of image enhancement quality provided by an embodiment of the present application.
  • FIG. 19 is a schematic diagram of an evaluation result of image enhancement quality provided by an embodiment of the present application.
  • FIG. 20 is a schematic diagram of an evaluation result of image enhancement quality provided by an embodiment of the present application.
  • FIG. 21 is a schematic block diagram of an image enhancement device provided by an embodiment of the present application.
  • FIG. 22 is a schematic diagram of the hardware structure of an image enhancement device provided by an embodiment of the present application.
  • the images in the embodiments of the present application may be static images (or referred to as static pictures) or dynamic images (or referred to as dynamic pictures).
  • the images in the present application may be videos or dynamic pictures, or the present application
  • the images in can also be static pictures or photos.
  • static images or dynamic images are collectively referred to as images.
  • Figure 1 shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is suitable for general artificial intelligence field requirements.
  • Intelligent Information Chain reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensing process of "data-information-knowledge-wisdom".
  • the infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the basic platform.
  • the infrastructure can communicate with the outside through sensors, and the computing power of the infrastructure can be provided by smart chips.
  • the smart chip here can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), and an application specific integrated circuit (application specific).
  • Hardware acceleration chips such as integrated circuit (ASIC) and field programmable gate array (FPGA).
  • the basic platform of infrastructure can include distributed computing framework and network related platform guarantee and support, and can include cloud storage and computing, interconnection network, etc.
  • data can be obtained through sensors and external communication, and then these data can be provided to the smart chip in the distributed computing system provided by the basic platform for calculation.
  • the data in the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence.
  • the data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • the above-mentioned data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.
  • machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, training, etc.
  • Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, using formal information to carry out machine thinking and solving problems according to the reasoning control strategy.
  • the typical function is search and matching.
  • Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, ranking, and prediction.
  • some general capabilities can be formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image Recognition and so on.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is an encapsulation of the overall solution of artificial intelligence, productizing intelligent information decision-making and realizing landing applications. Its application fields mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical, smart security, autonomous driving, safe city, smart terminal, etc.
  • Fig. 2 is a schematic diagram of an application scenario of an image enhancement method provided by an embodiment of the present application.
  • the image enhancement method in the embodiments of the present application can perform image enhancement processing on the input image (or input video) to obtain the input image (or input video).
  • the above-mentioned smart terminal may be mobile or fixed.
  • the smart terminal may be a mobile phone with image enhancement function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer, LC), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, augmented reality (AR)/virtual reality (VR), wearable Wearable device (WD) or an autonomous vehicle, etc., which are not limited in the embodiment of the present application.
  • image enhancement can also be referred to as image quality enhancement. Specifically, it can refer to processing the brightness, color, contrast, saturation, and/or dynamic range of the image to make the image
  • the indicators meet the preset conditions or preset indicators.
  • image enhancement and image quality enhancement have the same meaning.
  • Application scenario 1 Smart screen (AI screen)
  • the image enhancement method of the embodiment of the present application can also be applied to the field of smart screens; among them, smart screens are separated from traditional TVs and break through the limitations of large-screen categories.
  • smart screens are separated from traditional TVs and break through the limitations of large-screen categories.
  • mobile phones and smart screens will become the dual centers of users' smart lives.
  • Mobile phones will still be the user's personal center, while smart screens may become the family's emotional center.
  • Smart screens will take on more roles in the family, not only the family's audio-visual entertainment center, but also the information sharing center, control management center and multi-device interactive center.
  • the original source of the video can be processed by using the image enhancement method of the embodiment of the present application to enhance the picture of the source. Quality and get a better visual perception.
  • the image enhancement method of the embodiment of the application can be used to image the source of the old movie.
  • Enhanced processing can show the visual sense of modern movies.
  • the source of an old movie can be enhanced to a high-dynamic range (HDR) 10 or a high-quality video of the Dolby Vision (Dolby Vision) standard through the image enhancement method of the embodiment of the present application.
  • HDR high-dynamic range
  • Dolby Vision Dolby Vision
  • the present application provides an image enhancement method, including: acquiring an image to be processed (for example, the original film source of a movie, or an online film source) and a first high dynamic range HDR image corresponding to the image to be processed,
  • the image to be processed is an image of a first resolution
  • the first HDR image is an image of a second resolution
  • the first resolution is greater than the second resolution
  • the image to be processed is input
  • the neural network model obtains the color image features of the image to be processed, where the color image features are used to indicate areas of different brightness or areas of different color changes in the image to be processed
  • the first HDR image is input to the
  • the neural network model performs super-resolution processing; the super-resolution processed first HDR image and the color image feature are processed through the neural network model to obtain the second HDR corresponding to the image to be processed
  • An image for example, a film source with improved image quality
  • the second HDR image is an HDR image of the first resolution.
  • the present application provides an image enhancement method applied to a smart screen (or artificial intelligence screen), including: detecting a first operation instructed by a user to open the display mode interface of the smart screen; responding to the The first operation, the display mode selection interface is displayed on the smart screen; the second operation indicating the first display mode by the user is detected; in response to the second operation, the output image is displayed on the smart screen
  • the output image is a second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed, wherein the neural network model is applied in the image enhancement processing process, and the acquired first image of the image to be processed is
  • a high dynamic range HDR image is input to the neural network model for super-division processing; the second HDR image is processed by the neural network model on the super-resolution processed first HDR image and the to-be-processed image
  • the color image feature of the image is obtained by image enhancement processing; the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first
  • the above-mentioned display mode interface may include, but is not limited to, the following display modes: HDR mode, Dolby mode, Ultra HD mode, Blu-ray mode, and Smart mode.
  • the aforementioned output image is a second HDR image obtained after image enhancement processing is performed on the image to be processed, that is, an HDR image with the same resolution and size as the image to be processed; wherein, the image to be processed may mean that the electronic device passes through
  • the image/video captured by the camera, or the above-mentioned image to be processed may also be an image obtained from the inside of the electronic device (for example, an image stored in an album of the electronic device, or a picture obtained by the electronic device from the cloud).
  • the above-mentioned image to be processed may be an original film source stored in a database, or may also refer to a film source that is played in real time.
  • the image enhancement method of the image to be processed may be an offline method executed in the cloud.
  • the cloud server may perform image enhancement of the image to be processed to obtain an output image, and display the output image on a smart screen.
  • the above-mentioned image enhancement method may be executed by a local device, that is, it may refer to an output image obtained by performing image enhancement of an image to be processed on a smart screen.
  • the first operation of the user instructing to open the display mode interface of the smart screen may include, but is not limited to: instructing the smart screen to open the display mode interface through the control device, and may also include the user instructing the smart screen to open the display mode interface through voice.
  • the behavior, or, may also include other behaviors of the user instructing the smart screen to open the display mode interface; the above is an example and does not limit the application in any way.
  • the above-mentioned second operation of the user instructing the first display mode may include but is not limited to: instructing the first display model in the display mode interface through the control device, or may include the user instructing the first display model in the display mode interface through voice Alternatively, it may also include other actions of the user instructing to select the first display mode in the display mode interface; the foregoing is an example and does not limit the application in any way.
  • the above-mentioned first display mode may refer to the HDR mode or other professional modes.
  • the image enhancement method provided in the embodiments of the present application can be implemented through the first display mode, thereby improving the image quality of the output image, so that users who use smart screens can get better Good visual experience. It should be noted that the image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the related embodiments in FIG. 6 to FIG. 17, which will not be repeated here.
  • the image enhancement method of the embodiment of the present application can be applied to the shooting of a smart terminal device (for example, a mobile phone).
  • the image enhancement method in the embodiments of the present application can perform image enhancement processing on the acquired original image (or video) of poor quality to obtain an output image (or output video) with improved image quality.
  • the image enhancement method in the embodiments of the present application may be used to perform image enhancement processing on the acquired original image when the smart terminal is taking pictures in real time, and the output image after the image enhancement processing is displayed on the screen of the smart terminal.
  • the image enhancement method of the embodiment of the present application may be used to perform image enhancement processing on the acquired original image, and the output image after the image enhancement processing can be saved in the album of the smart terminal.
  • the present application proposes an image enhancement method, which is applied to an electronic device with a display screen and a camera, including: detecting a second operation of the user instructing the camera; in response to the second operation, in the display screen The output image is displayed internally, or the output image is saved in the electronic device.
  • the output image is a second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed, wherein the neural network model is applied to all
  • the acquired first high dynamic range HDR image of the image to be processed is input to the neural network model for super-division processing;
  • the second HDR image is processed by the neural network model on the super
  • the first HDR image after resolution processing and the color image characteristics of the image to be processed are obtained by image enhancement processing;
  • the image to be processed is an image with a first resolution, and the first HDR image is a second Resolution image, the first resolution is greater than the second resolution;
  • the second HDR image refers to an HDR image with a resolution of the first resolution;
  • the color image feature is used to indicate the The different brightness areas or different color change areas in the image to be processed.
  • it may further include: detecting a first operation for turning on the camera by the user; in response to the first operation, displaying a shooting interface on the display screen, the shooting interface including a viewfinder, so The frame includes the image to be processed.
  • image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the related embodiments in FIG. 6 to FIG. 17, which will not be repeated here.
  • the image enhancement method of the embodiment of the present application can be applied to the field of automatic driving.
  • it can be applied to the navigation system of an autonomous vehicle.
  • the image enhancement method in this application can enable the autonomous vehicle to obtain low-quality original road images (or original road video ) Perform image enhancement processing to obtain an enhanced road image (or road video), so as to realize the safety of autonomous vehicles.
  • the present application provides an image enhancement method, including: acquiring a road image to be processed and a first high dynamic range HDR image corresponding to the road image, where the road image is an image with a first resolution , The first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution;
  • the road image is input into the neural network model to obtain the color image feature of the road screen, wherein the color image feature is used to indicate different brightness areas or different color change areas in the road image; and the first HDR
  • the image is input to the neural network model for super-resolution processing; the super-resolution processed first HDR image and the color image feature are processed through the neural network model to obtain the corresponding road image A second HDR image, where the second HDR image is an HDR image of the first resolution; and the road information in the second HDR image is identified according to the second HDR image.
  • image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the related embodiments in FIG. 6 to FIG. 17, which will not be repeated here.
  • the image enhancement method of the embodiment of the present application can be applied to the field of safe cities, for example, the field of security.
  • the image enhancement method of the embodiment of the present application can be applied to the surveillance image enhancement of a safe city.
  • the image (or video) collected by the surveillance equipment in public places is often affected by factors such as weather and distance, and the image is blurred. Problems such as low image quality.
  • the image enhancement method of the present application can perform image enhancement on the collected pictures, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information can be provided for case detection.
  • the present application provides an image enhancement method, including: acquiring a street view image to be processed and a first high dynamic range HDR image corresponding to the street view image, wherein the street view image is an image with a first resolution
  • the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution
  • the street view image is input into a neural network model to obtain the color image characteristics of the street view image, wherein, The color image features are used to indicate areas of different brightness or areas of different color changes in the street view image
  • the first HDR image after the super-resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the street view image, wherein the second HDR image is of the first resolution HDR image; according to the second HDR image, identifying street view information in the second HDR image.
  • image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the related embodiments in FIG. 6 to FIG. 17, which will not be repeated here.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • the DNN is divided according to the positions of different layers.
  • the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated as far as the work of each layer is concerned. Simply put, it is the following linear relationship expression: in, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • DNN The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way of extracting image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.
  • the neural network can use the back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged.
  • the back-propagation algorithm is a back-propagation motion dominated by error loss, and aims to obtain the optimal parameters of the neural network model, such as the weight matrix.
  • FIG. 6 shows a system architecture 200 provided by an embodiment of the present application.
  • a data collection device 260 is used to collect training data.
  • the image enhancement model also referred to as an image enhancement network
  • the training data collected by the data collection device 260 can be training images.
  • the training data of the training image enhancement model in the embodiment of the present application may include original images and sample enhanced images.
  • the original image may refer to an image with lower image quality
  • the sample enhanced image may refer to an image with higher image quality.
  • it may refer to one or more of the brightness, color, details, etc.
  • image enhancement can also be referred to as image quality enhancement, which can specifically refer to processing the brightness, color, contrast, saturation, and/or dynamic range of an image, so that various indicators of the image meet preset conditions Or preset indicators.
  • image enhancement and image quality enhancement have the same meaning.
  • the data collection device 260 stores the training data in the database 230, and the training device 220 trains based on the training data maintained in the database 230 to obtain the target model/rule 201 (that is, the image enhancement model in the embodiment of the present application) .
  • the training device 220 inputs the training data into the image enhancement model until the difference between the predicted enhanced image output by the training image enhancement model and the sample enhanced image meets a preset condition (for example, the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold, Or, the difference between the predicted enhanced image and the sample enhanced image remains unchanged or no longer decreases), thereby completing the training of the target model/rule 201.
  • a preset condition for example, the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold, Or, the difference between the predicted enhanced image and the sample enhanced image remains unchanged or no longer decreases
  • the image enhancement model used to perform the image enhancement method in the embodiment of the present application can realize end-to-end training.
  • the image enhancement model can enhance the image (for example, the true value image) by the sample corresponding to the input image and the input image. Achieve end-to-end training.
  • the target model/rule 201 is obtained by training an image enhancement model. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the collection of the data collection device 260, and may also be received from other devices.
  • the training device 220 does not necessarily perform the training of the target model/rule 201 completely based on the training data maintained by the database 230. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of the embodiment. It should also be noted that at least part of the training data maintained in the database 230 may also be used to execute the process of the processing to be processed by the execution device 210.
  • the target model/rule 201 trained according to the training device 220 can be applied to different systems or devices, such as the execution device 210 shown in FIG. 6, the execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 210 is configured with an input/output (input/output, I/O) interface 212 for data interaction with external devices.
  • the user can input data to the I/O interface 212 through the client device 240.
  • the input data in this embodiment of the present application may include: a to-be-processed image input by the client device.
  • the preprocessing module 213 and the preprocessing module 214 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 212.
  • the preprocessing module 213 and the preprocessing module may not be provided. 214 (there may only be one preprocessing module), and the calculation module 211 is directly used to process the input data.
  • the execution device 210 When the execution device 210 preprocesses input data, or when the calculation module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call data, codes, etc. in the data storage system 250 for corresponding processing. , The data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 250.
  • the I/O interface 212 returns the processing result to the image-enhanced image to be processed as described above, and returns the resulting output image to the client device 240 to provide it to the user.
  • the training device 220 can generate corresponding target models/rules 201 based on different training data for different goals or tasks, and the corresponding target models/rules 201 can be used to achieve the above goals or complete The above tasks provide users with the desired results.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 212.
  • the client device 240 can automatically send input data to the I/O interface 212. If the client device 240 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 240. The user can view the result output by the execution device 210 on the client device 240, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 240 can also be used as a data collection terminal to collect the input data of the input I/O interface 212 and the output result of the output I/O interface 212 as new sample data, and store it in the database 230 as shown in the figure. Of course, it is also possible not to collect through the client device 240. Instead, the I/O interface 212 directly uses the input data input to the I/O interface 212 and the output result of the output I/O interface 212 as a new sample as shown in the figure. The data is stored in the database 230.
  • FIG. 6 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data The storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210.
  • the target model/rule 201 is trained according to the training device 220.
  • the target model/rule 201 may be an image enhancement model in the embodiment of the present application.
  • the image enhancement model provided in the embodiment of the present application may be Deep neural network, convolutional neural network, or, it can be deep convolutional neural network, etc.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • the deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of.
  • the convolutional neural network is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.
  • the convolutional neural network 300 may include an input layer 310, a convolutional layer/pooling layer 320 (wherein the pooling layer is optional), a fully connected layer 330 and an output layer 340.
  • the input layer 310 can obtain the image to be processed, and pass the obtained image to be processed to the convolutional layer/pooling layer 320 and the fully connected layer 330 for processing, and the processing result of the image can be obtained.
  • the convolutional layer/pooling layer 320 may include layers 321-326, for example: in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, and layer 323 is a convolutional layer. Layers, 324 is a pooling layer, 325 is a convolutional layer, and 326 is a pooling layer; in another implementation, 321 and 322 are convolutional layers, 323 is a pooling layer, and 324 and 325 are convolutional layers. Layer, 326 is the pooling layer, that is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 321 can include many convolution operators.
  • the convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. And so on, it depends on the value of stride) to complete the work of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract the edge information of the image, another weight matrix is used to extract the specific color of the image, and the other weight matrix is used to correct the unwanted images in the image. The noise is blurred and so on.
  • the multiple weight matrices have the same size (row ⁇ column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are merged to form The output of the convolution operation.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions. .
  • the initial convolutional layer (such as 321) often extracts more general features, and the general features can also be called low-level features; with the convolutional neural network With the 300-depth deepening, the features extracted by the subsequent convolutional layer (for example, 326) become more and more complex, for example, features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.
  • the pooling layer can be a convolutional layer followed by a layer.
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the purpose of the pooling layer is to reduce the size of the image space.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain an image with a smaller size.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of the average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 300 After processing by the convolutional layer/pooling layer 320, the convolutional neural network 300 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 300 needs to use the fully connected layer 330 to generate one or a group of required classes of output. Therefore, the fully connected layer 330 may include multiple hidden layers (331, 332 to 33n as shown in FIG. 7) and an output layer 340. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image enhancement, image recognition, image classification, image detection, and image super-resolution reconstruction, etc.
  • the output layer 340 After the multiple hidden layers in the fully connected layer 330, that is, the final layer of the entire convolutional neural network 300 is the output layer 340.
  • the output layer 340 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network shown in FIG. 7 is only used as an example of the structure of the image enhancement model of the embodiment of the present application.
  • the convolutional neural network used in the image enhancement method of the embodiment of the present application It can also exist in the form of other network models.
  • the image enhancement device may include the convolutional neural network 300 shown in FIG. 7, and the image enhancement device may perform image enhancement processing on the image to be processed to obtain a processed output image.
  • FIG. 8 is a hardware structure of a chip provided by an embodiment of the present application.
  • the chip includes a neural network processor 400 (neural-network processing unit, NPU).
  • the chip can be set in the execution device 210 as shown in FIG. 6 to complete the calculation work of the calculation module 211.
  • the chip can also be set in the training device 220 shown in FIG. 6 to complete the training work of the training device 220 and output the target model/rule 201.
  • the algorithms of each layer in the convolutional neural network as shown in FIG. 7 can be implemented in the chip as shown in FIG. 8.
  • the NPU 400 is mounted on a main central processing unit (CPU) as a coprocessor, and the main CPU allocates tasks.
  • the core part of the NPU 400 is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 403 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.
  • the arithmetic circuit 403 fetches the data corresponding to matrix B from the weight memory 402 and caches it on each PE in the arithmetic circuit 403.
  • the arithmetic circuit 403 fetches the matrix A data and matrix B from the input memory 401 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 408 (accumulator).
  • the vector calculation unit 407 can perform further processing on the output of the arithmetic circuit 403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 407 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 407 can store the processed output vector to the unified memory 406.
  • the vector calculation unit 407 may apply a nonlinear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 407 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 403, for example for use in subsequent layers in a neural network.
  • the unified memory 406 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller 405 (direct memory access controller, DMAC) to store the input data in the external memory into the input memory 401 and/or unified memory 406, and the weight data in the external memory into the weight memory 402 , And store the data in the unified memory 406 into the external memory.
  • DMAC direct memory access controller
  • the bus interface unit 410 (bus interface unit, BIU) is used to implement interaction between the main CPU, the DMAC, and the fetch memory 409 through the bus.
  • An instruction fetch buffer 409 (instruction fetch buffer) connected to the controller 404 is used to store instructions used by the controller 404.
  • the controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.
  • the unified memory 406, the input memory 401, the weight memory 402, and the instruction fetch memory 409 are all on-chip (On-Chip) memories.
  • the external memory is a memory external to the NPU.
  • the external memory can be a double data rate synchronous dynamic random access memory.
  • Memory double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • each layer in the convolutional neural network shown in FIG. 7 may be executed by the arithmetic circuit 403 or the vector calculation unit 407.
  • the execution device 210 in FIG. 6 introduced above can execute each step of the image enhancement method of the embodiment of the present application.
  • the CNN model shown in FIG. 7 and the chip shown in FIG. 8 can also be used to execute the image of the embodiment of the present application. Steps of the enhancement method.
  • FIG. 9 shows a system architecture 500 provided by an embodiment of the present application.
  • the system architecture includes a local device 520, a local device 530, an execution device 510, and a data storage system 550.
  • the local device 520 and the local device 530 are connected to the execution device 510 through a communication network.
  • the execution device 510 may be implemented by one or more servers.
  • the execution device 510 can be used in conjunction with other computing devices.
  • data storage for example: data storage, routers, load balancers and other equipment.
  • the execution device 510 may be arranged on one physical site or distributed on multiple physical sites.
  • the execution device 510 may use the data in the data storage system 550 or call the program code in the data storage system 550 to implement the image enhancement method of the embodiment of the present application.
  • execution device 510 may also be referred to as a cloud device, and in this case, the execution device 510 may be deployed in the cloud.
  • the execution device 510 may perform the following process: acquire a to-be-processed image and a first high dynamic range HDR image corresponding to the to-be-processed image, where the to-be-processed image is an image with a first resolution, and the first The HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the image to be processed is input into a neural network model to obtain the color image characteristics of the image to be processed, wherein the color Image features are used to indicate areas of different brightness or areas of different color changes in the image to be processed; input the first HDR image into the neural network model for super-resolution processing; and perform super-resolution processing on the neural network model through the neural network model.
  • the first HDR image after resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the image to be processed, where the second HDR image is an HDR of the first resolution image.
  • the image enhancement method of the embodiment of the present application may be an offline method executed in the cloud.
  • the image enhancement method of the embodiment of the present application may be executed by the execution device 510 described above.
  • the image enhancement method in the embodiment of the present application may be executed by the local device 520 or the local device 530.
  • image enhancement can be performed on the acquired image to be processed with poor image quality, so as to obtain the output image whose performance of the image to be processed is improved in terms of image details, image color, and image brightness, that is, the second image. HDR image.
  • the user may operate respective user devices (for example, the local device 520 and the local device 530) to interact with the execution device 510.
  • Each local device can represent any computing device, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.
  • Each user's local device can interact with the execution device 510 through a communication network of any communication mechanism/communication standard.
  • the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.
  • the local device 520 and the local device 530 can obtain the relevant parameters of the aforementioned neural network model from the execution device 510, deploy the neural network model on the local device 520 and the local device 530, and use the neural network model to perform Image enhancement processing, etc.
  • the neural network model can be directly deployed on the execution device 510, and the execution device 510 obtains the image to be processed and the first HDR image corresponding to the image to be processed from the local device 520 and the local device 530, and according to the neural network model Perform image enhancement processing to obtain a second HDR image corresponding to the image to be processed.
  • the aforementioned neural network model may be the superdivision network in the embodiment of the present application, for example, refer to the superdivision network 720 in the subsequent FIG. 11, FIG. 13 and FIG.
  • a "down-sampling+HDR+super-resolution” method is proposed to save computational overhead, so as to achieve the purpose of real-time image quality enhancement.
  • the original resolution image and the low resolution image enhanced image can be input to the pre-trained convolutional neural network, so that the image is super-divided (that is, the original resolution of the image is restored), and the output original resolution enhanced image is obtained.
  • High-resolution image enhancement is performed in the above manner.
  • the embodiments of the present application provide an image enhancement method and an image enhancement device, by acquiring a to-be-processed image and a first HDR image of the to-be-processed image, where the first HDR image is the low resolution corresponding to the to-be-processed image
  • the first HDR image is super-resolution processed by the color image characteristics of the image to be processed in the neural network model, thereby obtaining an enhanced image with the same resolution as the image to be processed, that is, the second HDR image;
  • a color attention mechanism is introduced on the basis of "downsampling + HDR + super resolution", that is, color image characteristics, so that in the process of image enhancement, the neural network model is improved for difficult areas in the image to be processed (over Bright and dark areas) restore effects in terms of color, brightness, contrast, and saturation, thereby improving the effect of image enhancement.
  • FIG. 10 shows a schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • the image enhancement method shown in FIG. 10 may be executed by an image enhancement apparatus, which may be the execution device 210 in FIG. It may be the execution device 510 in FIG. 9 or a local device.
  • the method shown in FIG. 10 includes steps 610 to 630, and steps 610 to 630 are respectively described in detail below.
  • Step 610 Acquire the first high dynamic range HDR image corresponding to the image to be processed and the color image characteristics of the image to be processed.
  • the color image feature is used to indicate different brightness areas or different color change areas in the image to be processed
  • the image to be processed is an image with a first resolution
  • the first HDR image is an image with a second resolution
  • the first resolution is greater than the second resolution
  • the above-mentioned image to be processed may refer to the image to be processed with image enhancement requirements; for example, the image to be processed may refer to the original image with higher resolution and poor image quality; for example, it may refer to the original image subject to weather, distance, shooting Due to the influence of environment and other factors, the acquired image to be processed has the problem of low image quality; low image quality includes but is not limited to: low image quality, including but not limited to image blur, or image color, brightness, saturation Degree, contrast, poor dynamic range, etc.
  • the above-mentioned image to be processed may be an image captured by an electronic device through a camera, or the above-mentioned image to be processed may also be an image obtained from the inside of the electronic device (for example, an image stored in an album of an electronic device, or an electronic device from Images obtained from the cloud).
  • the electronic device may be any one of the local device or the execution device shown in FIG. 9.
  • the first HDR image may be an image obtained by down-sampling and HDR enhancement of the image to be processed; among them, a high dynamic range (high dynamic range, HDR) image and a standard dynamic range (standard dynamic range, SDR) Compared with the image, it can provide more dynamic range and image color, that is, more image details can be included in the HDR image.
  • HDR high dynamic range
  • SDR standard dynamic range
  • the color image characteristics of the image to be processed can be acquired through target learning or traditional methods.
  • a target learning method or a traditional method may be used to process the image to be processed to obtain the color image characteristics of the image to be processed.
  • the core of the target learning method is to provide such a learning target so that the difference between the input image and the true value image (learning target) can be measured.
  • Such difference is usually reflected in the bright area and shadow (dark area) area of the image.
  • the color input image can be converted to a grayscale image or converted to a Other color domains of the luminance channel (such as YUV domain, LAB domain), and then set the lowest threshold and highest threshold, the channels (such as grayscale, Y channel in YUV domain, L channel in LAB domain) are lower and higher than the lowest threshold
  • the part with the highest threshold is regarded as the area with larger weight, and the other areas are set as the area with lower weight.
  • the color image characteristics of the image to be processed can be obtained through the self-encoding and decoding sub-network of the image to be processed.
  • the image to be processed can be down-sampled first to obtain a low-resolution image, and the low-resolution image corresponding to the image to be processed can be
  • the resolution image and the self-encoding and decoding sub-network obtain the color image characteristics; for example, refer to the following figure 12 to obtain the color image characteristics of the image to be processed through the low-resolution image of the image to be processed.
  • color image features can also be referred to as color guide maps.
  • the color image features can provide higher guidance for difficult areas (for example, bright areas or dark areas) in the image to be processed in the input neural network model. Information, so that the neural network model can pay more attention to the enhancement effect of difficult areas in the learning process.
  • Step 620 Input the first HDR image into the neural network model for super-resolution processing.
  • the above-mentioned super-resolution processing may refer to a convolution operation and an up-sampling operation, so that the resolution of the first HDR image is the same as the resolution of the image to be processed.
  • a texture attention mechanism can be introduced when performing super-resolution processing on the first HDR image.
  • the introduction of the texture attention mechanism in the neural network model is to enable the neural network model to learn details such as edges and textures in the image; the neural network model can be improved by using texture image features, also known as texture guide maps. For the learning of regions with higher weights in the texture guide map; thus, the process of super-division processing of the first HDR image can improve the ability of the super-division algorithm to recover image texture details, and avoid image blur or other sensory differences introduced after super-division processing.
  • multi-scale image features of the image to be processed can also be introduced; among them, multi-scale image features can refer to image features of different resolutions, by introducing multi-scale image features.
  • Features can make the neural network model introduce more image detail information, which helps to ensure the restoration of the first HDR image in terms of details.
  • super-resolution reduction processing can be performed based on a dual attention mechanism, where the dual attention mechanism can include a texture attention mechanism and a color attention mechanism .
  • the final enhanced image that is, the second HDR image of the image to be processed, is more realistic in terms of color, brightness, saturation, contrast, and texture details.
  • the value images are the same or close, which improves the effect of image enhancement.
  • the image to be processed can be input into the neural network model to obtain the multi-scale image features of the image to be processed, where any one of the multi-scale image features has a different scale;
  • the neural network model performs super-resolution processing on the first HDR image according to the texture image feature and the multi-scale image feature, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales.
  • FIG. 16 refer to the schematic flowchart of the self-guided multi-level super-division processing shown in FIG. 16 below.
  • multi-scale image features can be used to indicate image information of an image to be processed at different scales, where different scales can refer to different resolutions, and image information can refer to high-frequency information in the image; for example, The high-frequency information may include one or more of edge information, detail information, and texture information in the image.
  • performing super-resolution processing on the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model may include: performing scale adjustment processing on the texture image feature through the neural network model to obtain a texture image of the first scale
  • the first-scale texture image feature has the same scale as the first-scale image feature in the multi-scale image; the first HDR image and the first-scale texture image feature are multiplied by the neural network model to obtain the first Three image features: the third image feature and the first-scale image feature are combined channel operation, convolution operation, and up-sampling operation through the neural network model.
  • a multi-level super-division processing method is proposed, that is, to extract multi-scale feature information from the original resolution input image through feature dimensionality reduction; the super-division processing can be based on the first
  • the multi-scale features of the HDR image, ie the low-resolution HDR image and the image to be processed, that is the original input image, are restored by multi-scale super-division processing.
  • the low-resolution HDR image and the extracted feature information of the corresponding scale of the original input image can be fused, thereby
  • the second HDR image obtained from the image to be processed is the enhanced image after the image to be processed is enhanced.
  • Step 630 Process the super-resolution processed first HDR image and color image features through the neural network model to obtain a second HDR image corresponding to the image to be processed.
  • the second HDR image is an HDR image of the first resolution, that is, the second HDR image has the same resolution size as the image to be processed, and the second HDR image may refer to an enhanced image obtained after the image to be processed is enhanced.
  • a point multiplication operation and a convolution operation may be performed on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain the second HDR image.
  • the above-mentioned dot multiplication operation may refer to pixel-by-pixel multiplication.
  • FIG. 11 is a schematic diagram of a system architecture of an image enhancement method provided by an embodiment of the present application.
  • the system architecture may include an HDR network 710 and a super resolution (super resolution, SR) network 720; among them, the HDR network 710 may include a down-sampling unit 711 and an HDR enhancement unit 712; the SR network 720 may include The color attention unit 721, the super-division processing unit 722, and the HDR correction unit 723.
  • SR super resolution
  • the aforementioned down-sampling unit 711 may be used to perform down-sampling processing on an input image, such as a high-resolution image, to obtain a low-resolution image or a small-resolution image, for example, a 1080P resolution image.
  • the above-mentioned HDR enhancement unit 712 may be used to process the down-sampling processed low-resolution image through an HDR increase method to obtain a low-resolution HDR enhanced image.
  • the HDR enhancement method can use any neural network model with image enhancement to perform image enhancement processing; for example, various neural networks based on Unet or High Dynamic Range Network (HDRNet) can be used. .
  • HDRNet High Dynamic Range Network
  • the above-mentioned color attention unit 721 may be used to extract the color guide map of the input image by using a certain calculation method according to the input image (for example, the original resolution image); wherein, the color guide map may be the color corresponding to the input image. Image characteristics.
  • the color image characteristics of the image to be processed can be acquired through target learning or traditional methods.
  • a target learning method or a traditional method may be used to process the image to be processed to obtain the color image characteristics of the image to be processed.
  • the core of the target learning method is to provide such a learning target so that the difference between the input image and the true value image (learning target) can be measured.
  • Such difference is usually reflected in the bright area and shadow (dark area) area of the image.
  • the color input image can be converted to a grayscale image or converted to a Other color domains of the luminance channel (such as YUV domain, LAB domain), and then set the lowest threshold and highest threshold, the channels (such as grayscale, Y channel in YUV domain, L channel in LAB domain) are lower and higher than the lowest threshold
  • the part with the highest threshold is regarded as the area with larger weight, and the other areas are set as the area with lower weight.
  • the above-mentioned color guide map has higher guidance information for difficult areas (for example, bright areas or dark areas) in the input image, so that the SR network 720 can pay more attention to the enhancement effect of difficult areas in the learning process.
  • the HDR correction unit 723 can be used to combine the low-resolution enhanced image output by the HDR enhancement unit 712 and the image output by the super-division processing unit 722, combined with the color guide map obtained by the color attention unit 721, and use CNN for the super-resolution
  • the divided image is corrected in terms of color, brightness, contrast, saturation, etc., so as to ensure the consistency of the HDR effect before and after the super-division processing unit 722 and strengthen the enhancement effect for the difficult area of the image.
  • FIG. 12 is a schematic diagram of HDR correction through a color guide map provided by an embodiment of the present application.
  • the color guide map shown in Figure 12 that is, the color image features corresponding to the input image can be extracted by using a pre-training network, for example, an Encoder-Decoder Network can be used as a network for extracting the color guide map
  • the input image of the self-encoding and decoding network can refer to the low-resolution input image, and the color guide map can be obtained through the self-encoding and decoding network; among them, the self-encoding and decoding network can ensure that the input image is in the pre-training process by designing the objective function. Difficult areas (for example, bright areas or dark areas) correspond to more weight in the color guide map.
  • the following objective function can be used to train the self-encoding and decoding network:
  • a map represents the color guide map
  • I in represents the input image feature of the self-encoding and decoding network, for example, it can refer to the original resolution image feature or the low resolution image feature
  • I target represents a true value image with the same resolution as the input image.
  • the objective function shown above can make the self-encoding and decoding network have a greater difference between the input image and the target image in a certain area, and the color guide map will reflect a greater weight in this area.
  • the color guide map extraction network such as the self-encoding and decoding network shown in Figure 12, can no longer participate in the training; in the darker area of the input image and the area where the color of the image changes more, the color guide map The medium correspondence can show greater weight, thereby guiding the back-end HDR correction unit 723 to focus on learning these areas.
  • the input image may refer to the original resolution image; or, in order to save the amount of calculation, the input image may be a low-resolution image after down-sampling the original resolution image.
  • the color guide map may also not need to perform an upsampling operation.
  • the above-mentioned HDR correction unit 723 may be a pre-trained CNN network, where the HDR correction unit 723 may receive three parts of input data, where the first input data is the first original resolution output by the super-division processing unit 722 Enhanced image, the first original resolution enhanced image may refer to an image obtained after upsampling the low-resolution enhanced image output by the HDR enhancement unit 712; the second input data is the low-resolution enhanced image output by the HDR enhancement unit 712; The third input data is the color attention guide map output by the color attention unit 721; the HDR correction unit 723 performs HDR correction according to the above three parts of input data, and outputs the second original resolution enhanced image.
  • a texture attention unit can be introduced into the system architecture, as shown in FIG. 13.
  • FIG. 13 may also include a texture attention unit 724, which may be used to extract the texture guide image (that is, the high frequency information of the image) using a certain calculation method according to the input image, that is, the original resolution image.
  • the super-division processing unit 722 can also enhance the restoration of image detail texture and edge regions in the super-division process according to the texture guide map output by the texture attention unit 724.
  • FIG. 14 is a schematic diagram of the extraction process of the texture guide map provided by the embodiment of the present application.
  • the input data of the texture attention unit may be an input image without color information, for example, a grayscale image or Y channel of the YUV color gamut; for example, the input data is a Y channel image feature, and the image can be filtered through Gaussian filtering.
  • the output data and the input image are negatively residual so that the high frequency information of the image can be obtained, that is, the texture guide map.
  • the texture and edges in the image usually exist in the high frequency information of the image as shown in Figure 14. Shown.
  • the acquisition process of the texture guide map is not limited to this type of method, and algorithms such as edge detection may also be used to acquire the texture guide map, which is not limited in this application.
  • the texture attention unit 724 can be used to extract high-frequency information in the image as a texture guide map.
  • the extracted texture guide map will act on the super-division processing unit 722, which can improve the SR network 200's ability to guide the map.
  • the learning of areas with higher weights in the middle can enhance the learning of details such as image edges and textures by the SR network 720.
  • the above-mentioned super-resolution processing unit may adopt a self-guided multi-level super-division unit, that is, the super-division process may have multiple image sizes that progressively grow to the same size as the original resolution input image, as shown in Fig. 15 Shown.
  • FIG. 15 is a schematic diagram of a system architecture of an image enhancement method provided by an embodiment of the present application.
  • the system architecture may include an HDR network 710 and a super resolution (SR) network 720; among them, the HDR network 710 may include a down-sampling unit 711 and an HDR enhancement unit 712; the SR network 720 may It includes a color attention unit 721, a self-guided multi-level super-division unit 722, an HDR correction unit 723, a texture attention unit 724, and a multi-scale self-guided feature extraction unit 725.
  • SR super resolution
  • the aforementioned HDR network 710 is configured to input the image I according to the input original resolution (for example, 4K resolution), and perform HDR enhancement to obtain an HDR enhanced low-resolution HDR image (for example, 1080P resolution).
  • the input original resolution for example, 4K resolution
  • HDR enhancement to obtain an HDR enhanced low-resolution HDR image (for example, 1080P resolution).
  • the input image may be an original resolution image, such as a high-resolution image or a full-resolution image; the input image is processed by the down-sampling unit 711 to obtain a low-resolution image or a small-resolution image; The input HDR enhancement unit 712 is processed to obtain the output low-resolution HDR image.
  • the input image may be an original resolution image, such as a high-resolution image or a full-resolution image
  • the input image is processed by the down-sampling unit 711 to obtain a low-resolution image or a small-resolution image
  • the input HDR enhancement unit 712 is processed to obtain the output low-resolution HDR image.
  • the above-mentioned SR network 720 is used to perform super-division processing on the input image through the color attention unit 721 included in the above-mentioned SR network 720 to the multi-scale self-guided feature extraction unit 725 according to the input low-resolution HDR image and the original resolution image. Thereby, the original resolution of the image is restored, and the output original resolution enhanced image (for example, 4K resolution) is obtained.
  • the output original resolution enhanced image for example, 4K resolution
  • the multi-scale self-guided feature extraction unit 725 in the SR network 720 can be used to perform feature extraction through a convolutional neural network according to the input original resolution image, so as to obtain multiple scales Self-guided map.
  • the above-mentioned self-guided images of multiple scales may refer to image features of different scales obtained through down-sampling and convolution processing of different depths through input graphics.
  • the down-sampling method may include, but is not limited to, sampling interpolation methods. Or pixel rearrangement (space to depth) operations, etc., multiple scales may refer to multiple resolution sizes.
  • the above-mentioned color attention unit 721 may be used to extract the color guide map of the input image by using a certain calculation method according to the input image (for example, the original resolution image); wherein, the color guide map may be the color corresponding to the input image. Image characteristics.
  • the above-mentioned color guide map has higher guidance information for difficult areas (for example, bright areas or dark areas) in the input image, so that the SR network 720 can pay more attention to the enhancement effect of difficult areas in the learning process.
  • the above-mentioned texture attention unit 724 may be used to extract the texture guide image (ie, high-frequency information of the image) using a certain calculation method according to the input image, that is, the original resolution image.
  • the specific process can be seen in Figure 14 above, which will not be repeated here.
  • the input data in the self-guided multi-level super-division unit 722 may include image features of different scales input by the multi-scale self-guided feature extraction unit 725, and corresponding input images output by the texture attention unit 724.
  • the texture image feature of the HDR enhancement unit 712 and the low-resolution HDR enhanced image input by the HDR enhancement unit 712, the self-guided multi-level super-division unit 722 can perform up-sampling and convolution operations on the above-mentioned input data to perform image resolution restoration processing, thereby obtaining a super-division restoration Image H SR .
  • the above-mentioned HDR correction unit 723 may be used to correct the above-mentioned super-reduced image H SR in terms of color, brightness, contrast, saturation, etc., so as to obtain an original resolution enhanced image, and make the original resolution enhanced image It is infinitely close to the true value image; among them, the input data of the HDR correction unit 723 can be a super- resolution restoration image H SR , a low-resolution HDR image H L and a color attention image feature, so as to perform a convolution operation on the input data to obtain
  • the image feature of is the same as the true value image feature or the deviation is within the preset range.
  • FIG. 16 is a schematic flowchart of the self-guided multi-level super-division processing provided by an embodiment of the present application.
  • the multi-scale self-guided image obtained through down-sampling and feature extraction processing is image features with a resolution of W/2 ⁇ H/2 and a resolution of W /4 ⁇ H/4 image features;
  • the multi-scale guide image can be used as a priori information to be spliced with the output features of each level of super-division process to guide the super-division process; for example, for a resolution of W/2 ⁇ H/2 Scale, first adjust the scale of the texture attention image feature to obtain a texture attention image feature with a resolution of W/4 ⁇ H/4; set the texture attention image feature of W/4 ⁇ H/4 to a resolution of W /4 ⁇ H/4 HDR enhanced image is subjected to dot multiplication operation, which can refer to pixel-by-pixel multiplication; then, the resolution of the image feature obtained after dot multiplication and the original resolution of the input image is W/4 ⁇ H/4 image features are connected, that is, channel processing is merged; in the same way, for
  • FIG. 17 shows a schematic diagram of HDR correction through a color guide map after introducing multi-scale image features.
  • the low-resolution image can refer to the image characteristics of the U channel and the V channel of the low-resolution HDR image
  • the original resolution image can refer to The image features of the Y channel extracted from the image output by the self-guided multi-level super-splitting unit 722; the image features of the low-resolution U channel and the V channel are up-sampled, so that the scale of the image feature is the same as the original resolution image scale ;
  • Next connect the image features of the U channel and the V channel after the upsampling process with the Y channel image features of the original resolution image, that is, the combined channel operation; and then convert the combined channel processed YUV image into an RGB image; Perform an up-sampling operation on the color guide image to make the scale of the color guide image the same as
  • the color guide map may also not need to perform an upsampling operation.
  • the self-encoding and decoding network in FIG. 17 may be the same as that in FIG. 12, and will not be repeated here; for the same reason, the HDR correction unit may also be the same as the HDR correction unit in FIG. 12, and see the description in FIG. 12 , I won’t repeat it here.
  • PSNR peak signal to noise ratio
  • SSIM Structural similarity index
  • FIG. 18 to 20 are schematic diagrams of the evaluation results of the visual quality image enhancement quality provided by the embodiments of the present application. Among them, (a) in FIG. 18 is the input image, that is, the image to be processed; (b) in FIG. 18 is the true value image corresponding to the input image; (c) in FIG.
  • Figure 18 is the input image through the image enhancement method of this application The output image obtained by image enhancement processing;
  • Figure 18 (d) is the output image obtained by enhancing the input image through the HDRNet model;
  • Figure 18 (e) is the output image obtained by enhancing the input image through the RSGUnet model Output image;
  • Figure 18 (f) is the output image obtained by enhancing the input image through the SGN model; from Figure 18 (a) to Figure 18 (f), it can be seen that through the embodiment of the application
  • the output image obtained is the closest to the true value image, and there are no artifacts, etc.; in the same way, it can be seen from (a) in FIG. 19 to (f) in FIG.
  • the image enhancement method of the embodiment of the present application can reduce the halo problem while ensuring the HDR effect; similarly, it can be seen from (a) in FIG. 20 to (f) in FIG. 20
  • the color is closer to the true value image, and the texture details of the leaf veins are restored more clearly.
  • FIG. 21 is a schematic block diagram of an image enhancement device provided by an embodiment of the present application. It should be understood that the image enhancement apparatus 800 may execute the image enhancement method shown in FIG. 10.
  • the image enhancement device 800 includes: an acquisition unit 810 and a processing unit 820.
  • the acquiring unit 810 is configured to acquire the first high dynamic range HDR image corresponding to the image to be processed and the color image characteristics of the image to be processed, wherein the image to be processed is an image of the first resolution, so
  • the first HDR image is an image with a second resolution, the first resolution is greater than the second resolution, and the color image feature is used to indicate areas of different brightness or areas of different color changes in the image to be processed
  • the processing unit 820 is configured to input the first HDR image into a neural network model for super-resolution processing; use the neural network model to perform super-resolution processing on the first HDR image and the color
  • the image feature is subjected to image enhancement processing to obtain a second HDR image corresponding to the image to be processed, where the second HDR image refers to an HDR image with a resolution of the first resolution.
  • the acquiring unit 810 is further configured to:
  • the processing unit 820 is specifically configured to:
  • the acquiring unit 810 is further configured to:
  • the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;
  • the processing unit 820 is specifically configured to:
  • the acquiring unit 810 is further configured to:
  • the texture image feature is used to indicate the edge area or the texture area of the image to be processed; acquire the multi-scale image feature of the image to be processed, wherein the multiple The scale image feature is used to indicate the image information of the image to be processed in the case of different scales, and the scale size of any one of the multi-scale image features is different;
  • the processing unit 820 is specifically configured to:
  • the acquiring unit 810 is specifically configured to:
  • the processing unit 820 is specifically configured to:
  • the first-scale image features perform combined channel operation, convolution operation, and up-sampling operation.
  • processing unit 820 is specifically configured to:
  • image enhancement device 800 is embodied in the form of a functional unit.
  • unit herein can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a "unit” may be a software program, a hardware circuit, or a combination of the two that realizes the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • FIG. 22 is a schematic diagram of the hardware structure of an image enhancement device provided by an embodiment of the present application.
  • the image enhancement apparatus 900 shown in FIG. 22 includes a memory 901, a processor 902, a communication interface 903, and a bus 904.
  • the memory 901, the processor 902, and the communication interface 903 implement communication connections between each other through the bus 904.
  • the memory 901 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 901 may store a program.
  • the processor 902 is configured to execute each step of the image enhancement method of the embodiment of the present application, for example, execute each step shown in FIG. 10 to FIG. step.
  • the image enhancement device shown in the embodiment of the present application may be a server, for example, it may be a server in the cloud, or may also be a chip configured in a server in the cloud; or, the image enhancement device shown in the embodiment of the present application
  • the device can be a smart terminal or a chip configured in the smart terminal.
  • the image enhancement method disclosed in the above embodiments of the present application may be applied to the processor 902 or implemented by the processor 902.
  • the processor 902 may be an integrated circuit chip with signal processing capabilities.
  • the steps of the above-mentioned image enhancement method can be completed by hardware integrated logic circuits in the processor 902 or instructions in the form of software.
  • the processor 902 may be a chip including the NPU shown in FIG. 8.
  • the aforementioned processor 902 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor, a digital signal processor (digital signal processor, DSP), and an application specific integrated circuit (application integrated circuit).
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processor
  • application integrated circuit application specific integrated circuit
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory (RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc. mature in the field Storage medium.
  • the storage medium is located in the memory 901, and the processor 902 reads the instructions in the memory 901, and combines its hardware to complete the functions required by the units included in the image enhancement device shown in FIG. 21 in the implementation of this application, or execute the method of this application Each step of the image enhancement method shown in FIG. 10 to FIG. 17 of the embodiment.
  • the communication interface 903 uses a transceiving device such as but not limited to a transceiver to implement communication between the device 900 and other devices or a communication network.
  • a transceiving device such as but not limited to a transceiver to implement communication between the device 900 and other devices or a communication network.
  • the bus 904 may include a path for transferring information between various components of the image intensifying device 900 (for example, the memory 901, the processor 902, and the communication interface 903).
  • the image enhancement device 900 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the image enhancement device 900 may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned image enhancement device 900 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned image enhancement device 900 may also only include the necessary devices for implementing the embodiments of the present application, and not necessarily all the devices shown in FIG. 22.
  • the embodiment of the present application also provides a chip, which includes a transceiver unit and a processing unit.
  • the transceiver unit may be an input/output circuit or a communication interface;
  • the processing unit is a processor, microprocessor, or integrated circuit integrated on the chip.
  • the chip can execute the image enhancement method in the above method embodiment.
  • the embodiment of the present application also provides a computer-readable storage medium with an instruction stored thereon, and the image enhancement method in the foregoing method embodiment is executed when the instruction is executed.
  • the embodiments of the present application also provide a computer program product containing instructions that, when executed, execute the image enhancement method in the foregoing method embodiments.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • a part of the processor may also include a non-volatile random access memory.
  • the processor may also store device type information.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • a part of the processor may also include a non-volatile random access memory.
  • the processor may also store device type information.
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Processing (AREA)

Abstract

一种图像增强方法以及图像增强装置，该图像增强方法包括：获取待处理图像对应的第一高动态范围HDR图像以及所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率（610）；将所述第一HDR图像输入至神经网络模型进行超分辨处理（620）；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像（630）。上述方法能够提升图像增强处理的效果。

Description

图像增强方法以及图像增强装置
本申请要求于2020年02月19日提交中国专利局、申请号为202010102590.4、申请名称为“图像增强方法以及图像增强装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，更具体地，涉及计算机视觉领域中的图像增强方法以及图像增强装置。
背景技术
计算机视觉是各个应用领域，如制造业、检验、文档分析、医疗诊断，和军事等领域中各种智能/自主系统中不可分割的一部分，它是一门关于如何运用照相机/摄像机和计算机来获取我们所需的，被拍摄对象的数据与信息的学问。形象地说，就是给计算机安装上眼睛(照相机/摄像机)和大脑(算法)用来代替人眼对目标进行识别、跟踪和测量等，从而使计算机能够感知环境。因为感知可以看作是从感官信号中提取信息，所以计算机视觉也可以看作是研究如何使人工系统从图像或多维数据中“感知”的科学。总的来说，计算机视觉就是用各种成像系统代替视觉器官获取输入信息，再由计算机来代替大脑对这些输入信息完成处理和解释。计算机视觉的最终研究目标就是使计算机能像人那样通过视觉观察和理解世界，具有自主适应环境的能力。
图像增强又可以称为图像质量增强是图像处理领域重要的一个分支，通过图像增强技术可以在不重新采集数据的情况下改善图像质量，以满足更多实际应用需求。随着深度学习方法的发展，尤其是基于卷积神经网络(Convolutional neural networks，CNN)的方法进行图像处理是近年来人工智能领域发展的关键推动力，在计算机视觉的多种任务，比如图像复原或者图像增强取得了令人瞩目的效果。
目前，为了满足较大分辨率图像(例如，4K分辨率)进行图像增强的实时需要提出了“下采样+高动态范围(High dynamic range，HDR)+超分处理”的方式来节约计算开销，以达到实时图像质量增强的目的；即将原始分辨率输入图像进行下采样，在低分辨率下进行图像增强，得到低分辨率增强图像；根据原始分辨率输入图像和低分辨率增强图像，通过预训练CNN还原原始分辨率，得到原始分辨率增强图像。但是，由于基于深度学习的超分方法通常仅对原始分辨率输入图像的亮度通道进行复杂超分过程，而对于颜色通道通常采用简单上采样，从而到导致图像增强方法存在超分辨率处理前后图像在颜色、亮度、对比度、饱和度等方面不一致的问题。因此，在满足图像增强的实时需求的情况下，如何提高增强图像处理后的图像质量成为一个亟需解决的问题。
发明内容
本申请提供一种图像增强方法以及图像增强装置，能够在满足图像增强处理的实时需 求的情况下使得增强处理后的图像在颜色、亮度、对比度、饱和度等方面的性能得以增强，从而能够提升图像增强处理的效果。
第一方面，提供了一种图像增强方法，包括：获取待处理图像对应的第一高动态范围HDR图像与所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述第一HDR图像输入神经网络模型进行超分辨处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像。
需要说明的是，第二HDR图像是指与待处理图像分辨率大小相同的图像；其中，分辨率大小相同可以是指图像中具有相同数量的像素点，通过对待处理图像进行图像增强可以使得待处理图像中的至少部分像素点的像素值得到调整，从而使得在视觉上第二HDR图像的视觉效果优于待处理图像。
其中，上述待处理图像可以是指具有图像增强需求的待处理图像；比如，待处理图像可以是指分辨率较高并且画质较差的原始图像；例如，可以是指受到天气、距离、拍摄环境等因素的影响，获取的待处理图像存在图像画质较低的问题；图像画质较低包括但不限于：图像模糊、或者图像颜色、亮度、饱和度、对比度、动态范围较差等。
在一种可能的实现方式中，第一HDR图像可以是通过对待处理图像进行下采样处理以及HDR增强后得到的图像；其中，高动态范围(high dynamic range，HDR)图像与普通动态范围(standard dynamic range，SDR)图像相比可以提供更多的动态范围和图像色彩，即在HDR图像中可以包括更多图像细节。
需要说明的是，上述颜色图像特征又可以称为颜色引导图，通过颜色图像特征可以对于输入神经网络模型中的待处理图像中的困难区域(例如，亮区或者暗区)具有更高的引导信息，使得神经网络模型在学习过程中可以更加注重困难区域的增强效果。
在本申请的实施例中，通过获取待处理图像以及待处理图像的第一HDR图像，其中，第一HDR图像即待处理图像对应的低分辨率增强图像，在神经网络模型中通过待处理图像的颜色图像特征对第一HDR图像进行超分辨率处理，从而得到与待处理图像的分辨率大小相同的增强图像即第二HDR图像；通过在对第一HDR图像进行超分辨率处理时可以引入了颜色注意力机制，即颜色图像特征从而使得在进行图像增强的过程中，提高神经网络模型对于待处理图像中的困难区域(过亮、过暗区域)在颜色、亮度、对比度以及饱和度等方面的恢复效果，从而提高图像增强的效果。
结合第一方面，在第一方面的某些实现方式，还包括：获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
所述将所述第一HDR图像输入所述神经网络模型进行超分辨率处理，包括：通过所述神经网络模型根据所述纹理图像特征对所述第一HDR图像进行超分辨率处理。
需要说明的是，在神经网络模型中引入纹理注意力机制是为了使得神经网络模型对于图像中的边缘与纹理等细节方面的学习；通过使用纹理图像特征又称为纹理引导图可以提升神经网络模型对于纹理引导图中权重较高区域的学习；从而对第一HDR图像超分处理 的过程能够提高超分算法对于图像纹理细节的恢复能力，避免超分处理后引入的图像模糊或者其它感官差异。
在本申请的实施例中，在对第一HDR图像进行超分处理的过程中可以基于双注意力机制进行超分还原处理，其中，双注意力机制可以包括纹理注意力机制与颜色注意力机制；即通过将颜色图像特征与纹理图像特征用于第一HDR图像的超分处理过程中，使得最终得到的增强图像即待处理图像的第二HDR图像在颜色、亮度、饱和度、对比度以及纹理细节等方面与真值图像相同或者接近，从而提升了图像增强的效果。
结合第一方面，在第一方面的某些实现方式，还包括：获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
所述将所述第一HDR图像输入神经网络模型进行超分辨处理，包括：
通过所述神经网络模型根据所述多尺度图像特征对所述第一HDR图像进行超分辨率处理。
其中，多尺度图像特征可以是指不同分辨率的图像特征，通过引入多尺度的图像特征可以使得神经网络模型中引入更多的图像细节信息，有利于保证第一HDR图像在细节等方面的还原。
需要说明的是，多尺度图像特征可以用于指示在不同尺度的情况下待处理图像的图像信息，其中，不同尺度可以是指不同的分辨率大小，图像信息可以是指图像中的高频信息；比如，高频信息可以包括图像中的边缘信息、细节信息以及纹理信息中的一种或者多种。
在本申请的实施例中，可以获取待处理图像的多尺度图像特征即对待处理图像从原始分辨率中通过特征降维的方式，提取待处理图像的多尺度的特征信息；在超分处理过程中可以基于第一HDR图像即低分辨率HDR图像以及待处理图像即原始输入图像的多尺度特征进行多尺度的超分处理还原，低分辨率HDR图像与提取的原始输入图像对应尺度的特征信息可以进行融合，从而得到待处理图像的第二HDR图像即对待处理图像进行增强处理后的增强图像。
结合第一方面，在第一方面的某些实现方式，还包括：获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
所述将所述第一HDR图像输入所述神经网络模型进行超分辨率处理，包括：通过所述神经网络模型根据所述纹理图像特征与所述多尺度特征对所述第一HDR图像进行超分辨率处理。
在本申请的实施例中，在对第一HDR图像进行超分处理的过程中可以既引入双注意力机制又引入多尺度图像特征；其中，双注意力机制可以包括纹理注意力机制与颜色注意力机制，即颜色图像特征与纹理特征，通过双注意力机制与多尺度图像特征使得最终得到的增强图像即待处理图像的第二HDR图像在颜色、亮度、饱和度、对比度以及纹理细节等方面与真值图像相同或者接近，从而提升了图像增强的效果，即提高图像增强处理后图像质量。
结合第一方面，在第一方面的某些实现方式，所述通过所述神经网络模型根据所述纹理图像特征与所述多尺度图像特征对所述第一HDR图像进行超分辨率处理，包括：
获取第一尺度的纹理图像特征，其中，所述第一尺度的纹理图像特征与所述多尺度图像中的第一尺度图像特征的尺度相同；
通过所述神经网络模型对所述第一HDR图像与所述第一尺度的纹理图像特征进行点乘操作，得到第三图像特征；
通过所述神经网络模型对所述第三图像特征与所述第一尺度图像特征进行合通道操作、卷积操作以及上采样操作。
结合第一方面，在第一方面的某些实现方式，所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，包括：
所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行点乘操作与卷积操作，得到所述第二HDR图像。
第二方面，提供一种图像增强方法，应用于智慧屏(或者，人工智能屏)，包括：检测到用户指示打开所述智慧屏的显示模式界面的第一操作；响应于所述第一操作，在所述智慧屏上显示所述显示模式选择界面；检测到所述用户指示第一显示模式的第二操作；响应于所述第二操作，在所述智慧屏上显示输出图像，所述输出图像是针对待处理图像进行图像增强处理后得到的第二高动态范围HDR图像，
其中，神经网络模型应用于所述图像增强处理过程中，将获取的所述待处理图像的第一高动态范围HDR图像输入所述神经网络模型进行超分处理；所述第二HDR图像通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述待处理图像的颜色图像特征进行图像增强处理得到的；所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像；所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域。
可选地，上述显示模式界面可以包括但不限于以下显示模式：HDR模式、杜比模式、超清模式、蓝光模式以及智能模式。
应理解，上述输出图像是针对待处理图像进行图像增强处理后得到的第二HDR图像，即与待处理图像分辨率大小相同的HDR图像；其中，待处理图像可以是指可以是指电子设备通过摄像头拍摄到的图像/视频，或者，上述待处理图像还可以是从电子设备内部获得的图像(例如，电子设备的相册中存储的图像，或者，电子设备从云端获取的图片)。比如，上述待处理图像可以是数据库中存储的原始片源，或者，也可以是指实时播放的片源。
在一种可能的实现方式中，上述待处理图像的图像增强方法可以是在云端执行的离线方法，比如，可以由云端服务器执行对待处理图像的图像增强得到输出图像，通过智慧屏显示输出图像。
在另一种可能的实现方式中，上述图像增强方法可以是由本地设备执行，即可以是指在智慧屏中执行对待处理图像的图像增强得到的输出图像。在一种可能的实现方式中，上述用户指示打开所述智慧屏的显示模式界面的第一操作可以包括但不限于：通过控制设备 指示智慧屏打开显示模式界面，也可以包括用户通过语音指示智慧屏打开显示模式界面的行为，或者，还可以包括用户其它的指示智慧屏打开显示模式界面的行为；上述为举例说明，并不对本申请作任何限定。
同理，上述用户指示第一显示模式的第二操作可以包括但不限于：通过控制设备指示显示模式界面中的第一显示模型，也可以包括用户通过语音指示显示模式界面中的第一显示模型的行为，或者，还可以包括用户其它的指示选择显示模式界面中的第一显示模式的行为；上述为举例说明，并不对本申请作任何限定。
其中，上述第一显示模式可以是指HDR模式或者其他专业模式，通过第一显示模式可以实现本申请实施例提供的图像增强方法，从而提升输出图像的图像质量，使得使用智慧屏的用户获得更好的视觉体验感。
需要说明的是，上述对待处理图像进行特征增强处理的具体流程可以根据上述第一方面以及第一方面的任意一种实现方式得到。
结合第二方面，在第二方面的某些实现方式，还包括：获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
所述将所述第一HDR图像输入所述神经网络模型进行超分辨率处理，包括：通过所述神经网络模型根据所述纹理图像特征对所述第一HDR图像进行超分辨率处理。
结合第二方面，在第二方面的某些实现方式，还包括：获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
所述将所述第一HDR图像输入神经网络模型进行超分辨处理：
通过所述神经网络模型根据所述多尺度图像特征对所述第一HDR图像进行超分辨率处理。
结合第二方面，在第二方面的某些实现方式，还包括：获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
所述将所述第一HDR图像输入所述神经网络模型进行超分辨率处理，包括：
通过所述神经网络模型根据所述纹理图像特征与所述多尺度特征对所述第一HDR图像进行超分辨率处理。
应理解，多尺度图像特征可以用于指示在不同尺度的情况下待处理图像的图像信息，其中，不同尺度可以是指不同的分辨率大小，图像信息可以是指图像中的高频信息；比如，高频信息可以包括图像中的边缘信息、细节信息以及纹理信息中的一种或者多种。
结合第二方面，在第二方面的某些实现方式，所述通过所述神经网络模型根据所述纹理图像特征与所述多尺度图像特征对所述第一HDR图像进行超分辨率处理，包括：
获取第一尺度的纹理图像特征，其中，所述第一尺度的纹理图像特征与所述多尺度图像中的第一尺度图像特征的尺度相同；
通过所述神经网络模型对所述第一HDR图像与所述第一尺度的纹理图像特征进行点乘操作，得到第三图像特征；
通过所述神经网络模型对所述第三图像特征与所述第一尺度图像特征进行合通道操作、卷积操作以及上采样操作。
结合第二方面，在第二方面的某些实现方式，所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，包括：
所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行点乘操作与卷积操作，得到所述第二HDR图像。
第三方面，提供了一种图像增强方法，应用于具有显示屏和摄像头的电子设备，包括：检测到用户指示相机的第二操作；响应于所述第二操作，在所述显示屏内显示输出图像，或者在所述电子设备中保存输出图像，所述输出图像是针对待处理图像进行图像增强处理后得到的第二高动态范围HDR图像，其中，神经网络模型应用于所述图像增强处理过程中，将获取的所述待处理图像的第一高动态范围HDR图像输入所述神经网络模型进行超分处理；所述第二HDR图像通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述待处理图像的颜色图像特征进行图像增强处理得到的；所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像；所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域。
可选地，还可以包括：检测到所述用户用于打开相机的第一操作；响应于所述第一操作，在所述显示屏上显示拍摄界面，所述拍摄界面上包括取景框，所述取景框内包括所述待处理图像。
其中，上述对待处理图像进行特征增强处理的具体流程可以根据上述第一方面以及第一方面的任意一种实现方式得到。
在一种可能的实现方式中，本申请实施例提供的图像增强方法可以应用于智能终端的拍照领域，通过本申请实施例的图像增强方法可以对图像增强处理得到的输出图像在颜色、亮度、对比度、饱和度等方面的性能得以提升。
例如，可以是在智能终端进行实时拍照时，对获取的原始图像进行图像增强处理，将图像增强处理后的输出图像显示在智能终端的屏幕上，或者，还可以通过对获取的原始图像进行图像增强处理，将图像增强处理后的输出图像保存至智能终端的相册中。
第四方面，提供了一种图像增强方法，包括：获取待处理的道路图像以及所述道路图像对应的第一高动态范围HDR图像，其中，所述道路图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述道路图像输入神经网络模型得到所述道路画面的颜色图像特征，其中，所述颜色图像特征用于指示所述道路图像中的不同亮度区域或者不同颜色变化区域；将所述第一HDR图像输入所述神经网络模型进行超分辨率处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行处理，得到所述道路图像对应的第二HDR图像，其中，所述第二HDR图像为所述第一分辨率的HDR图像；根据所述第二HDR图像，识别所述第二HDR图像中的道路信息。
其中，上述对道路图像进行特征增强处理的具体流程可以根据上述第一方面以及第一方面的任意一种实现方式得到。
在一种可能的实现方式中，本申请实施例提供的图像增强方法可以应用于自动驾驶领域。例如，可以应用于自动驾驶车辆的导航系统中，通过本申请中的图像增强方法可以使得自动驾驶车辆在道路行驶的导航过程中，通过获取的画质较低的原始道路画面进行图像增强处理，得到增强处理后的道路图像，从而实现自动驾驶车辆的安全性。
第五方面，提供了一种图像增强方法，包括：获取待处理的街景图像以及所述街景图像对应的第一高动态范围HDR图像，其中，所述街景图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述街景图像输入神经网络模型得到所述街景图像的颜色图像特征，其中，所述颜色图像特征用于指示所述街景图像中的不同亮度区域或者不同颜色变化区域；将所述第一HDR图像输入所述神经网络模型进行超分辨率处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行处理，得到所述街景图像对应的第二HDR图像，其中，所述第二HDR图像为所述第一分辨率的HDR图像；根据所述第二HDR图像，识别所述第二HDR图像中的街景信息。
其中，上述对街景图像进行特征增强处理的具体流程可以根据上述第一方面以及第一方面的任意一种实现方式得到。
在一种可能的实现方式中，本申请实施例提供的图像增强方法可以应用于安防领域。例如，本申请实施例的图像增强方法可以应用于平安城市的监控图像增强，比如，公共场合的监控设备采集到的图像(或者，视频)往往受到天气、距离等因素的影响，存在图像模糊，图像画质较低等问题。通过本申请的图像增强方法可以对采集到的原始图像进行图像增强，从而可以为公安人员恢复出车牌号码、清晰人脸等重要信息，为案件侦破提供重要的线索信息。
应理解，在上述第一方面中对相关内容的扩展、限定、解释和说明也适用于第二方面、第三方面、第四方面以及第五方面中相同的内容。
第六方面，提供一种图像增强装置，包括用于执行上述第一方面至第五方面以及第一方面至第五方面中的任意一种实现方式中的图像增强方法的模块/单元。
第七方面，提供一种图像增强装置，包括：存储器，用于存储程序；处理器，用于执行该存储器存储的程序，当该存储器存储的程序被执行时，该处理器用于执行：获取待处理图像对应的第一高动态范围HDR图像与所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述第一HDR图像输入神经网络模型进行超分辨处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像。
在一种可能的实现方式中，上述图像增强装置中包括的处理器还用于上述第一方面至第五方面以及第一方面至第五方面中的任意一种实现方式中的图像增强方法。
第八方面，提供了一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行上述第一方面至第五方面以及第一方面至第五方面中的任意一种实现方式中的图像增强方法。
第九方面，提供了一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第一方面至第五方面以及第一方面至第五方面中的任意一种实现方式中的图像增强方法。
第十方面，提供了一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行上述第一方面至第五方面以及第一方面至第五方面中的任意一种实现方式中的图像增强方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行上述第一方面至第五方面以及第一方面至第五方面中的任意一种实现方式中的图像增强方法。
附图说明
图1是本申请实施例提供的一种人工智能主体框架示意图；
图2是本申请实施例提供的一种应用场景的示意图；
图3是本申请实施例提供的另一种应用场景的示意图；
图4是本申请实施例提供的再一种应用场景的示意图；
图5是本申请实施例提供的再一种应用场景的示意图；
图6是本申请实施例提供的系统架构的结构示意图；
图7是本申请实施例提供的一种卷积神经网络结构示意图；
图8是本申请实施例提供的一种芯片硬件结构示意图；
图9是本申请实施例提供了一种系统架构的示意图；
图10是本申请实施例提供的图像增强处理的示意图；
图11是本申请实施例提供的一种图像增强方法的系统架构的示意图；
图12是本申请实施例提供的通过颜色引导图进行HDR校正的流程示意图；
图13是本申请实施例提供的另一种图像增强方法的系统架构的示意图；
图14是本申请实施例提供的纹理引导图的提取过程的示意图；
图15是本申请实施例提供的再一种图像增强方法的系统架构的示意图；
图16是本申请实施例提供的自引导多级超分处理的示意性流程图；
图17是本申请实施例提供的引入多尺度图像特征后通过颜色引导图进行HDR校正的示意图；
图18是本申请实施例提供的图像增强质量的测评结果的示意图；
图19是本申请实施例提供的图像增强质量的测评结果的示意图；
图20是本申请实施例提供的图像增强质量的测评结果的示意图；
图21是本申请实施例提供的图像增强装置的示意性框图；
图22是本申请实施例提供的图像增强装置的硬件结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施 例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
应理解，本申请实施例中的图像可以为静态图像(或称为静态画面)或动态图像(或称为动态画面)，例如，本申请中的图像可以为视频或动态图片，或者，本申请中的图像也可以为静态图片或照片。为了便于描述，本申请在下述实施例中将静态图像或动态图像统一称为图像。
还应理解，在本申请的各实施例中，“第一”、“第二”、“第三”等仅是为了指代不同的对象，并不表示对指代的对象有其它限定。
图1示出一种人工智能主体框架示意图，该主体框架描述了人工智能系统总体工作流程，适用于通用的人工智能领域需求。
下面从“智能信息链”(水平轴)和“信息技术(information technology，IT)价值链”(垂直轴)两个维度对上述人工智能主题框架100进行详细的阐述。
“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。
“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到系统的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施110
基础设施为人工智能系统提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。
基础设施可以通过传感器与外部沟通，基础设施的计算能力可以由智能芯片提供。
这里的智能芯片可以是中央处理器(central processing unit，CPU)、神经网络处理器(neural-network processing unit，NPU)、图形处理器(graphics processing unit，GPU)、专门应用的集成电路(application specific integrated circuit，ASIC)以及现场可编程门阵列(field programmable gate array，FPGA)等硬件加速芯片。
基础设施的基础平台可以包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。
例如，对于基础设施来说，可以通过传感器和外部沟通获取数据，然后将这些数据提供给基础平台提供的分布式计算系统中的智能芯片进行计算。
(2)数据120
基础设施的上一层的数据用于表示人工智能领域的数据来源。该数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有系统的业务数据以及力、位移、液位、温度、湿度等感知数据。
(3)数据处理130
上述数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等处理方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能系统中，模拟人类的智能推理方式，依据推理控制策略，利 用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力140
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用系统，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用150
智能产品及行业应用指人工智能系统在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能制造、智能交通、智能家居、智能医疗、智能安防、自动驾驶，平安城市，智能终端等。
图2是本申请实施例提供的图像增强方法的应用场景的示意图。
如图2所示，本申请实施例的技术方案可以应用于智能终端，本申请实施例中的图像增强方法可以对输入图像(或输入视频)进行图像增强处理，得到该输入图像(或输入视频)经图像增强后的输出图像(或输出视频)。
上述智能终端可以为移动的或固定的，例如，该智能终端可以是具有图像增强功能的移动电话、平板个人电脑(tablet personal computer，TPC)、媒体播放器、智能电视、笔记本电脑(laptop computer，LC)、个人数字助理(personal digital assistant，PDA)、个人计算机(personal computer，PC)、照相机、摄像机、智能手表、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)，可穿戴式设备(wearable device，WD)或者自动驾驶的车辆等，本申请实施例对此不作限定。
需要说明的是，在本申请的实施例中图像增强也可以称为图像质量增强，具体可以是指对图像的亮度、颜色、对比度、饱和度和/或动态范围等进行处理，以使得该图像的各项指标满足预设的条件或者预设指标。在本申请实施例中，图像增强和图像质量增强具有相同的涵义。
下面对本申请实施例的具体应用场景进行举例说明。
应用场景一：智慧屏(AI屏)
在一个实施例中，本申请实施例的图像增强方法还可以应用于智慧屏领域；其中，智慧屏是区隔于传统电视、突破大屏品类限制。随着智慧屏等新产品的诞生，未来手机和智慧屏将成为用户智慧生活的双中心，手机仍然是用户个人中心，而智慧屏则可能成为家庭情感中心。智慧屏将承担家庭中的更多角色，不仅是家庭的影音娱乐中心，更是信息共享中心、控制管理中心和多设备交唱互中心。例如，在利用智慧屏等播放视频时，为了显示更好的图像质量(画质)，可以对视频的原始片源采用本申请实施例的图像增强方法进行图像增强处理，以提升片源的画质，获得更好的视觉观感。
示例性地，在使用智慧屏播放老电影(老电影的片源的时间比较早、片源的画质较差)时，可以使用本申请实施例的图像增强方法对老电影的片源进行图像增强处理，能显示现代电影的视觉感官。比如，可以通过本申请实施例的图像增强方法将老电影的片源增强为高动态范围图像(high-dynamic range，HDR)10，或者杜比视界(Dolby vision)标准的高质量视频。
示例性地，本申请提供了一种图像增强方法，包括：获取待处理图像(例如，电影的 原始片源，或者在线片源)以及所述待处理图像对应的第一高动态范围HDR图像，其中，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述待处理图像输入神经网络模型得到所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域；将所述第一HDR图像输入所述神经网络模型进行超分辨率处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行处理，得到所述待处理图像对应的第二HDR图像(例如，提升画质的片源)，其中，所述第二HDR图像为所述第一分辨率的HDR图像。
示例性地，本申请提供了一种图像增强方法，应用于智慧屏(或者，人工智能屏)，包括：检测到用户指示打开所述智慧屏的显示模式界面的第一操作；响应于所述第一操作，在所述智慧屏上显示所述显示模式选择界面；检测到所述用户指示第一显示模式的第二操作；响应于所述第二操作，在所述智慧屏上显示输出图像，所述输出图像是针对待处理图像进行图像增强处理后得到的第二高动态范围HDR图像，其中，神经网络模型应用于所述图像增强处理过程中，将获取的所述待处理图像的第一高动态范围HDR图像输入所述神经网络模型进行超分处理；所述第二HDR图像通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述待处理图像的颜色图像特征进行图像增强处理得到的；所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像；所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域。
可选地，上述显示模式界面可以包括但不限于以下显示模式：HDR模式、杜比模式、超清模式、蓝光模式以及智能模式。
应理解，上述输出图像是针对待处理图像进行图像增强处理后得到的第二HDR图像，即与待处理图像分辨率大小相同的HDR图像；其中，待处理图像可以是指可以是指电子设备通过摄像头拍摄到的图像/视频，或者，上述待处理图像还可以是从电子设备内部获得的图像(例如，电子设备的相册中存储的图像，或者，电子设备从云端获取的图片)。比如，上述待处理图像可以是数据库中存储的原始片源，或者，也可以是指实时播放的片源。
在一种可能的实现方式中，上述待处理图像的图像增强方法可以是在云端执行的离线方法，比如，可以由云端服务器执行对待处理图像的图像增强得到输出图像，通过智慧屏显示输出图像。
在另一种可能的实现方式中，上述图像增强方法可以是由本地设备执行，即可以是指在智慧屏中执行对待处理图像的图像增强得到的输出图像。
应理解，上述用户指示打开所述智慧屏的显示模式界面的第一操作可以包括但不限于：通过控制设备指示智慧屏打开显示模式界面，也可以包括用户通过语音指示智慧屏打开显示模式界面的行为，或者，还可以包括用户其它的指示智慧屏打开显示模式界面的行为；上述为举例说明，并不对本申请作任何限定。
同理，上述用户指示第一显示模式的第二操作可以包括但不限于：通过控制设备指示显示模式界面中的第一显示模型，也可以包括用户通过语音指示显示模式界面中的第一显 示模型的行为，或者，还可以包括用户其它的指示选择显示模式界面中的第一显示模式的行为；上述为举例说明，并不对本申请作任何限定。
其中，上述第一显示模式可以是指HDR模式或者其他专业模式，通过第一显示模式可以实现本申请实施例提供的图像增强方法，从而提升输出图像的图像质量，使得使用智慧屏的用户获得更好的视觉体验感。需要说明的是，本申请实施例提供的图像增强的方法同样适用于后面图6至图17中相关实施例中对图像增强方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应用场景二：智能终端拍照领域
在一个实施例中，如图3所示，本申请实施例的图像增强方法可以应用于智能终端设备(例如，手机)的拍摄。通过本申请实施例的图像增强方法可以对获取的质量较差的原始图像(或者视频)进行图像增强处理得到画质提升的输出图像(或者输出视频)。
需要说明的是，在图3中为了区别于灰度图像部分，彩色图像部分通过斜线填充来表示。
示例性地，可以通过本申请实施例的图像增强方法在智能终端进行实时拍照时，对获取的原始图像进行图像增强处理，将图像增强处理后的输出图像显示在智能终端的屏幕上。
示例性地，可以通过本申请实施例的图像增强方法对获取的原始图像进行图像增强处理，将图像增强处理后的输出图像保存至智能终端的相册中。
示例性地，本申请提出了一种图像增强方法，应用于具有显示屏和摄像头的电子设备，包括：检测到用户指示相机的第二操作；响应于所述第二操作，在所述显示屏内显示输出图像，或者在所述电子设备中保存输出图像，所述输出图像是针对所述待处理图像进行图像增强处理后得到的第二高动态范围HDR图像，其中，神经网络模型应用于所述图像增强处理过程中，将获取的所述待处理图像的第一高动态范围HDR图像输入所述神经网络模型进行超分处理；所述第二HDR图像通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述待处理图像的颜色图像特征进行图像增强处理得到的；所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像；所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域。
可选地，还可以包括：检测到所述用户用于打开相机的第一操作；响应于所述第一操作，在所述显示屏上显示拍摄界面，所述拍摄界面上包括取景框，所述取景框内包括所述待处理图像。
需要说明的是，本申请实施例提供的图像增强方法同样适用于后面图6至图17中相关实施例中对图像增强方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应用场景三：自动驾驶领域
在一个实施例中，如图4所示，本申请实施例的图像增强方法可以应用于自动驾驶领域。例如，可以应用于自动驾驶车辆的导航系统中，通过本申请中的图像增强方法可以使得自动驾驶车辆在道路行驶的导航过程中，通过获取的画质较低的原始道路图像(或原始道路视频)进行图像增强处理，得到增强处理后的道路图像(或者道路视频)，从而实现 自动驾驶车辆的安全性。
示例性地，本申请提供了一种图像增强方法，包括：获取待处理的道路图像以及所述道路图像对应的第一高动态范围HDR图像，其中，所述道路图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；
将所述道路图像输入神经网络模型得到所述道路画面的颜色图像特征，其中，所述颜色图像特征用于指示所述道路图像中的不同亮度区域或不同颜色变化区域；将所述第一HDR图像输入所述神经网络模型进行超分辨率处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行处理，得到所述道路图像对应的第二HDR图像，其中，所述第二HDR图像为所述第一分辨率的HDR图像；根据所述第二HDR图像，识别所述第二HDR图像中的道路信息。
需要说明的是，本申请实施例提供的图像增强的方法同样适用于后面图6至图17中相关实施例中对图像增强方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应用场景四：平安城市领域
在一个实施例中，如图5所示，本申请实施例的图像增强方法可以应用于平安城市领域，比如，安防领域。例如，本申请实施例的图像增强方法可以应用于平安城市的监控图像增强，比如，公共场合的监控设备采集到的图像(或者，视频)往往受到天气、距离等因素的影响，存在图像模糊，图像画质较低等问题。通过本申请的图像增强方法可以对采集到的图片进行图像增强，从而可以为公安人员恢复出车牌号码、清晰人脸等重要信息，为案件侦破提供重要的线索信息。
示例性地，本申请提供了一种图像增强方法，包括：获取待处理的街景图像以及所述街景图像对应的第一高动态范围HDR图像，其中，所述街景图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述街景图像输入神经网络模型得到所述街景图像的颜色图像特征，其中，所述颜色图像特征用于指示所述街景图像中的不同亮度区域或者不同颜色变化区域；将所述第一HDR图像输入所述神经网络模型进行超分辨率处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行处理，得到所述街景图像对应的第二HDR图像，其中，所述第二HDR图像为所述第一分辨率的HDR图像；根据所述第二HDR图像，识别所述第二HDR图像中的街景信息。
需要说明的是，本申请实施例提供的图像增强的方法同样适用于后面图6至图17中相关实施例中对图像增强方法相关内容的扩展、限定、解释和说明，此处不再赘述。
应理解，上述为对应用场景的举例说明，并不对本申请的应用场景作任何限定。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例可能涉及的神经网络的相关术语和概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以x s和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2021076859-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，W s为x s的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来 将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有多层隐含层的神经网络。按照不同层的位置对DNN进行划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2021076859-appb-000002
其中，
Figure PCTCN2021076859-appb-000003
是输入向量，
Figure PCTCN2021076859-appb-000004
是输出向量，
Figure PCTCN2021076859-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2021076859-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2021076859-appb-000007
由于DNN层数多，系数W和偏移向量
Figure PCTCN2021076859-appb-000008
的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2021076859-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2021076859-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的 过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
(5)反向传播算法
神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经网络模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如权重矩阵。
图6示出了本申请实施例提供的一种系统架构200。
在图6中，数据采集设备260用于采集训练数据。针对本申请实施例的图像增强方法来说，可以通过训练数据对图像增强模型(又称为图像增强网络)进行进一步训练，即数据采集设备260采集的训练数据可以是训练图像。
示例性地，在本申请实施例中训练图像增强模型的训练数据可以包括原始图像、样本增强图像。
例如，原始图像可以是指图像画质较低的图像，样本增强图像可以是指图像画质较高的图像，比如，可以是指相对于样本图像而言在亮度、颜色、细节等一个或多个方面均得到提升后的图像。
应理解，图像增强也可以称为图像质量增强，具体可以是指对图像的亮度、颜色、对比度、饱和度和/或动态范围等进行处理，以使得该图像的各项指标满足预设的条件或者预设指标。在本申请实施例中，图像增强和图像质量增强具有相同的涵义。
在采集到训练数据之后，数据采集设备260将这些训练数据存入数据库230，训练设备220基于数据库230中维护的训练数据训练得到目标模型/规则201(即本申请实施例中的图像增强模型)。训练设备220将训练数据输入图像增强模型，直到训练图像增强模型输出的预测增强图像与样本增强图像之间的差值满足预设条件(例如，预测增强图像与样本增强图像差值小于一定阈值，或者预测增强图像与样本增强图像的差值保持不变或不再减少)，从而完成目标模型/规则201的训练。
示例性地，本申请实施例中用于执行图像增强方法的图像增强模型可以实现端到端的训练，比如，图像增强模型可以通过输入图像与输入图像对应的样本增强图像(例如，真值图像)实现端到端的训练。
在本申请提供的实施例中，该目标模型/规则201是通过训练图像增强模型得到的。需要说明的是，在实际的应用中，所述数据库230中维护的训练数据不一定都来自于数据采集设备260的采集，也有可能是从其他设备接收得到的。
另外需要说明的是，训练设备220也不一定完全基于数据库230维护的训练数据进行目标模型/规则201的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上 述描述不应该作为对本申请实施例的限定。还需要说明的是，数据库230中维护的训练数据中的至少部分数据也可以用于执行设备210对待处理处理进行处理的过程。
根据训练设备220训练得到的目标模型/规则201可以应用于不同的系统或设备中，如应用于图6所示的执行设备210，所述执行设备210可以是终端，如手机终端，平板电脑，笔记本电脑，AR/VR，车载终端等，还可以是服务器或者云端等。
在图6中，执行设备210配置输入/输出(input/output，I/O)接口212，用于与外部设备进行数据交互，用户可以通过客户设备240向I/O接口212输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待处理图像。
预处理模块213和预处理模块214用于根据I/O接口212接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块213和预处理模块214(也可以只有其中的一个预处理模块)，而直接采用计算模块211对输入数据进行处理。
在执行设备210对输入数据进行预处理，或者在执行设备210的计算模块211执行计算等相关的处理过程中，执行设备210可以调用数据存储系统250中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统250中。
最后，I/O接口212将处理结果，如上述得到待处理图像增强图像，即将得到的输出图像返回给客户设备240，从而提供给用户。
值得说明的是，训练设备220可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则201，该相应的目标模型/规则201即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在图6中所示情况下，在一种情况下，用户可以手动给定输入数据，该手动给定可以通过I/O接口212提供的界面进行操作。
另一种情况下，客户设备240可以自动地向I/O接口212发送输入数据，如果要求客户设备240自动发送输入数据需要获得用户的授权，则用户可以在客户设备240中设置相应权限。用户可以在客户设备240查看执行设备210输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备240也可以作为数据采集端，采集如图所示输入I/O接口212的输入数据及输出I/O接口212的输出结果作为新的样本数据，并存入数据库230。当然，也可以不经过客户设备240进行采集，而是由I/O接口212直接将如图所示输入I/O接口212的输入数据及输出I/O接口212的输出结果，作为新的样本数据存入数据库230。
值得注意的是，图6仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图6中，数据存储系统250相对执行设备210是外部存储器，在其它情况下，也可以将数据存储系统250置于执行设备210中。
如图6所示，根据训练设备220训练得到目标模型/规则201，该目标模型/规则201在本申请实施例中可以是图像增强模型，具体的，本申请实施例提供的图像增强模型可以是深度神经网络，卷积神经网络，或者，可以是深度卷积神经网络等。
下面结合图7重点对卷积神经网络的结构进行详细的介绍。如上文的基础概念介绍所述，卷积神经网络是一种带有卷积结构的深度神经网络，是一种深度学习(deep learning)架构，深度学习架构是指通过机器学习的算法，在不同的抽象层级上进行多个层次的学习。 作为一种深度学习架构，卷积神经网络是一种前馈(feed-forward)人工神经网络，该前馈人工神经网络中的各个神经元可以对输入其中的图像作出响应。
本申请实施例中图像增强模型的结构可以如图7所示。在图7中，卷积神经网络300可以包括输入层310，卷积层/池化层320(其中，池化层为可选的)，全连接层330以及输出层340。其中，输入层310可以获取待处理图像，并将获取到的待处理图像交由卷积层/池化层320以及全连接层330进行处理，可以得到图像的处理结果。下面对图7中的CNN 300中内部的层结构进行详细的介绍。
卷积层/池化层320：
如图7所示卷积层/池化层320可以包括如示例321-326层，举例来说：在一种实现中，321层为卷积层，322层为池化层，323层为卷积层，324层为池化层，325为卷积层，326为池化层；在另一种实现方式中，321、322为卷积层，323为池化层，324、325为卷积层，326为池化层，即卷积层的输出可以作为随后的池化层的输入，也可以作为另一个卷积层的输入以继续进行卷积操作。
下面将以卷积层321为例，介绍一层卷积层的内部工作原理。
卷积层321可以包括很多个卷积算子，卷积算子也称为核，其在图像处理中的作用相当于一个从输入图像矩阵中提取特定信息的过滤器，卷积算子本质上可以是一个权重矩阵，这个权重矩阵通常被预先定义，在对图像进行卷积操作的过程中，权重矩阵通常在输入图像上沿着水平方向一个像素接着一个像素(或两个像素接着两个像素等，这取决于步长stride的取值)的进行处理，从而完成从图像中提取特定特征的工作。该权重矩阵的大小应该与图像的大小相关，需要注意的是，权重矩阵的纵深维度(depth dimension)和输入图像的纵深维度是相同的，在进行卷积运算的过程中，权重矩阵会延伸到输入图像的整个深度。因此，和一个单一的权重矩阵进行卷积会产生一个单一纵深维度的卷积化输出，但是大多数情况下不使用单一权重矩阵，而是应用多个尺寸(行×列)相同的权重矩阵，即多个同型矩阵。每个权重矩阵的输出被堆叠起来形成卷积图像的纵深维度，这里的维度可以理解为由上面所述的“多个”来决定。
不同的权重矩阵可以用来提取图像中不同的特征，例如，一个权重矩阵用来提取图像边缘信息，另一个权重矩阵用来提取图像的特定颜色，又一个权重矩阵用来对图像中不需要的噪点进行模糊化等。该多个权重矩阵尺寸(行×列)相同，经过该多个尺寸相同的权重矩阵提取后的卷积特征图的尺寸也相同，再将提取到的多个尺寸相同的卷积特征图合并形成卷积运算的输出。
这些权重矩阵中的权重值在实际应用中需要经过大量的训练得到，通过训练得到的权重值形成的各个权重矩阵可以用来从输入图像中提取信息，从而使得卷积神经网络300进行正确的预测。
当卷积神经网络300有多个卷积层的时候，初始的卷积层(例如321)往往提取较多的一般特征，一般特征也可以称之为低级别的特征；随着卷积神经网络300深度的加深，越往后的卷积层(例如326)提取到的特征越来越复杂，比如，高级别的语义之类的特征，语义越高的特征越适用于待解决的问题。
池化层：
由于常常需要减少训练参数的数量，因此卷积层之后常常需要周期性的引入池化层， 在如图7中320所示例的321-326各层，可以是一层卷积层后面跟一层池化层，也可以是多层卷积层后面接一层或多层池化层。在图像处理过程中，池化层的目的就是减少图像的空间大小。池化层可以包括平均池化算子和/或最大池化算子，以用于对输入图像进行采样得到较小尺寸的图像。平均池化算子可以在特定范围内对图像中的像素值进行计算产生平均值作为平均池化的结果。最大池化算子可以在特定范围内取该范围内值最大的像素作为最大池化的结果。
另外，就像卷积层中用权重矩阵的大小应该与图像尺寸相关一样，池化层中的运算符也应该与图像的大小相关。通过池化层处理后输出的图像尺寸可以小于输入池化层的图像的尺寸，池化层输出的图像中每个像素点表示输入池化层的图像的对应子区域的平均值或最大值。
全连接层330：
在经过卷积层/池化层320的处理后，卷积神经网络300还不足以输出所需要的输出信息。因为如前所述，卷积层/池化层320只会提取特征，并减少输入图像带来的参数。然而为了生成最终的输出信息(所需要的类信息或其他相关信息)，卷积神经网络300需要利用全连接层330来生成一个或者一组所需要的类的数量的输出。因此，在全连接层330中可以包括多层隐含层(如图7所示的331、332至33n)以及输出层340，该多层隐含层中所包含的参数可以根据具体的任务类型的相关训练数据进行预先训练得到，例如该任务类型可以包括图像增强，图像识别，图像分类，图像检测以及图像超分辨率重建等等。
在全连接层330中的多层隐含层之后，也就是整个卷积神经网络300的最后层为输出层340，该输出层340具有类似分类交叉熵的损失函数，具体用于计算预测误差，一旦整个卷积神经网络300的前向传播(如图7由310至340方向的传播为前向传播)完成，反向传播(如图7由340至310方向的传播为反向传播)就会开始更新前面提到的各层的权重值以及偏差，以减少卷积神经网络300的损失，及卷积神经网络300通过输出层输出的结果和理想结果之间的误差。
需要说明的是，图7所示的卷积神经网络仅作为一种本申请实施例图像增强模型的结构示例，在具体的应用中，本申请实施例的图像增强方法所采用的卷积神经网络还可以以其他网络模型的形式存在。
本申请的实施例中，图像增强装置可以包括图7所示的卷积神经网络300，该图像增强装置可以对待处理图像进行图像增强处理，得到处理后的输出图像。
图8是本申请实施例提供的一种芯片的硬件结构，该芯片包括神经网络处理器400(neural-network processing unit，NPU)。该芯片可以被设置在如图6所示的执行设备210中，用以完成计算模块211的计算工作。该芯片也可以被设置在如图6所示的训练设备220中，用以完成训练设备220的训练工作并输出目标模型/规则201。如图7所示的卷积神经网络中各层的算法均可在如图8所示的芯片中得以实现。
NPU 400作为协处理器挂载到主中央处理器(central processing unit，CPU)上，由主CPU分配任务。NPU 400的核心部分为运算电路403，控制器404控制运算电路403提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路403内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路403是二维脉动阵列。运算电路403还可以是一维脉动阵列或者能够执 行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路403是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路403从权重存储器402中取矩阵B相应的数据，并缓存在运算电路403中每一个PE上。运算电路403从输入存储器401中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器408(accumulator)中。
向量计算单元407可以对运算电路403的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元407可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现种，向量计算单元能407将经处理的输出的向量存储到统一存储器406。例如，向量计算单元407可以将非线性函数应用到运算电路403的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元407生成归一化的值、合并值，或二者均有。
在一些实现中，处理过的输出的向量能够用作到运算电路403的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器406用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器405(direct memory access controller，DMAC)将外部存储器中的输入数据存入到输入存储器401和/或统一存储器406、将外部存储器中的权重数据存入权重存储器402，以及将统一存储器406中的数据存入外部存储器。
总线接口单元410(bus interface unit，BIU)，用于通过总线实现主CPU、DMAC和取指存储器409之间进行交互。
与控制器404连接的取指存储器409(instruction fetch buffer)，用于存储控制器404使用的指令。控制器404用于调用取指存储器409中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器406，输入存储器401，权重存储器402以及取指存储器409均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
其中，图7所示的卷积神经网络中各层的运算可以由运算电路403或向量计算单元407执行。
上文中介绍的图6中的执行设备210能够执行本申请实施例的图像增强方法的各个步骤，图7所示的CNN模型和图8所示的芯片也可以用于执行本申请实施例的图像增强方法的各个步骤。
图9所示是本申请实施例提供了一种系统架构500。该系统架构包括本地设备520、本地设备530以及执行设备510和数据存储系统550，其中，本地设备520和本地设备530通过通信网络与执行设备510连接。
示例性地，执行设备510可以由一个或多个服务器实现。
可选的，执行设备510可以与其它计算设备配合使用。例如：数据存储器、路由器、负载均衡器等设备。执行设备510可以布置在一个物理站点上，或者分布在多个物理站点上。执行设备510可以使用数据存储系统550中的数据，或者调用数据存储系统550中的程序代码来实现本申请实施例的图像增强方法。
需要说明的是，上述执行设备510也可以称为云端设备，此时执行设备510可以部署在云端。
具体地，执行设备510可以执行以下过程：获取待处理图像以及所述待处理图像对应的第一高动态范围HDR图像，其中，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；将所述待处理图像输入神经网络模型得到所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域；将所述第一HDR图像输入所述神经网络模型进行超分辨率处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行处理，得到所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像为所述第一分辨率的HDR图像。
在一种可能的实现方式中，本申请实施例的图像增强方法可以是在云端执行的离线方法，比如，可以由上述执行设备510中执行本申请实施例的图像增强方法。
在一种可能的实现方式中，本申请实施例的图像增强方法可以是由本地设备520或者本地设备530执行。
在本申请的实施例中，可以对获取的画质较差的待处理图像进行图像增强，从而得到待处理图像在图像细节、图像颜色以及图像亮度等方面性能均得到提升的输出图像即第二HDR图像。
例如，用户可以操作各自的用户设备(例如，本地设备520和本地设备530)与执行设备510进行交互。每个本地设备可以表示任何计算设备，例如，个人计算机、计算机工作站、智能手机、平板电脑、智能摄像头、智能汽车或其他类型蜂窝电话、媒体消费设备、可穿戴设备、机顶盒、游戏机等。
每个用户的本地设备可以通过任何通信机制/通信标准的通信网络与执行设备510进行交互，通信网络可以是广域网、局域网、点对点连接等方式，或它们的任意组合。
在一种实现方式中，本地设备520、本地设备530可以从执行设备510获取到上述神经网络模型的相关参数，将神经网络模型部署在本地设备520、本地设备530上，利用该神经网络模型进行图像增强处理等。
在另一种实现中，执行设备510上可以直接部署神经网络模型，执行设备510通过从本地设备520和本地设备530获取待处理图像以及待处理图像对应的第一HDR图像，并根据神经网络模型进行图像增强处理，得到待处理图像对应的第二HDR图像。
例如，上述神经网络模型可以是本申请实施例中的超分网络，比如，参见后续图11、图13以及图15中的超分网络720。
目前，为了满足高分辨率图像的实时增强，提出了“下采样+HDR+超分”的方式来节省计算开销，以达到实时图像质量增强的目的。可以向预先训练的卷积神经网络中输入原始分辨率图像以及低分辨率图像增强图像，从而对图像进行超分处理(即还原图像的原始分辨率)，得到输出的原始分辨率增强图像。通过上述方式进行高分辨率的图像增强，一 方面由于出于节省计算开销的目的，基于深度学习的超分方法通常仅对图像的亮度通道进行复杂超分过程，而对于图像的颜色通道可以采用简单上采样，从而导致无法保证原始分辨率增强图像与低分辨率增强图像在颜色、亮度、对比度、饱和度和动态范围等方面的一致性；另一方面，上述图像增强方法中关于如何根据原始分辨率图像和低分辨率增强图像进行超分处理的过程未进行详细说明；若采用简单的超分处理，则会存在图像模糊、纹理细节的丢失等问题；若采用过于复杂的超分过程，则会存在整个卷积神经网络处理时间增加，同时可能引入棋盘格、伪像(Artifact)等问题。
有鉴于此，本申请实施例提供了一种图像增强方法以及图像增强装置，通过获取待处理图像以及待处理图像的第一HDR图像，其中，第一HDR图像即待处理图像对应的低分辨率增强图像，在神经网络模型中通过待处理图像的颜色图像特征对第一HDR图像进行超分辨率处理，从而得到与待处理图像的分辨率大小相同的增强图像即第二HDR图像；在本申请实施例中在“下采样+HDR+超分”的基础上引入了颜色注意力机制，即颜色图像特征从而使得在进行图像增强的过程中，提高神经网络模型对于待处理图像中的困难区域(过亮、过暗区域)在颜色、亮度、对比度以及饱和度等方面的恢复效果，从而提高图像增强的效果。
图10示出了本申请实施例提供的图像增强方法的示意性流程图，图10所示图像增强方法可以由图像增强装置执行，该图像增强装置具体可以是图6中的执行设备210，也可以是图9中的执行设备510或者本地设备。图10所示的方法包括步骤610至630，下面分别对步骤610至630进行详细的介绍。
步骤610、获取待处理图像对应的第一高动态范围HDR图像以及待处理图像的颜色图像特征。
其中，颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率。
例如，上述待处理图像可以是指具有图像增强需求的待处理图像；比如，待处理图像可以是指分辨率较高并且画质较差的原始图像；例如，可以是指受到天气、距离、拍摄环境等因素的影响，获取的待处理图像存在图像画质较低的问题；图像画质较低包括但不限于：图像画质较低，包括但不限于图像模糊、或者图像颜色、亮度、饱和度、对比度、动态范围较差等。
例如，上述待处理图像可以是电子设备通过摄像头拍摄到的图像，或者，上述待处理图像还可以是从电子设备内部获得的图像(例如，电子设备的相册中存储的图像，或者，电子设备从云端获取的图片)。例如，电子设备可以是图9所示的本地设备或者执行设备中的任意一个。
示例性地，第一HDR图像可以是通过对待处理图像进行下采样处理以及HDR增强后得到的图像；其中，高动态范围(high dynamic range，HDR)图像与普通动态范围(standard dynamic range，SDR)图像相比可以提供更多的动态范围和图像色彩，即在HDR图像中可以包括更多图像细节。
在本申请的实施例中，可以通过目标学习的方式或者传统的方法获取待处理图像的颜色图像特征。
示例性地，可以采用目标学习的方式或传统方法对待处理图像进行处理得到待处理图像的颜色图像特征。
其中，目标学习方法的核心在于提供这样的学习目标使得可以衡量输入图像与真值图像(学习目标)的差异性，这样的差异性通常体现于图像的亮区和阴影(暗区)区域。
此外，还可以采用传统的方法获取待处理图像的颜色图像特征，此时可以不再需要真值图像，也不存在学习过程；比如，可以通过将彩色输入图像转换为灰度图像或转换到具有亮度通道的其他颜色域(例如YUV域、LAB域)，然后设置最低阈值和最高阈值，通道(例如灰度、YUV域中的Y通道、LAB域的L通道)中低于和高于最低阈值和最高阈值的部分作为权值更大的区域，其他区域置为权值更小的区域。
在一个示例中，可以通过待处理图像经过自编码解码子网络得到待处理图像的颜色图像特征。
在一个示例中，由于受限于终端设备的运行速度、运行显存以及功耗等因素，为了节约计算量；可以先对待处理图像进行下采样处理得到低分辨率图像，通过待处理图像对应的低分辨率图像与自编码解码子网络得到颜色图像特征；例如，参见后续图12所示通过待处理图像的低分辨图像得到待处理图像的颜色图像特征。
需要说明的是，上述颜色图像特征又可以称为颜色引导图，通过颜色图像特征可以对于输入神经网络模型中的待处理图像中的困难区域(例如，亮区或者暗区)具有更高的引导信息，使得神经网络模型在学习过程中可以更加注重困难区域的增强效果。
步骤620、将第一HDR图像输入神经网络模型进行超分辨率处理。
其中，上述超分辨率处理可以是指卷积操作与上采样操作，从而使得第一HDR图像的分辨率大小与待处理图像的分辨率大小相同。
进一步，由于为了使得神经网络模型对第一HDR图像进行超分辨处理时更加注重边缘区域以及纹理特征，则可以在对第一HDR图像进行超分辨率处理时引入纹理注意力机制。
可选地，在一种可能的实现方式中，可以将待处理图像输入所述神经网络模型得到待处理图像的纹理图像特征，其中，纹理图像特征可以用于指示待处理图像的边缘区域以及纹理区域；将第一HDR图像输入神经网络模型进行超分辨率处理可以是指通过神经网络模型根据纹理图像特征对第一HDR图像进行超分辨率处理。
需要说明的是，在神经网络模型中引入纹理注意力机制是为了使得神经网络模型对于图像中的边缘与纹理等细节方面的学习；通过使用纹理图像特征又称为纹理引导图可以提升神经网络模型对于纹理引导图中权重较高区域的学习；从而对第一HDR图像超分处理的过程能够提高超分算法对于图像纹理细节的恢复能力，避免超分处理后引入的图像模糊或者其它感官差异。
进一步地，在对第一HDR图像进行超分处理的过程中还可以引入待处理图像的多尺度图像特征；其中，多尺度图像特征可以是指不同分辨率的图像特征，通过引入多尺度的图像特征可以使得神经网络模型中引入更多的图像细节信息，有利于保证第一HDR图像在细节等方面的还原。
在本申请的实施例中，在对第一HDR图像进行超分处理的过程中可以基于双注意力机制进行超分还原处理，其中，双注意力机制可以包括纹理注意力机制与颜色注意力机制。 通过将颜色注意力机制与纹理注意力机制用于超分处理过程中，使得最终得到的增强图像即待处理图像的第二HDR图像在颜色、亮度、饱和度、对比度以及纹理细节等方面与真值图像相同或者接近，提升了图像增强的效果。
可选地，在一种可能的实现方式中，可以将待处理图像输入神经网络模型得到待处理图像的多尺度图像特征，其中，多尺度图像特征中的任意一个图像特征的尺度大小不同；可以通过神经网络模型根据纹理图像特征与多尺度图像特征对第一HDR图像进行超分辨率处理，其中，多尺度图像特征用于指示在不同尺度的情况下待处理图像的图像信息。例如，可以参见后续图16所示的自引导多级超分处理的示意性流程图。
应理解，多尺度图像特征可以用于指示在不同尺度的情况下待处理图像的图像信息，其中，不同尺度可以是指不同的分辨率大小，图像信息可以是指图像中的高频信息；比如，高频信息可以包括图像中的边缘信息、细节信息以及纹理信息中的一种或者多种。
示例性地，通过神经网络模型根据纹理图像特征与多尺度图像特征对第一HDR图像进行超分辨率处理可以包括：通过神经网络模型对纹理图像特征进行尺度调整处理，得到第一尺度的纹理图像特征，其中，第一尺度的纹理图像特征与多尺度图像中的第一尺度图像特征的尺度相同；通过神经网络模型对第一HDR图像与第一尺度的纹理图像特征进行点乘操作，得到第三图像特征；通过神经网络模型对第三图像特征与所述第一尺度图像特征进行合通道操作、卷积操作以及上采样操作。
在本申请的实施例中，提出了一种多级的超分处理方法即从原始分辨率输入图像中通过特征降维的方式，提取多尺度的特征信息；超分处理过程中可以基于第一HDR图像即低分辨率HDR图像以及待处理图像即原始输入图像的多尺度特征进行多尺度的超分处理还原，低分辨率HDR图像与提取的原始输入图像对应尺度的特征信息可以进行融合，从而得到待处理图像的第二HDR图像即对待处理图像进行增强处理后的增强图像。
步骤630、通过神经网络模型对超分辨率处理后的第一HDR图像与颜色图像特征进行处理，得到待处理图像对应的第二HDR图像。
其中，第二HDR图像为第一分辨率的HDR图像，即第二HDR图像与待处理图像的分辨率大小相同，第二HDR图像可以是指对待处理图像进行增强处理后得到的增强图像。
示例性地，可以通过神经网络模型对超分辨率处理后的第一HDR图像与颜色图像特征进行点乘操作与卷积操作，得到第二HDR图像。
其中，上述点乘操作可以是指逐像素点的相乘。
图11是本申请实施例提供的一种图像增强方法的系统架构的示意图。如图11所示，系统架构中可以包括HDR网络710与超分辨率(super resolution，SR)网络720；其中，HDR网络710中可以包括下采样单元711与HDR增强单元712；SR网络720可以包括颜色注意力单元721、超分处理单元722和HDR校正单元723。
示例性地，上述下采样单元711可以用于对输入图像，比如高分辨率图像进行下采样处理，得到低分辨率图像或者小分辨率图像，例如，1080P分辨率图像。
示例性地，上述HDR增强单元712可以用于对下采样处理后的低分辨率图像通过HDR增加方法进行处理，得到低分辨率HDR增强图像。
其中，HDR增强方法可以是采用任意具有图像增强的神经网络模型进行图像增强处理；例如，可以采用Unet或者高动态范围网络(High dynamic range net，HDRNet)为基 础框架的各类神经网络。.
示例性地，上述颜色注意力单元721可以用于根据输入图像(例如，原始分辨率图像)通过使用某种计算方式提取输入图像的颜色引导图；其中，颜色引导图可以是输入图像对应的颜色图像特征。
在本申请的实施例中，可以通过目标学习的方式或者传统的方法获取待处理图像的颜色图像特征。
示例性地，可以采用目标学习的方式或传统方法对待处理图像进行处理得到待处理图像的颜色图像特征。
其中，目标学习方法的核心在于提供这样的学习目标使得可以衡量输入图像与真值图像(学习目标)的差异性，这样的差异性通常体现于图像的亮区和阴影(暗区)区域。
此外，还可以采用传统的方法获取待处理图像的颜色图像特征，此时可以不再需要真值图像，也不存在学习过程；比如，可以通过将彩色输入图像转换为灰度图像或转换到具有亮度通道的其他颜色域(例如YUV域、LAB域)，然后设置最低阈值和最高阈值，通道(例如灰度、YUV域中的Y通道、LAB域的L通道)中低于和高于最低阈值和最高阈值的部分作为权值更大的区域，其他区域置为权值更小的区域。
应理解，上述颜色引导图对于输入图像中的困难区域(例如，亮区或者暗区)具有更高的引导信息，使得SR网络720在学习过程中可以更加注重困难区域的增强效果。
示例性地，HDR校正单元723可以用于根据HDR增强单元712输出的低分辨率增强图像与超分处理单元722输出的图像，结合颜色注意力单元721所得到的颜色引导图，利用CNN对于超分后的图像在颜色、亮度、对比度、饱和度等方面进行校正，从而保证HDR效果在经过超分处理单元722前后的一致性并加强对于图像困难区域的增强效果。
下面结合图12对通过颜色引导图进行HDR校正的流程进行具体描述。图12是本申请实施例提供的通过颜色引导图进行HDR校正的示意图。
其中，图12所示的颜色引导图，即输入图像对应的颜色图像特征可以是采用预训练网络进行提取，比如可以采用自编码解码网络(Encoder-Decoder Network)作为用于提取颜色引导图的网络，自编码解码网络的输入图像可以是指低分辨率输入图像，经过自编码解码网络可以获得颜色引导图；其中，自编码解码网络在预训练过程，可以通过设计目标函数保证在输入图像中的困难区域(例如，亮区或者暗区)对应在颜色引导图中具有更大的权重。
例如，可以采用以下目标函数对自编码解码网络进行训练：
Figure PCTCN2021076859-appb-000011
其中，
Figure PCTCN2021076859-appb-000012
表示对输入图像的通道维度求取最大值；A map表示颜色引导图；I in表示自编码解码网络的输入图像特征，比如，可以是指原始分辨率图像特征或者低分辨率图像特征；I target表示与输入图像同分辨率的真值图像。
如上所示的目标函数可以使得自编码解码网络对于输入图像与目标图像在区别越大的某个区域，颜色引导图中将在此区域体现出更大的权重。
在SR网络720的训练过程中，颜色引导图提取网络例如图12所示的自编码解码网络 可以不再参与训练；在输入图像中的较暗区域以及图像颜色变化更多的区域，颜色引导图中对应可以表现出更大的权重，从而引导后端HDR校正单元723对于这些区域的重点学习。
在一个示例中，如图12所示，输入图像可以是指原始分辨图像；或者，为了节约计算量，输入图像可以是对原始分辨率图像进行下采样处理后的低分辨率图像。
需要说明的是，若上述颜色引导图是通过原始分辨率图像得到的，则颜色引导图也可以不需要进行上采样操作。
示例性地，上述HDR校正单元723可以为预先训练的CNN网络，其中，HDR校正单元723可以接收三部分输入数据，其中，第一输入数据为经过超分处理单元722输出的第一原始分辨率增强图像，第一原始分辨增强图像可以是指对HDR增强单元712输出的低分辨率增强图像进行上采样处理后得到的图像；第二输入数据为HDR增强单元712输出的低分辨率增强图像；第三输入数据为颜色注意力单元721输出的颜色注意力引导图；在HDR校正单元723中根据上述三部分输入数据进行HDR校正，输出第二原始分辨率增强图像。
进一步地，为了使得超分处理单元722在进行超分处理时更加注重学习输入图像的中的纹理与边缘，因此可以在系统架构中引入纹理注意力单元，如图13所示。
图13中还可以包括纹理注意力单元724，纹理注意力单元724可以用于根据输入图像即原始分辨率图像，使用某种计算方式提取纹理引导图(即图像的高频信息)。超分处理单元722在进行上采样与卷积操作时还可以根据纹理注意力单元724输出的纹理引导图来加强超分过程中对于图像细节纹理以及边缘区域的恢复。
下面结合图14对获取纹理引导图的过程进行描述。图14是本申请实施例提供的纹理引导图的提取过程的示意图。
示例性地，纹理注意力单元的输入数据可以为无颜色信息的输入图像，例如，灰度图或YUV色域的Y通道；比如，输入数据为Y通道图像特征，经过高斯滤波可以滤除图像中高频信息，经过滤波得到输出数据与输入图像进行负向残差从而可以得到图像高频信息，即纹理引导图，其中，图像中的纹理和边缘通常存在于图像的高频信息中如图14所示。
需要说明的是，纹理引导图的获取过程不限于此类方法，还可以采用边缘检测等算法获取纹理引导图，本申请对此不作任何限定。
在本申请的实施例中，纹理注意力单元724可以用于提取图像中高频信息作为纹理引导图，提取的纹理引导图将作用于超分处理单元722，从而能够可提升SR网络200对于引导图中权重较高区域的学习，能够增强SR网络720对于图像边缘和纹理等细节方面的学习。
在一个示例中，上述超分处理单元可以采用自引导多级超分单元，即超分过程可以存在多个图像尺寸大小递进式增长至与原始辨率输入图像相同的尺寸大小，如图15所示。
图15是本申请实施例提供的图像增强方法的系统架构的示意图。如图15所示，系统架构中可以包括HDR网络710与超分辨率(super resolution，SR)网络720；其中，HDR网络710中可以包括下采样单元711与HDR增强单元712；SR网络720中可以包括颜色注意力单元721、自引导多级超分单元722、HDR校正单元723、纹理注意力单元724以 及多尺度自引导特征提取单元725。
其中，上述HDR网络710用于根据输入的原始分辨率输入图像I(例如，4K分辨率)，进行HDR增强从而得到HDR增强后的低分辨率HDR图像(例如，1080P分辨率)。
例如，输入图像可以为原始分辨率图像，比如，高分辨率图像或者全分辨率图像；将输入图像通过下采样单元711的处理后得到低分辨率图像或者小分辨率图像；将低分辨率图像输入HDR增强单元712中进行处理，得到输出的低分辨率HDR图像。
其中，上述SR网络720用于根据输入的低分辨率HDR图像以及原始分辨率图像，通过上述SR网络720中包括的颜色注意力单元721至多尺度自引导特征提取单元725对输入图像进行超分处理从而还原图像的原始分辨率，得到输出的原始分辨率增强图像(例如，4K分辨率)。
示例性地，如图15所示，SR网络720中的多尺度自引导特征提取单元725可以用于根据输入的原始分辨率图像，通过卷积神经网络进行特征提取，从而能够获取多个尺度的自引导图。
需要说明的是，上述多个尺度的自引导图可以是指通过输入图形通过下采样和不同深度的卷积处理得到的不同尺度的图像特征，其中，下采样方式可以包括但不限于采样插值方法或像素重排(space to depth)操作等，多个尺度可以是指多个分辨率大小。
示例性地，上述颜色注意力单元721可以用于根据输入图像(例如，原始分辨率图像)通过使用某种计算方式提取输入图像的颜色引导图；其中，颜色引导图可以是输入图像对应的颜色图像特征。
应理解，上述颜色引导图对于输入图像中的困难区域(例如，亮区或者暗区)具有更高的引导信息，使得SR网络720在学习过程中可以更加注重困难区域的增强效果。
示例性地，上述纹理注意力单元724可以用于根据输入图像即原始分辨率图像，使用某种计算方式提取纹理引导图(即图像的高频信息)。具体流程可以参见上述图14，此处不再赘述。
示例性地，如图15所示，自引导多级超分单元722中的输入数据可以包括多尺度自引导特征提取单元725输入的不同尺度的图像特征、纹理注意力单元724输出的输入图像对应的纹理图像特征以及HDR增强单元712输入的低分辨率HDR增强图像，自引导多级超分单元722可以对上述输入数据进行上采样与卷积操作进行图像分辨率还原处理，从而得到超分还原图像H SR。
需要说明的是，在上述超分处理过程中引入了纹理注意力图像特征可以使得在进行图像超分处理时，加强超分过程对于图像细节纹理和边缘区域的恢复；此外，输入多尺度的图像特征可以在进行超分处理的过程中使得输入更多的图像细节与信息。
示例性地，上述HDR校正单元723可以用于对上述超分还原图像H SR进行在颜色、亮度、对比度、饱和度等方面进行校正，从而得到原始分辨率增强图像，并且使得原始分辨率增强图像与真值图像无限接近；其中，HDR校正单元723的输入数据可以是超分还原图像H SR、低分辨率HDR图像H L以及颜色注意力图像特征，从而对输入数据进行卷积操作，使得得到的图像特征与真值图像特征相同或者偏差在预设范围内。
例如，图16是本申请实施例提供的自引导多级超分处理的示意性流程图。
如图16所示，假设输入的原始分辨图像为W×H，通过下采样以及特征提取处理得到 的多尺度自引导图为分辨率为W/2×H/2的图像特征以及分辨率为W/4×H/4的图像特征；可以将多尺度引导图作为先验信息与各级超分过程的输出特征进行拼接，引导超分过程；比如，对于分辨率为W/2×H/2尺度，先对纹理注意力图像特征进行尺度调整，得到分辨率为W/4×H/4的纹理注意力图像特征；将W/4×H/4的纹理注意力图像特征与分辨率为W/4×H/4的HDR增强图像进行点乘操作，即可以是指逐像素点相乘；接着，将点乘处理后得到的图像特征与原始分辨率输入图像的分辨率为W/4×H/4的图像特征进行连接操作，即合并通道处理；同理，对于分辨率为W/2×H/2尺度，先对纹理注意力图像特征进行尺度调整，得到分辨率为W/2×H/2的纹理注意力图像特征；将W/2×H/2的纹理注意力图像特征与分辨率为W/2×H/2的HDR增强图像进行点乘操作，即可以是指逐像素点相乘；接着将点乘处理后得到的图像特征与原始分辨率输入图像的分辨率为W/2×H/2的图像特征进行连接操作，即合并通道处理；最后对分辨率为W/2×H/2合通道处理后的图像特征进行上采样与卷积操作，得到原始分辨率增强图像。
应理解，上述是以分辨率为W/4×H/4、分辨率为W/2×H/2以及分辨率为W×H的3级渐进式超分过程进行举例进行说明，还可以其它的多级渐进式还原，本申请对此不作任何限定。
示例性地，图17所示为引入多尺度图像特征后通过颜色引导图进行HDR校正的示意图。如图17所示，以YUV域图像进行超分为例进行详细描述，其中，低分辨率图像可以是指低分辨率HDR图像的U通道以及V通道的图像特征，原始分辨率图像可以是指通过自引导多级超分单元722输出的图像提取的Y通道的图像特征；对低分辨率的U通道以及V通道的图像特征进行上采样操作，使得图像特征的尺度与原始分辨率图像尺度相同；接着，将上采样处理后的U通道以及V通道的图像特征与原始分辨率图像的Y通道图像特征进行连接操作，即合通道操作；然后将合通道处理后的YUV图像转换为RGB图像；将颜色引导图进行上采样操作使得颜色引导图的尺度与原始分辨率图像相同；对转换后的RGB图像与上采样操作后的颜色引导图进行点乘操作，即逐像素点的相乘；进而对点乘操作后的图像进行HDR校正即卷积操作，使得最终输出的原始分辨率增强图像与真值图像的差异小于预设阈值。
需要说明的是，若上述颜色引导图是通过原始分辨率图像得到的，则颜色引导图也可以不需要进行上采样操作。
应理解，图17中的自编码解码网络与图12中的可以相同，此处不再赘述；同理，HDR校正单元也可以与图12中的HDR校正单元相同，可以参见图12中的描述，此处不再赘述。
表1
Figure PCTCN2021076859-appb-000013
表1是本申请实施例的图像增强方法与基准算法的性能测试结果。表1中所示为本申 请中的模型与现有的几种模型在进行图像增强处理时的性能测试结果，其中，测试指标为峰值信噪比(Peak signal to noise ratio，PSNR)和结构相似性(Structural similarity index，SSIM)以及计算量，其中，计算量可以用于表示执行乘累加操作次数(Multiply accumulate，MAC)，1G＝10^ 9。
从表1所示的测试实验结果可以看出，本方案的神经网络模型在定量指标PSNR和SSIM远超全局缩放模型(Range scaling global U-Net，RSGUnet)与自导模型(Self-guided network，SGN)，同时相比HDRNet在PSNR和SSIM损失1.85％和0.26％的结果下，单帧处理时间缩短56.2％，单帧计算量减少86.6％。在同等测试环境中，可以看出本申请实施例提供的图像增强方法具有更优的处理速度和计算开销，可同时满足图像质量增强效果并实时处理的需求。
图18至图20是本申请实施例提供的视觉质量图像增强质量的测评结果的示意图。其中，图18中的(a)为输入图像即待处理图像；图18中的(b)为输入图像对应的真值图像；图18中的(c)为通过本申请的图像增强方法对输入图像进行增强处理得到的输出图像；图18中的(d)为通过HDRNet模型对输入图像进行增强处理得到的输出图像；图18中的(e)为通过RSGUnet模型对输入图像进行增强处理得到的输出图像；图18中的(f)为通过SGN模型对输入图像进行增强处理得到的输出图像；从图18中的(a)至图18中的(f)可以看出，通过本申请实施例得到的输出图像与真值图像最接近，不存在伪像等问题；同理，通过图19中的(a)至图19中的(f)可以看出，通过本申请实施例得到的输出图像对于天空光晕问题，即本申请实施例的图像增强方法能够在保证HDR效果的同时，削弱光晕问题；同理，通过图20中的(a)至图20中的(f)可以看出，通过本申请实施例得到的输出图像对于高亮部分，颜色更接近于真值图像，同时叶脉的纹理细节恢复得更加清晰。
应理解，上述举例说明是为了帮助本领域技术人员理解本申请实施例，而非要将本申请实施例限于所例示的具体数值或具体场景。本领域技术人员根据所给出的上述举例说明，显然可以进行各种等价的修改或变化，这样的修改或变化也落入本申请实施例的范围内。
上文结合图1至图20，详细描述了本申请实施例提供的图像增强方法，下面将结合图21和图22，详细描述本申请的装置实施例。应理解，本申请实施例中的图像增强装置可以执行前述本申请实施例的各种图像增强方法，即以下各种产品的具体工作过程，可以参考前述方法实施例中的对应过程。
图21是本申请实施例提供的图像增强装置的示意性框图。应理解，图像增强装置800可以执行图10所示的图像增强方法。该图像增强装置800包括：获取单元810和处理单元820。
其中，所述获取单元810，用于获取待处理图像对应的第一高动态范围HDR图像与所述待处理图像的颜色图像特征，其中，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域；所述处理单元820，用于将所述第一HDR图像输入神经网络模型进行超分辨处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到 所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像。
可选地，作为一个实施例，所述获取单元810还用于：
获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
所述处理单元820具体用于：
通过所述神经网络模型根据所述纹理图像特征对所述第一HDR图像进行超分辨率处理。
可选地，作为一个实施例，所述获取单元810还用于：
获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
所述处理单元820具体用于：
通过所述神经网络模型根据所述多尺度图像特征对所述第一HDR图像进行超分辨率处理。
可选地，作为一个实施例，所述获取单元810还用于：
获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
所述处理单元820具体用于：
通过所述神经网络模型根据所述纹理图像特征与所述多尺度特征对所述第一HDR图像进行超分辨率处理。
可选地，作为一个实施例，所述获取单元810具体用于：
获取第一尺度的纹理图像特征，其中，所述第一尺度的纹理图像特征与所述多尺度图像中的第一尺度图像特征的尺度相同；
所述处理单元820具体用于：
通过所述神经网络模型对所述第一HDR图像与所述第一尺度的纹理图像特征进行点乘操作，得到第三图像特征；通过所述神经网络模型对所述第三图像特征与所述第一尺度图像特征进行合通道操作、卷积操作以及上采样操作。
可选地，作为一个实施例，所述处理单元820具体用于：
所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行点乘操作与卷积操作，得到所述第二HDR图像。
需要说明的是，上述图像增强装置800以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现，对此不作具体限定。
例如，“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
图22是本申请实施例提供的图像增强装置的硬件结构示意图。如图22所示的图像增强装置900(该装置900具体可以是一种计算机设备)包括存储器901、处理器902、通信接口903以及总线904。其中，存储器901、处理器902、通信接口903通过总线904实现彼此之间的通信连接。
存储器901可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器901可以存储程序，当存储器901中存储的程序被处理器902执行时，处理器902用于执行本申请实施例的图像增强方法的各个步骤，例如，执行图10至图17所示的各个步骤。
应理解，本申请实施例所示的图像增强装置可以是服务器，例如，可以是云端的服务器，或者，也可以是配置于云端的服务器中的芯片；或者，本申请实施例所示的图像增强装置可以是智能终端，也可以是配置于智能终端中的芯片。
上述本申请实施例揭示的图像增强方法可以应用于处理器902中，或者由处理器902实现。处理器902可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述图像增强方法的各步骤可以通过处理器902中的硬件的集成逻辑电路或者软件形式的指令完成。例如，处理器902可以是包含图8所示的NPU的芯片。
上述的处理器902可以是中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器901，处理器902读取存储器901中的指令，结合其硬件完成本申请实施中图21所示的图像增强装置中包括的单元所需执行的功能，或者，执行本申请方法实施例的图10至图17所示的图像增强方法的各个步骤。
通信接口903使用例如但不限于收发器一类的收发装置，来实现装置900与其他设备或通信网络之间的通信。
总线904可包括在图像增强装置900各个部件(例如，存储器901、处理器902、通信接口903)之间传送信息的通路。
应注意，尽管上述图像增强装置900仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，图像增强装置900还可以包括实现正常运行所必须的其他器件。同时，根据具体需要本领域的技术人员应当理解，上述图像增强装 置900还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，上述图像增强装置900也可仅仅包括实现本申请实施例所必须的器件，而不必包括图22中所示的全部器件。
本申请实施例还提供一种芯片，该芯片包括收发单元和处理单元。其中，收发单元可以是输入输出电路、通信接口；处理单元为该芯片上集成的处理器或者微处理器或者集成电路。该芯片可以执行上述方法实施例中的图像增强方法。
本申请实施例还提供一种计算机可读存储介质，其上存储有指令，该指令被执行时执行上述方法实施例中的图像增强方法。
本申请实施例还提供一种包含指令的计算机程序产品，该指令被执行时执行上述方法实施例中的图像增强方法。
还应理解，本申请实施例中，该存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。处理器的一部分还可以包括非易失性随机存取存储器。例如，处理器还可以存储设备类型的信息。
还应理解，本申请实施例中，该存储器可以包括只读存储器和随机存取存储器，并向处理器提供指令和数据。处理器的一部分还可以包括非易失性随机存取存储器。例如，处理器还可以存储设备类型的信息。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能模块可以集成在一个处理单元中，也可以是各 个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (14)

  1. 一种图像增强方法，其特征在于，包括：
    获取待处理图像对应的第一高动态范围HDR图像与所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；
    将所述第一HDR图像输入神经网络模型进行超分处理；
    通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像。
  2. 如权利要求1所述的图像增强方法，其特征在于，还包括：
    获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
    所述将所述第一HDR图像输入神经网络模型进行超分辨率处理，包括：
    通过所述神经网络模型根据所述纹理图像特征对所述第一HDR图像进行超分辨率处理。
  3. 如权利要求1所述的图像增强方法，其特征在于，还包括：
    获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
    所述将所述第一HDR图像输入神经网络模型进行超分辨处理：
    通过所述神经网络模型根据所述多尺度图像特征对所述第一HDR图像进行超分辨率处理。
  4. 如权利要求1所述的图像增强方法，其特征在于，还包括：
    获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
    获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
    所述将所述第一HDR图像输入神经网络模型进行超分辨率处理，包括：
    通过所述神经网络模型根据所述纹理图像特征与所述多尺度特征对所述第一HDR图像进行超分辨率处理。
  5. 如权利要求4所述的图像增强方法，其特征在于，所述通过所述神经网络模型根据所述纹理图像特征与所述多尺度图像特征对所述第一HDR图像进行超分辨率处理，包括：
    获取第一尺度的纹理图像特征，其中，所述第一尺度的纹理图像特征与所述多尺度图像中的第一尺度图像特征的尺度相同；
    通过所述神经网络模型对所述第一HDR图像与所述第一尺度的纹理图像特征进行点 乘操作，得到第三图像特征；
    通过所述神经网络模型对所述第三图像特征与所述第一尺度图像特征进行合通道操作、卷积操作以及上采样操作。
  6. 如权利要求1至5中任一项所述的图像增强方法，其特征在于，所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，包括：
    所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行点乘操作与卷积操作，得到所述第二HDR图像。
  7. 一种图像增强装置，其特征在于，包括：
    获取单元，用于获取待处理图像对应的第一高动态范围HDR图像与所述待处理图像的颜色图像特征，其中，所述颜色图像特征用于指示所述待处理图像中的不同亮度区域或者不同颜色变化区域，所述待处理图像为第一分辨率的图像，所述第一HDR图像为第二分辨率的图像，所述第一分辨率大于所述第二分辨率；
    处理单元，用于将所述第一HDR图像输入神经网络模型进行超分辨处理；通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行图像增强处理，得到所述待处理图像对应的第二HDR图像，其中，所述第二HDR图像是指分辨率大小为所述第一分辨率的HDR图像。
  8. 如权利要求7所述的图像增强装置，其特征在于，所述获取单元还用于：
    获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
    所述处理单元具体用于：
    通过所述神经网络模型根据所述纹理图像特征对所述第一HDR图像进行超分辨率处理。
  9. 如权利要求7所述的图像增强装置，其特征在于，所述获取单元还用于：
    获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
    所述处理单元具体用于：
    通过所述神经网络模型根据所述多尺度图像特征对所述第一HDR图像进行超分辨率处理。
  10. 如权利要求7所述的图像增强装置，其特征在于，所述获取单元还用于：
    获取所述待处理图像的纹理图像特征，其中，所述纹理图像特征用于指示所述待处理图像的边缘区域或者纹理区域；
    获取所述待处理图像的多尺度图像特征，其中，所述多尺度图像特征用于指示在不同尺度的情况下所述待处理图像的图像信息，所述多尺度图像特征中的任意一个图像特征的尺度大小不同；
    所述处理单元具体用于：
    通过所述神经网络模型根据所述纹理图像特征与所述多尺度特征对所述第一HDR图像进行超分辨率处理。
  11. 如权利要求10所述的图像增强装置，其特征在于，所述获取单元具体用于：
    获取第一尺度的纹理图像特征，其中，所述第一尺度的纹理图像特征与所述多尺度图像中的第一尺度图像特征的尺度相同；
    所述处理单元具体用于：
    通过所述神经网络模型对所述第一HDR图像与所述第一尺度的纹理图像特征进行点乘操作，得到第三图像特征；
    通过所述神经网络模型对所述第三图像特征与所述第一尺度图像特征进行合通道操作、卷积操作以及上采样操作。
  12. 如权利要求7至11中任一项所述的图像增强装置，其特征在于，所述处理单元具体用于：
    所述通过所述神经网络模型对所述超分辨率处理后的所述第一HDR图像与所述颜色图像特征进行点乘操作与卷积操作，得到所述第二HDR图像。
  13. 一种图像增强装置，其特征在于，包括：
    存储器，用于存储程序；
    处理器，用于执行所述存储器存储的程序，当所述处理器执行所述存储器存储的程序时，所述处理器用于执行权利要求1至6中任一项所述的图像增强方法。
  14. 一种计算机可读介质，其特征在于，所述计算机可读介质存储有程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行如权利要求1至6中任一项所述的图像增强方法。
PCT/CN2021/076859 2020-02-19 2021-02-19 图像增强方法以及图像增强装置 WO2021164731A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP21757305.4A EP4105877A4 (en) 2020-02-19 2021-02-19 IMAGE ENHANCEMENT METHOD AND IMAGE ENHANCEMENT APPARATUS

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010102590.4A CN113284054B (zh) 2020-02-19 2020-02-19 图像增强方法以及图像增强装置
CN202010102590.4 2020-02-19

Publications (1)

Publication Number Publication Date
WO2021164731A1 true WO2021164731A1 (zh) 2021-08-26

Family

ID=77275122

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/076859 WO2021164731A1 (zh) 2020-02-19 2021-02-19 图像增强方法以及图像增强装置

Country Status (3)

Country Link
EP (1) EP4105877A4 (zh)
CN (2) CN118710571A (zh)
WO (1) WO2021164731A1 (zh)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113628152A (zh) * 2021-09-15 2021-11-09 南京天巡遥感技术研究院有限公司 一种基于多尺度特征选择性融合的暗光图像增强方法
CN113706386A (zh) * 2021-09-04 2021-11-26 大连钜智信息科技有限公司 一种基于注意力机制的超分辨率重建方法
CN114898177A (zh) * 2022-05-09 2022-08-12 梅卡曼德（北京）机器人科技有限公司 缺陷图像生成方法、模型训练方法、设备、介质及产品
CN115648215A (zh) * 2022-11-04 2023-01-31 北京能创科技有限公司 基于注意力机制和孪生反卷积的服务机器人抓取检测方法
WO2023029418A1 (zh) * 2021-09-02 2023-03-09 深圳市中兴微电子技术有限公司 图像超分辨率模型训练方法、装置和计算机可读存储介质
WO2023125657A1 (zh) * 2021-12-28 2023-07-06 维沃移动通信有限公司 图像处理方法、装置和电子设备
CN116433508A (zh) * 2023-03-16 2023-07-14 湖北大学 一种基于Swin-Unet的灰度图像着色校正方法
WO2023185706A1 (zh) * 2022-03-31 2023-10-05 京东方科技集团股份有限公司 影像处理方法、影像处理装置、存储介质
CN117876242A (zh) * 2024-03-11 2024-04-12 深圳大学 眼底图像增强方法、装置、设备及可读存储介质
CN118097580A (zh) * 2024-04-24 2024-05-28 华东交通大学 一种基于Yolov4网络的危险行为防护方法及系统

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112634160A (zh) * 2020-12-25 2021-04-09 北京小米松果电子有限公司 拍照方法及装置、终端、存储介质
CN113744169A (zh) * 2021-09-07 2021-12-03 讯飞智元信息科技有限公司 图像增强方法、装置、电子设备和存储介质
CN116137015A (zh) * 2021-11-17 2023-05-19 荣耀终端有限公司 渲染方法及装置
CN114286005A (zh) * 2021-12-29 2022-04-05 合众新能源汽车有限公司 一种车辆天窗的图像显示方法及装置
CN114418857A (zh) * 2022-01-25 2022-04-29 Oppo广东移动通信有限公司 图像显示方法、装置、头戴显示设备及存储介质
CN115171882B (zh) * 2022-07-07 2023-06-02 广东工业大学 一种基于多先验嵌入的y型网络的智能医学辅助诊断方法及系统
CN117726734A (zh) * 2023-05-30 2024-03-19 荣耀终端有限公司 图像处理方法和电子设备
CN118470809B (zh) * 2023-10-30 2024-09-27 杭州名光微电子科技有限公司 融合人脸和活体掌静脉的对象识别系统及其方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160080626A1 (en) * 2014-09-16 2016-03-17 Google Technology Holdings LLC Computational Camera Using Fusion of Image Sensors
CN109889800A (zh) * 2019-02-28 2019-06-14 深圳市商汤科技有限公司 图像增强方法和装置、电子设备、存储介质
CN109919869A (zh) * 2019-02-28 2019-06-21 腾讯科技（深圳）有限公司 一种图像增强方法、装置及存储介质
CN110163807A (zh) * 2019-03-20 2019-08-23 哈尔滨工业大学 一种基于期望亮通道的低照度图像增强方法
CN110264424A (zh) * 2019-06-20 2019-09-20 北京理工大学 一种基于生成对抗网络的模糊视网膜眼底图像增强方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109345485B (zh) * 2018-10-22 2021-04-16 北京达佳互联信息技术有限公司 一种图像增强方法、装置、电子设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160080626A1 (en) * 2014-09-16 2016-03-17 Google Technology Holdings LLC Computational Camera Using Fusion of Image Sensors
CN109889800A (zh) * 2019-02-28 2019-06-14 深圳市商汤科技有限公司 图像增强方法和装置、电子设备、存储介质
CN109919869A (zh) * 2019-02-28 2019-06-21 腾讯科技（深圳）有限公司 一种图像增强方法、装置及存储介质
CN110163807A (zh) * 2019-03-20 2019-08-23 哈尔滨工业大学 一种基于期望亮通道的低照度图像增强方法
CN110264424A (zh) * 2019-06-20 2019-09-20 北京理工大学 一种基于生成对抗网络的模糊视网膜眼底图像增强方法

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023029418A1 (zh) * 2021-09-02 2023-03-09 深圳市中兴微电子技术有限公司 图像超分辨率模型训练方法、装置和计算机可读存储介质
CN113706386A (zh) * 2021-09-04 2021-11-26 大连钜智信息科技有限公司 一种基于注意力机制的超分辨率重建方法
CN113628152A (zh) * 2021-09-15 2021-11-09 南京天巡遥感技术研究院有限公司 一种基于多尺度特征选择性融合的暗光图像增强方法
CN113628152B (zh) * 2021-09-15 2023-11-17 南京天巡遥感技术研究院有限公司 一种基于多尺度特征选择性融合的暗光图像增强方法
WO2023125657A1 (zh) * 2021-12-28 2023-07-06 维沃移动通信有限公司 图像处理方法、装置和电子设备
WO2023185706A1 (zh) * 2022-03-31 2023-10-05 京东方科技集团股份有限公司 影像处理方法、影像处理装置、存储介质
CN114898177A (zh) * 2022-05-09 2022-08-12 梅卡曼德（北京）机器人科技有限公司 缺陷图像生成方法、模型训练方法、设备、介质及产品
CN115648215A (zh) * 2022-11-04 2023-01-31 北京能创科技有限公司 基于注意力机制和孪生反卷积的服务机器人抓取检测方法
CN115648215B (zh) * 2022-11-04 2024-01-26 北京能创科技有限公司 基于注意力机制和孪生反卷积的服务机器人抓取检测方法
CN116433508B (zh) * 2023-03-16 2023-10-27 湖北大学 一种基于Swin-Unet的灰度图像着色校正方法
CN116433508A (zh) * 2023-03-16 2023-07-14 湖北大学 一种基于Swin-Unet的灰度图像着色校正方法
CN117876242A (zh) * 2024-03-11 2024-04-12 深圳大学 眼底图像增强方法、装置、设备及可读存储介质
CN117876242B (zh) * 2024-03-11 2024-05-28 深圳大学 眼底图像增强方法、装置、设备及可读存储介质
CN118097580A (zh) * 2024-04-24 2024-05-28 华东交通大学 一种基于Yolov4网络的危险行为防护方法及系统

Also Published As

Publication number Publication date
CN118710571A (zh) 2024-09-27
CN113284054A (zh) 2021-08-20
EP4105877A4 (en) 2023-08-09
EP4105877A1 (en) 2022-12-21
CN113284054B (zh) 2024-07-02

Similar Documents

Publication Publication Date Title
WO2021164731A1 (zh) 图像增强方法以及图像增强装置
US20220188999A1 (en) Image enhancement method and apparatus
CN110532871B (zh) 图像处理的方法和装置
US20230214976A1 (en) Image fusion method and apparatus and training method and apparatus for image fusion model
WO2021164234A1 (zh) 图像处理方法以及图像处理装置
WO2020192483A1 (zh) 图像显示方法和设备
WO2021135657A1 (zh) 图像处理方法、装置和图像处理系统
CN109993707B (zh) 图像去噪方法和装置
WO2022134971A1 (zh) 一种降噪模型的训练方法及相关装置
CN113011562B (zh) 一种模型训练方法及装置
CN111667399A (zh) 风格迁移模型的训练方法、视频风格迁移的方法以及装置
WO2021063341A1 (zh) 图像增强方法以及装置
WO2021042774A1 (zh) 图像恢复方法、图像恢复网络训练方法、装置和存储介质
CN112070664A (zh) 一种图像处理方法以及装置
CN112348747A (zh) 图像增强方法、装置及存储介质
CN111951195A (zh) 图像增强方法及装置
WO2024002211A1 (zh) 一种图像处理方法及相关装置
CN113284055B (zh) 一种图像处理的方法以及装置
CN115131256A (zh) 图像处理模型、图像处理模型的训练方法及装置
CN114627034A (zh) 一种图像增强方法、图像增强模型的训练方法及相关设备
CN113066018A (zh) 一种图像增强方法及相关装置
WO2021189321A1 (zh) 一种图像处理方法和装置
CN113256556A (zh) 一种图像选择方法及装置
CN112581379B (zh) 图像增强方法以及装置
WO2023028866A1 (zh) 图像处理方法、装置和车辆

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21757305

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021757305

Country of ref document: EP

Effective date: 20220919