WO2021164731A1

WO2021164731A1 - Image enhancement method and image enhancement apparatus

Info

Publication number: WO2021164731A1
Application number: PCT/CN2021/076859
Authority: WO
Inventors: 马翼鹏; 汪涛; 彭竞阳; 宋风龙
Original assignee: 华为技术有限公司
Priority date: 2020-02-19
Filing date: 2021-02-19
Publication date: 2021-08-26
Also published as: EP4105877A4; EP4105877A1; CN113284054A

Abstract

An image enhancement method and an image enhancement apparatus. The image enhancement method comprises: acquiring a first high dynamic range (HDR) image corresponding to an image to be processed and a color image feature of the image to be processed, wherein the color image feature is used for indicating different brightness areas or different color change areas in the image to be processed, the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution (610); inputting the first HDR image into a neural network model, so as to perform super-resolution processing (620); and performing, by means of the neural network model, image enhancement processing on the first HDR image after same is subjected to the super-resolution processing, and the color image feature, so as to obtain a second HDR image corresponding to the image to be processed, wherein the second HDR image refers to an HDR image, the resolution size of which is the first resolution (630). The method can improve the effect of image enhancement processing.

Description

Image enhancement method and image enhancement device

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 202010102590.4, and the application name is "Image Enhancement Method and Image Enhancement Device" on February 19, 2020, the entire content of which is incorporated into this application by reference .

Technical field

This application relates to the field of artificial intelligence, and more specifically, to an image enhancement method and an image enhancement device in the computer vision field.

Background technique

Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. To put it vividly, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data. In general, computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

Image enhancement can also be referred to as image quality enhancement. It is an important branch in the field of image processing. Image enhancement technology can improve image quality without re-collecting data to meet more practical application requirements. With the development of deep learning methods, especially image processing based on convolutional neural networks (Convolutional Neural Networks, CNN) is a key driving force for the development of artificial intelligence in recent years, in various tasks of computer vision, such as image restoration Or image enhancement has achieved eye-catching results.

At present, in order to meet the real-time needs of image enhancement for larger resolution images (for example, 4K resolution), a method of "downsampling + high dynamic range (HDR) + super-division processing" is proposed to save computational overhead. In order to achieve the purpose of real-time image quality enhancement; that is, the original resolution input image is down-sampled, and the image enhancement is performed at low resolution to obtain a low-resolution enhanced image; according to the original resolution input image and the low-resolution enhanced image, Train the CNN to restore the original resolution and get the original resolution enhanced image. However, because the deep learning-based super-division method usually only performs a complex super-division process on the brightness channel of the original resolution input image, and the color channel is usually simple up-sampling, which leads to the existence of the image enhancement method before and after the super-resolution processing. Inconsistent issues in color, brightness, contrast, saturation, etc. Therefore, in the case of meeting the real-time requirements of image enhancement, how to improve the image quality after enhanced image processing becomes an urgent problem to be solved.

Summary of the invention

The present application provides an image enhancement method and an image enhancement device, which can enhance the performance of the enhanced image in terms of color, brightness, contrast, saturation, etc., while meeting the real-time requirements of image enhancement processing, so as to improve The effect of image enhancement processing.

In a first aspect, an image enhancement method is provided, including: acquiring a first high dynamic range HDR image corresponding to an image to be processed and a color image feature of the image to be processed, wherein the color image feature is used to indicate the Different brightness areas or different color change areas in the image to be processed, the image to be processed is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the Second resolution; input the first HDR image into a neural network model for super-resolution processing; image the first HDR image and the color image characteristics after the super-resolution processing through the neural network model The enhancement processing is performed to obtain a second HDR image corresponding to the image to be processed, where the second HDR image refers to an HDR image with a resolution of the first resolution.

It should be noted that the second HDR image refers to an image with the same resolution size as the image to be processed; where the same resolution size can mean that the image has the same number of pixels, and the image to be processed can be enhanced by image enhancement. The pixel values of at least part of the pixels in the processed image are adjusted, so that the visual effect of the second HDR image is better than that of the image to be processed.

Among them, the above-mentioned image to be processed may refer to the image to be processed with image enhancement requirements; for example, the image to be processed may refer to the original image with higher resolution and poor image quality; for example, it may refer to the original image subject to weather, distance, shooting Due to factors such as environment, the acquired image to be processed has the problem of low image quality; low image quality includes but is not limited to: image blur, or image color, brightness, saturation, contrast, and dynamic range are poor.

In a possible implementation manner, the first HDR image may be an image obtained by down-sampling and HDR enhancement of the image to be processed; among them, the high dynamic range (HDR) image is compared with the standard dynamic range (standard dynamic range) image. Dynamic range (SDR) images can provide more dynamic range and image colors than HDR images, that is, HDR images can include more image details.

It should be noted that the above-mentioned color image features can also be referred to as color guide maps. The color image features can provide higher guidance for difficult areas (for example, bright areas or dark areas) in the image to be processed in the input neural network model. Information, so that the neural network model can pay more attention to the enhancement effect of difficult areas in the learning process.

In the embodiment of the present application, by acquiring the image to be processed and the first HDR image of the image to be processed, the first HDR image is the low-resolution enhanced image corresponding to the image to be processed, and the image to be processed is passed in the neural network model. The color image characteristics of the first HDR image are super-resolution processed, so as to obtain an enhanced image with the same resolution size as the image to be processed, that is, the second HDR image; it can be introduced when super-resolution processing is performed on the first HDR image The color attention mechanism, that is, the color image characteristics, enables the neural network model to improve the color, brightness, contrast, and saturation of the difficult areas (too bright and dark areas) in the image to be processed during the image enhancement process. The recovery effect of the aspect, thereby improving the effect of image enhancement.

With reference to the first aspect, some implementations of the first aspect further include: acquiring a texture image feature of the image to be processed, wherein the texture image feature is used to indicate an edge area or a texture area of the image to be processed ；

The inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image according to the texture image feature through the neural network model.

It should be noted that the introduction of the texture attention mechanism in the neural network model is to enable the neural network model to learn details such as edges and textures in the image; the neural network model can be improved by using texture image features, also known as texture guide maps. For the learning of regions with higher weights in the texture guide map; thus, the process of super-division processing of the first HDR image can improve the ability of the super-division algorithm to recover image texture details, and avoid image blur or other sensory differences introduced after super-division processing.

In the embodiment of the present application, in the process of performing super-resolution processing on the first HDR image, the super-resolution reduction processing can be performed based on the dual attention mechanism, where the dual attention mechanism can include a texture attention mechanism and a color attention mechanism. ; That is, by using color image features and texture image features in the super-division processing of the first HDR image, the resulting enhanced image is the second HDR image of the image to be processed in terms of color, brightness, saturation, contrast, and texture The details and other aspects are the same as or close to the true value image, which improves the effect of image enhancement.

With reference to the first aspect, some implementations of the first aspect further include: acquiring a multi-scale image feature of the image to be processed, wherein the multi-scale image feature is used to indicate the to-be-processed image at different scales. Processing image information of the image, any one of the multi-scale image features has a different scale size;

The inputting the first HDR image into a neural network model for super-resolution processing includes:

Performing super-resolution processing on the first HDR image according to the multi-scale image feature through the neural network model.

Among them, multi-scale image features can refer to image features of different resolutions. By introducing multi-scale image features, the neural network model can introduce more image detail information, which is beneficial to ensure the restoration of the first HDR image in terms of details. .

It should be noted that multi-scale image features can be used to indicate the image information of the image to be processed in different scales, where different scales can refer to different resolutions, and image information can refer to high-frequency information in the image. ; For example, the high-frequency information may include one or more of edge information, detail information, and texture information in the image.

In the embodiments of the present application, the multi-scale image features of the image to be processed can be obtained, that is, the multi-scale feature information of the image to be processed can be extracted from the original resolution of the image to be processed by feature reduction; It can perform multi-scale super-division processing restoration based on the multi-scale features of the first HDR image, ie the low-resolution HDR image and the original input image to be processed, and the low-resolution HDR image and the extracted feature information of the original input image at the corresponding scale The fusion may be performed to obtain the second HDR image of the image to be processed, that is, the enhanced image after the image to be processed is enhanced.

Acquire the multi-scale image feature of the image to be processed, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;

The inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image according to the texture image feature and the multi-scale feature through the neural network model. Resolution processing.

In the embodiment of the present application, both the dual attention mechanism and the multi-scale image feature can be introduced during the super-resolution processing of the first HDR image; among them, the dual attention mechanism can include the texture attention mechanism and the color attention mechanism. Force mechanism, namely color image features and texture features, through dual attention mechanism and multi-scale image features, the resulting enhanced image is the second HDR image of the image to be processed in terms of color, brightness, saturation, contrast, and texture details. It is the same as or close to the true value image, thereby improving the effect of image enhancement, that is, improving the image quality after image enhancement processing.

With reference to the first aspect, in some implementation manners of the first aspect, the performing super-resolution processing on the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model includes :

Acquiring a texture image feature of a first scale, wherein the texture image feature of the first scale has the same scale as the first scale image feature in the multi-scale image;

Performing a dot multiplication operation on the first HDR image and the texture image feature of the first scale by using the neural network model to obtain a third image feature;

Perform a combined channel operation, a convolution operation, and an up-sampling operation on the third image feature and the first-scale image feature through the neural network model.

With reference to the first aspect, in some implementation manners of the first aspect, the image enhancement processing is performed on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain The second HDR image corresponding to the image to be processed includes:

Performing a dot multiplication operation and a convolution operation on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain the second HDR image.

In a second aspect, an image enhancement method is provided, which is applied to a smart screen (or an artificial intelligence screen), including: detecting a first operation that a user instructs to open a display mode interface of the smart screen; responding to the first operation , Displaying the display mode selection interface on the smart screen; detecting a second operation indicating the first display mode by the user; in response to the second operation, displaying an output image on the smart screen, the The output image is the second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed,

Wherein, the neural network model is applied in the image enhancement processing process, and the acquired first high dynamic range HDR image of the image to be processed is input into the neural network model for super-division processing; the second HDR image passes through all The neural network model is obtained by performing image enhancement processing on the color image features of the super-resolution processed first HDR image and the image to be processed; the image to be processed is an image of the first resolution, so The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution; The color image feature is used to indicate areas of different brightness or areas of different color changes in the image to be processed.

Optionally, the above-mentioned display mode interface may include, but is not limited to, the following display modes: HDR mode, Dolby mode, Ultra HD mode, Blu-ray mode, and Smart mode.

It should be understood that the aforementioned output image is a second HDR image obtained after image enhancement processing is performed on the image to be processed, that is, an HDR image with the same resolution and size as the image to be processed; wherein, the image to be processed may mean that the electronic device passes through The image/video captured by the camera, or the above-mentioned image to be processed may also be an image obtained from the inside of the electronic device (for example, an image stored in an album of the electronic device, or a picture obtained by the electronic device from the cloud). For example, the above-mentioned image to be processed may be an original film source stored in a database, or may also refer to a film source that is played in real time.

In a possible implementation, the image enhancement method of the image to be processed may be an offline method executed in the cloud. For example, the cloud server may perform image enhancement of the image to be processed to obtain an output image, and display the output image on a smart screen.

In another possible implementation manner, the above-mentioned image enhancement method may be executed by a local device, that is, it may refer to an output image obtained by performing image enhancement of an image to be processed on a smart screen. In a possible implementation manner, the first operation of the user instructing to open the display mode interface of the smart screen may include but is not limited to: instructing the smart screen to open the display mode interface through the control device, or may include the user instructing the smart screen through voice The behavior of the screen opening the display mode interface may also include other behaviors of the user instructing the smart screen to open the display mode interface; the above is an example and does not limit the application in any way.

Similarly, the above-mentioned second operation of the user instructing the first display mode may include but is not limited to: instructing the first display model in the display mode interface through the control device, or may include the user instructing the first display model in the display mode interface through voice Alternatively, it may also include other actions of the user instructing to select the first display mode in the display mode interface; the foregoing is an example and does not limit the application in any way.

Among them, the above-mentioned first display mode may refer to the HDR mode or other professional modes. The image enhancement method provided in the embodiments of the present application can be implemented through the first display mode, thereby improving the image quality of the output image, so that users who use smart screens can get better Good visual experience.

It should be noted that the specific process of performing feature enhancement processing on the image to be processed may be obtained according to the first aspect and any one of the implementation manners of the first aspect.

With reference to the second aspect, in some implementations of the second aspect, the method further includes: acquiring a texture image feature of the image to be processed, wherein the texture image feature is used to indicate an edge area or a texture area of the image to be processed ；

With reference to the second aspect, in some implementations of the second aspect, the method further includes: acquiring a multi-scale image feature of the image to be processed, wherein the multi-scale image feature is used to indicate the to-be-processed image at different scales. Processing image information of the image, any one of the multi-scale image features has a different scale size;

The inputting the first HDR image into a neural network model for super-resolution processing:

The inputting the first HDR image into the neural network model for super-resolution processing includes:

Perform super-resolution processing on the first HDR image according to the texture image feature and the multi-scale feature through the neural network model.

It should be understood that multi-scale image features can be used to indicate image information of an image to be processed at different scales, where different scales can refer to different resolutions, and image information can refer to high-frequency information in the image; for example, The high-frequency information may include one or more of edge information, detail information, and texture information in the image.

With reference to the second aspect, in some implementation manners of the second aspect, the super-resolution processing of the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model includes :

With reference to the second aspect, in some implementation manners of the second aspect, the image enhancement processing is performed on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain The second HDR image corresponding to the image to be processed includes:

In a third aspect, an image enhancement method is provided, which is applied to an electronic device with a display screen and a camera, including: detecting a second operation of the user instructing the camera; in response to the second operation, displaying on the display screen Output an image, or save an output image in the electronic device, the output image is the second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed, wherein the neural network model is applied to the image enhancement processing In the process, the acquired first high dynamic range HDR image of the image to be processed is input into the neural network model for super-resolution processing; the second HDR image is processed by the neural network model after the super-resolution processing The first HDR image and the color image characteristics of the image to be processed are obtained by image enhancement processing; the image to be processed is an image with a first resolution, and the first HDR image is an image with a second resolution , The first resolution is greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution; the color image feature is used to indicate the image to be processed Different brightness areas or different color change areas in.

Optionally, it may further include: detecting a first operation for turning on the camera by the user; in response to the first operation, displaying a shooting interface on the display screen, the shooting interface including a viewfinder, so The frame includes the image to be processed.

Wherein, the foregoing specific process of performing feature enhancement processing on the image to be processed may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.

In a possible implementation manner, the image enhancement method provided by the embodiment of this application can be applied to the camera field of smart terminals. The image enhancement method of the embodiment of this application can perform image enhancement processing on the output image in terms of color, brightness, and brightness. Contrast, saturation and other aspects of performance can be improved.

For example, when the smart terminal is taking a real-time photo, image enhancement processing is performed on the acquired original image, and the output image after the image enhancement processing is displayed on the screen of the smart terminal, or it can also be performed by performing image processing on the acquired original image. Enhancement processing, save the output image after image enhancement processing to the album of the smart terminal.

In a fourth aspect, an image enhancement method is provided, including: acquiring a road image to be processed and a first high dynamic range HDR image corresponding to the road image, wherein the road image is an image with a first resolution, and The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the road image is input into a neural network model to obtain the color image characteristics of the road screen, wherein the The color image feature is used to indicate different brightness areas or different color change areas in the road image; the first HDR image is input to the neural network model for super-resolution processing; and the super-resolution processing is performed by the neural network model. The first HDR image after resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the road image, where the second HDR image is an HDR image of the first resolution ; According to the second HDR image, identify road information in the second HDR image.

Wherein, the above-mentioned specific process of performing feature enhancement processing on the road image can be obtained according to the above-mentioned first aspect and any one of the implementation manners of the first aspect.

In a possible implementation manner, the image enhancement method provided in the embodiment of the present application may be applied to the field of automatic driving. For example, it can be applied to the navigation system of an autonomous vehicle. The image enhancement method in this application can enable the autonomous vehicle to perform image enhancement processing through the acquired original road images with lower image quality during the navigation process of the autonomous vehicle while driving on the road. Obtain the enhanced road image, so as to realize the safety of self-driving vehicles.

In a fifth aspect, an image enhancement method is provided, including: acquiring a street view image to be processed and a first high dynamic range HDR image corresponding to the street view image, wherein the street view image is an image with a first resolution, so The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the street view image is input into a neural network model to obtain the color image characteristics of the street view image, wherein the The color image features are used to indicate areas of different brightness or areas of different color changes in the street view image; input the first HDR image into the neural network model for super-resolution processing; use the neural network model to perform super-resolution processing; The first HDR image after resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the street view image, where the second HDR image is an HDR image of the first resolution ; According to the second HDR image, identify street view information in the second HDR image.

Wherein, the foregoing specific process of performing feature enhancement processing on the street view image may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.

In a possible implementation manner, the image enhancement method provided in the embodiment of the present application can be applied to the security field. For example, the image enhancement method of the embodiment of the present application can be applied to the surveillance image enhancement of a safe city. For example, the image (or video) collected by the surveillance equipment in public places is often affected by factors such as weather and distance, and the image is blurred. Problems such as low image quality. The image enhancement method of the present application can perform image enhancement on the collected original images, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information can be provided for case detection.

It should be understood that the expansion, limitation, explanation and description of the related content in the above-mentioned first aspect are also applicable to the same content in the second aspect, the third aspect, the fourth aspect, and the fifth aspect.

In a sixth aspect, an image enhancement device is provided, which includes a module/unit for executing the image enhancement method in any one of the foregoing first to fifth aspects and the first to fifth aspects.

In a seventh aspect, an image enhancement device is provided, including: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute: The first high dynamic range HDR image corresponding to the processed image and the color image feature of the image to be processed, wherein the color image feature is used to indicate areas of different brightness or areas of different color changes in the image to be processed, and The image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution; the first HDR image is input to the neural network model Performing super-resolution processing; performing image enhancement processing on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain a second HDR image corresponding to the image to be processed, Wherein, the second HDR image refers to an HDR image with a resolution of the first resolution.

In a possible implementation manner, the processor included in the foregoing image enhancement device is also used in the image enhancement method in any one of the foregoing first aspect to the fifth aspect and the first aspect to the fifth aspect.

In an eighth aspect, a computer-readable medium is provided, and the computer-readable medium stores program code for device execution, and the program code includes the program code for executing the first aspect to the fifth aspect and the first aspect to the fifth aspect. The image enhancement method in any one of the implementations.

In a ninth aspect, a computer program product containing instructions is provided. When the computer program product runs on a computer, the computer executes any one of the first aspect to the fifth aspect and the first aspect to the fifth aspect. The image enhancement method in the implementation mode.

In a tenth aspect, a chip is provided. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface, and executes the first to fifth aspects and the first aspect. To the image enhancement method in any one of the implementation manners of the fifth aspect.

Optionally, as an implementation manner, the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory. When the instructions are executed, the The processor is configured to execute the image enhancement method in any one of the foregoing first aspect to the fifth aspect and the first aspect to the fifth aspect.

Description of the drawings

FIG. 1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application;

Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of yet another application scenario provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of another application scenario provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a system architecture provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a convolutional neural network structure provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of image enhancement processing provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of a system architecture of an image enhancement method provided by an embodiment of the present application;

FIG. 12 is a schematic flowchart of HDR correction through a color guide diagram provided by an embodiment of the present application; FIG.

FIG. 13 is a schematic diagram of the system architecture of another image enhancement method provided by an embodiment of the present application;

FIG. 14 is a schematic diagram of an extraction process of a texture guide map provided by an embodiment of the present application;

15 is a schematic diagram of the system architecture of still another image enhancement method provided by an embodiment of the present application;

FIG. 16 is a schematic flowchart of a self-guided multi-level super-division processing provided by an embodiment of the present application;

FIG. 17 is a schematic diagram of HDR correction through a color guide map after introducing multi-scale image features according to an embodiment of the present application;

FIG. 18 is a schematic diagram of an evaluation result of image enhancement quality provided by an embodiment of the present application;

FIG. 19 is a schematic diagram of an evaluation result of image enhancement quality provided by an embodiment of the present application;

FIG. 20 is a schematic diagram of an evaluation result of image enhancement quality provided by an embodiment of the present application;

FIG. 21 is a schematic block diagram of an image enhancement device provided by an embodiment of the present application;

FIG. 22 is a schematic diagram of the hardware structure of an image enhancement device provided by an embodiment of the present application.

Detailed ways

The following describes the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

It should be understood that the images in the embodiments of the present application may be static images (or referred to as static pictures) or dynamic images (or referred to as dynamic pictures). For example, the images in the present application may be videos or dynamic pictures, or the present application The images in can also be static pictures or photos. For ease of description, in the following embodiments of the present application, static images or dynamic images are collectively referred to as images.

It should also be understood that in the embodiments of the present application, "first", "second", "third", etc. are only used to refer to different objects, and do not mean that there are other limitations on the referred objects.

Figure 1 shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is suitable for general artificial intelligence field requirements.

The following is a detailed explanation of the above-mentioned artificial intelligence theme framework 100 from two dimensions of "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).

"Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensing process of "data-information-knowledge-wisdom".

The "IT value chain" from the underlying infrastructure of human intelligence, information (providing and processing technology realization) to the industrial ecological process of the system, reflects the value that artificial intelligence brings to the information technology industry.

(1) Infrastructure 110

The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the basic platform.

The infrastructure can communicate with the outside through sensors, and the computing power of the infrastructure can be provided by smart chips.

The smart chip here can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), and an application specific integrated circuit (application specific). Hardware acceleration chips such as integrated circuit (ASIC) and field programmable gate array (FPGA).

The basic platform of infrastructure can include distributed computing framework and network related platform guarantee and support, and can include cloud storage and computing, interconnection network, etc.

For example, for infrastructure, data can be obtained through sensors and external communication, and then these data can be provided to the smart chip in the distributed computing system provided by the basic platform for calculation.

(2) Data 120

The data in the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.

(3) Data processing 130

The above-mentioned data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.

Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, training, etc.

Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, using formal information to carry out machine thinking and solving problems according to the reasoning control strategy. The typical function is search and matching.

Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, ranking, and prediction.

(4) Universal ability 140

After the above-mentioned data processing is performed on the data, some general capabilities can be formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image Recognition and so on.

(5) Smart products and industry applications 150

Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is an encapsulation of the overall solution of artificial intelligence, productizing intelligent information decision-making and realizing landing applications. Its application fields mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical, smart security, autonomous driving, safe city, smart terminal, etc.

Fig. 2 is a schematic diagram of an application scenario of an image enhancement method provided by an embodiment of the present application.

As shown in Figure 2, the technical solutions of the embodiments of the present application can be applied to smart terminals. The image enhancement method in the embodiments of the present application can perform image enhancement processing on the input image (or input video) to obtain the input image (or input video). ) The output image (or output video) after image enhancement.

The above-mentioned smart terminal may be mobile or fixed. For example, the smart terminal may be a mobile phone with image enhancement function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer, LC), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, augmented reality (AR)/virtual reality (VR), wearable Wearable device (WD) or an autonomous vehicle, etc., which are not limited in the embodiment of the present application.

It should be noted that, in the embodiments of the present application, image enhancement can also be referred to as image quality enhancement. Specifically, it can refer to processing the brightness, color, contrast, saturation, and/or dynamic range of the image to make the image The indicators meet the preset conditions or preset indicators. In the embodiments of the present application, image enhancement and image quality enhancement have the same meaning.

The specific application scenarios of the embodiments of the present application are described below with examples.

Application scenario 1: Smart screen (AI screen)

In one embodiment, the image enhancement method of the embodiment of the present application can also be applied to the field of smart screens; among them, smart screens are separated from traditional TVs and break through the limitations of large-screen categories. With the birth of new products such as smart screens, in the future, mobile phones and smart screens will become the dual centers of users' smart lives. Mobile phones will still be the user's personal center, while smart screens may become the family's emotional center. Smart screens will take on more roles in the family, not only the family's audio-visual entertainment center, but also the information sharing center, control management center and multi-device interactive center. For example, when playing a video on a smart screen, etc., in order to display better image quality (picture quality), the original source of the video can be processed by using the image enhancement method of the embodiment of the present application to enhance the picture of the source. Quality and get a better visual perception.

Exemplarily, when using a smart screen to play an old movie (the source time of the old movie is relatively early and the image quality of the source is poor), the image enhancement method of the embodiment of the application can be used to image the source of the old movie. Enhanced processing can show the visual sense of modern movies. For example, the source of an old movie can be enhanced to a high-dynamic range (HDR) 10 or a high-quality video of the Dolby Vision (Dolby Vision) standard through the image enhancement method of the embodiment of the present application.

Exemplarily, the present application provides an image enhancement method, including: acquiring an image to be processed (for example, the original film source of a movie, or an online film source) and a first high dynamic range HDR image corresponding to the image to be processed, The image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution; and the image to be processed is input The neural network model obtains the color image features of the image to be processed, where the color image features are used to indicate areas of different brightness or areas of different color changes in the image to be processed; the first HDR image is input to the The neural network model performs super-resolution processing; the super-resolution processed first HDR image and the color image feature are processed through the neural network model to obtain the second HDR corresponding to the image to be processed An image (for example, a film source with improved image quality), wherein the second HDR image is an HDR image of the first resolution.

Exemplarily, the present application provides an image enhancement method applied to a smart screen (or artificial intelligence screen), including: detecting a first operation instructed by a user to open the display mode interface of the smart screen; responding to the The first operation, the display mode selection interface is displayed on the smart screen; the second operation indicating the first display mode by the user is detected; in response to the second operation, the output image is displayed on the smart screen The output image is a second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed, wherein the neural network model is applied in the image enhancement processing process, and the acquired first image of the image to be processed is A high dynamic range HDR image is input to the neural network model for super-division processing; the second HDR image is processed by the neural network model on the super-resolution processed first HDR image and the to-be-processed image The color image feature of the image is obtained by image enhancement processing; the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution The second HDR image refers to an HDR image with a resolution of the first resolution; the color image feature is used to indicate different brightness areas or different color change areas in the image to be processed.

In another possible implementation manner, the above-mentioned image enhancement method may be executed by a local device, that is, it may refer to an output image obtained by performing image enhancement of an image to be processed on a smart screen.

It should be understood that the first operation of the user instructing to open the display mode interface of the smart screen may include, but is not limited to: instructing the smart screen to open the display mode interface through the control device, and may also include the user instructing the smart screen to open the display mode interface through voice. The behavior, or, may also include other behaviors of the user instructing the smart screen to open the display mode interface; the above is an example and does not limit the application in any way.

Among them, the above-mentioned first display mode may refer to the HDR mode or other professional modes. The image enhancement method provided in the embodiments of the present application can be implemented through the first display mode, thereby improving the image quality of the output image, so that users who use smart screens can get better Good visual experience. It should be noted that the image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the related embodiments in FIG. 6 to FIG. 17, which will not be repeated here.

Application scenario 2: Smart terminal camera field

In one embodiment, as shown in FIG. 3, the image enhancement method of the embodiment of the present application can be applied to the shooting of a smart terminal device (for example, a mobile phone). The image enhancement method in the embodiments of the present application can perform image enhancement processing on the acquired original image (or video) of poor quality to obtain an output image (or output video) with improved image quality.

It should be noted that in FIG. 3, in order to be distinguished from the gray-scale image portion, the color image portion is represented by hatching.

Exemplarily, the image enhancement method in the embodiments of the present application may be used to perform image enhancement processing on the acquired original image when the smart terminal is taking pictures in real time, and the output image after the image enhancement processing is displayed on the screen of the smart terminal.

Exemplarily, the image enhancement method of the embodiment of the present application may be used to perform image enhancement processing on the acquired original image, and the output image after the image enhancement processing can be saved in the album of the smart terminal.

Exemplarily, the present application proposes an image enhancement method, which is applied to an electronic device with a display screen and a camera, including: detecting a second operation of the user instructing the camera; in response to the second operation, in the display screen The output image is displayed internally, or the output image is saved in the electronic device. The output image is a second high dynamic range HDR image obtained after image enhancement processing is performed on the image to be processed, wherein the neural network model is applied to all In the process of image enhancement processing, the acquired first high dynamic range HDR image of the image to be processed is input to the neural network model for super-division processing; the second HDR image is processed by the neural network model on the super The first HDR image after resolution processing and the color image characteristics of the image to be processed are obtained by image enhancement processing; the image to be processed is an image with a first resolution, and the first HDR image is a second Resolution image, the first resolution is greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution; the color image feature is used to indicate the The different brightness areas or different color change areas in the image to be processed.

It should be noted that the image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the related embodiments in FIG. 6 to FIG. 17, which will not be repeated here.

Application scenario 3: Autonomous driving field

In an embodiment, as shown in FIG. 4, the image enhancement method of the embodiment of the present application can be applied to the field of automatic driving. For example, it can be applied to the navigation system of an autonomous vehicle. The image enhancement method in this application can enable the autonomous vehicle to obtain low-quality original road images (or original road video ) Perform image enhancement processing to obtain an enhanced road image (or road video), so as to realize the safety of autonomous vehicles.

Exemplarily, the present application provides an image enhancement method, including: acquiring a road image to be processed and a first high dynamic range HDR image corresponding to the road image, where the road image is an image with a first resolution , The first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution;

The road image is input into the neural network model to obtain the color image feature of the road screen, wherein the color image feature is used to indicate different brightness areas or different color change areas in the road image; and the first HDR The image is input to the neural network model for super-resolution processing; the super-resolution processed first HDR image and the color image feature are processed through the neural network model to obtain the corresponding road image A second HDR image, where the second HDR image is an HDR image of the first resolution; and the road information in the second HDR image is identified according to the second HDR image.

Application Scenario 4: Safe City Field

In one embodiment, as shown in FIG. 5, the image enhancement method of the embodiment of the present application can be applied to the field of safe cities, for example, the field of security. For example, the image enhancement method of the embodiment of the present application can be applied to the surveillance image enhancement of a safe city. For example, the image (or video) collected by the surveillance equipment in public places is often affected by factors such as weather and distance, and the image is blurred. Problems such as low image quality. The image enhancement method of the present application can perform image enhancement on the collected pictures, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information can be provided for case detection.

Exemplarily, the present application provides an image enhancement method, including: acquiring a street view image to be processed and a first high dynamic range HDR image corresponding to the street view image, wherein the street view image is an image with a first resolution The first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the street view image is input into a neural network model to obtain the color image characteristics of the street view image, wherein, The color image features are used to indicate areas of different brightness or areas of different color changes in the street view image; input the first HDR image to the neural network model for super-resolution processing; and perform super-resolution processing on the neural network model through the neural network model The first HDR image after the super-resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the street view image, wherein the second HDR image is of the first resolution HDR image; according to the second HDR image, identifying street view information in the second HDR image.

It should be understood that the foregoing is an example of an application scenario, and does not limit the application scenario of the present application in any way.

Since the embodiments of the present application involve a large number of applications of neural networks, in order to facilitate understanding, the following first introduces related terms and concepts of neural networks that may be involved in the embodiments of the present application.

(1) Neural network

A neural network can be composed of neural units. A neural unit can refer to _{an arithmetic unit that takes x s} and intercept 1 as inputs. The output of the arithmetic unit can be:

Among them, s=1, 2,...n, n is a natural number greater than 1, W _s is the weight of x _s , and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be a region composed of several neural units.

(2) Deep neural network

Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with multiple hidden layers. The DNN is divided according to the positions of different layers. The neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the number of layers in the middle are all hidden layers. The layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.

Although DNN looks complicated, it is not complicated as far as the work of each layer is concerned. Simply put, it is the following linear relationship expression:

in,

Is the input vector,

Is the output vector,

Is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just the input vector

After such a simple operation, the output vector is obtained

Due to the large number of DNN layers, the coefficient W and the offset vector

The number is also relatively large. The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as

The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.

In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

It should be noted that there is no W parameter in the input layer. In deep neural networks, more hidden layers make the network more capable of portraying complex situations in the real world. In theory, a model with more parameters is more complex and has a greater "capacity", which means that it can complete more complex learning tasks. Training the deep neural network is also the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vector W of many layers).

(3) Convolutional neural network

Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer. The feature extractor can be regarded as a filter. The convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network. In the convolutional layer of a convolutional neural network, a neuron can be connected to only part of the neighboring neurons. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Sharing weight can be understood as the way of extracting image information has nothing to do with location. The convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.

(4) Loss function

In the process of training a deep neural network, because it is hoped that the output of the deep neural network is as close as possible to the value that you really want to predict, you can compare the predicted value of the current network with the target value you really want, and then based on the difference between the two To update the weight vector of each layer of neural network (of course, there is usually an initialization process before the first update, that is, pre-configured parameters for each layer in the deep neural network), for example, if the predicted value of the network If it is high, adjust the weight vector to make its prediction lower, and keep adjusting until the deep neural network can predict the really wanted target value or a value very close to the really wanted target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value". This is the loss function or objective function, which is used to measure the difference between the predicted value and the target value. Important equation. Among them, take the loss function as an example. The higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of reducing this loss as much as possible.

(5) Backpropagation algorithm

The neural network can use the back propagation (BP) algorithm to modify the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal to the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged. The back-propagation algorithm is a back-propagation motion dominated by error loss, and aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

FIG. 6 shows a system architecture 200 provided by an embodiment of the present application.

In FIG. 6, a data collection device 260 is used to collect training data. For the image enhancement method of the embodiment of the present application, the image enhancement model (also referred to as an image enhancement network) can be further trained through training data, that is, the training data collected by the data collection device 260 can be training images.

Exemplarily, the training data of the training image enhancement model in the embodiment of the present application may include original images and sample enhanced images.

For example, the original image may refer to an image with lower image quality, and the sample enhanced image may refer to an image with higher image quality. For example, it may refer to one or more of the brightness, color, details, etc. The image after all aspects have been improved.

It should be understood that image enhancement can also be referred to as image quality enhancement, which can specifically refer to processing the brightness, color, contrast, saturation, and/or dynamic range of an image, so that various indicators of the image meet preset conditions Or preset indicators. In the embodiments of the present application, image enhancement and image quality enhancement have the same meaning.

After the training data is collected, the data collection device 260 stores the training data in the database 230, and the training device 220 trains based on the training data maintained in the database 230 to obtain the target model/rule 201 (that is, the image enhancement model in the embodiment of the present application) . The training device 220 inputs the training data into the image enhancement model until the difference between the predicted enhanced image output by the training image enhancement model and the sample enhanced image meets a preset condition (for example, the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold, Or, the difference between the predicted enhanced image and the sample enhanced image remains unchanged or no longer decreases), thereby completing the training of the target model/rule 201.

Exemplarily, the image enhancement model used to perform the image enhancement method in the embodiment of the present application can realize end-to-end training. For example, the image enhancement model can enhance the image (for example, the true value image) by the sample corresponding to the input image and the input image. Achieve end-to-end training.

In the embodiment provided in this application, the target model/rule 201 is obtained by training an image enhancement model. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the collection of the data collection device 260, and may also be received from other devices.

In addition, it should be noted that the training device 220 does not necessarily perform the training of the target model/rule 201 completely based on the training data maintained by the database 230. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of the embodiment. It should also be noted that at least part of the training data maintained in the database 230 may also be used to execute the process of the processing to be processed by the execution device 210.

The target model/rule 201 trained according to the training device 220 can be applied to different systems or devices, such as the execution device 210 shown in FIG. 6, the execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.

In FIG. 6, the execution device 210 is configured with an input/output (input/output, I/O) interface 212 for data interaction with external devices. The user can input data to the I/O interface 212 through the client device 240. The input data in this embodiment of the present application may include: a to-be-processed image input by the client device.

The preprocessing module 213 and the preprocessing module 214 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 212. In the embodiment of the present application, the preprocessing module 213 and the preprocessing module may not be provided. 214 (there may only be one preprocessing module), and the calculation module 211 is directly used to process the input data.

When the execution device 210 preprocesses input data, or when the calculation module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call data, codes, etc. in the data storage system 250 for corresponding processing. , The data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 250.

Finally, the I/O interface 212 returns the processing result to the image-enhanced image to be processed as described above, and returns the resulting output image to the client device 240 to provide it to the user.

It is worth noting that the training device 220 can generate corresponding target models/rules 201 based on different training data for different goals or tasks, and the corresponding target models/rules 201 can be used to achieve the above goals or complete The above tasks provide users with the desired results.

In the case shown in FIG. 6, in one case, the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 212.

In another case, the client device 240 can automatically send input data to the I/O interface 212. If the client device 240 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 240. The user can view the result output by the execution device 210 on the client device 240, and the specific presentation form may be a specific manner such as display, sound, and action. The client device 240 can also be used as a data collection terminal to collect the input data of the input I/O interface 212 and the output result of the output I/O interface 212 as new sample data, and store it in the database 230 as shown in the figure. Of course, it is also possible not to collect through the client device 240. Instead, the I/O interface 212 directly uses the input data input to the I/O interface 212 and the output result of the output I/O interface 212 as a new sample as shown in the figure. The data is stored in the database 230.

It is worth noting that FIG. 6 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation. For example, in FIG. 6, the data The storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210.

As shown in FIG. 6, the target model/rule 201 is trained according to the training device 220. The target model/rule 201 may be an image enhancement model in the embodiment of the present application. Specifically, the image enhancement model provided in the embodiment of the present application may be Deep neural network, convolutional neural network, or, it can be deep convolutional neural network, etc.

The following describes the structure of the convolutional neural network in detail with reference to Figure 7. As mentioned in the introduction to the basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture. The deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of. As a deep learning architecture, the convolutional neural network is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.

The structure of the image enhancement model in the embodiment of the present application may be as shown in FIG. 7. In FIG. 7, the convolutional neural network 300 may include an input layer 310, a convolutional layer/pooling layer 320 (wherein the pooling layer is optional), a fully connected layer 330 and an output layer 340. Wherein, the input layer 310 can obtain the image to be processed, and pass the obtained image to be processed to the convolutional layer/pooling layer 320 and the fully connected layer 330 for processing, and the processing result of the image can be obtained. The following describes the internal layer structure in CNN 300 in FIG. 7 in detail.

Convolutional layer/pooling layer 320:

As shown in Fig. 7, the convolutional layer/pooling layer 320 may include layers 321-326, for example: in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, and layer 323 is a convolutional layer. Layers, 324 is a pooling layer, 325 is a convolutional layer, and 326 is a pooling layer; in another implementation, 321 and 322 are convolutional layers, 323 is a pooling layer, and 324 and 325 are convolutional layers. Layer, 326 is the pooling layer, that is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.

The following will take the convolutional layer 321 as an example to introduce the internal working principle of a convolutional layer.

The convolution layer 321 can include many convolution operators. The convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix. The convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. And so on, it depends on the value of stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same. During the convolution operation, the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row×column) are applied. That is, multiple homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.

Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract the edge information of the image, another weight matrix is used to extract the specific color of the image, and the other weight matrix is used to correct the unwanted images in the image. The noise is blurred and so on. The multiple weight matrices have the same size (row×column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are merged to form The output of the convolution operation.

The weight values in these weight matrices need to be obtained through a lot of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions. .

When the convolutional neural network 300 has multiple convolutional layers, the initial convolutional layer (such as 321) often extracts more general features, and the general features can also be called low-level features; with the convolutional neural network With the 300-depth deepening, the features extracted by the subsequent convolutional layer (for example, 326) become more and more complex, for example, features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.

Pooling layer:

Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolutional layer. In the 321-326 layers exemplified by 320 in Figure 7, it can be a convolutional layer followed by a layer. The pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers. In the image processing process, the purpose of the pooling layer is to reduce the size of the image space. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain an image with a smaller size. The average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of the average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.

In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

Fully connected layer 330:

After processing by the convolutional layer/pooling layer 320, the convolutional neural network 300 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 300 needs to use the fully connected layer 330 to generate one or a group of required classes of output. Therefore, the fully connected layer 330 may include multiple hidden layers (331, 332 to 33n as shown in FIG. 7) and an output layer 340. The parameters contained in the multiple hidden layers can be based on specific task types. The relevant training data of the, for example, the task type can include image enhancement, image recognition, image classification, image detection, and image super-resolution reconstruction, etc.

After the multiple hidden layers in the fully connected layer 330, that is, the final layer of the entire convolutional neural network 300 is the output layer 340. The output layer 340 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 300 (as shown in Figure 7, the propagation from 310 to 340 directions is forward propagation) is completed, the back propagation (as shown in Figure 7 is the propagation from 340 to 310 directions as back propagation). Start to update the aforementioned weight values and deviations of each layer to reduce the loss of the convolutional neural network 300 and the error between the output result of the convolutional neural network 300 through the output layer and the ideal result.

It should be noted that the convolutional neural network shown in FIG. 7 is only used as an example of the structure of the image enhancement model of the embodiment of the present application. In specific applications, the convolutional neural network used in the image enhancement method of the embodiment of the present application It can also exist in the form of other network models.

In the embodiment of the present application, the image enhancement device may include the convolutional neural network 300 shown in FIG. 7, and the image enhancement device may perform image enhancement processing on the image to be processed to obtain a processed output image.

FIG. 8 is a hardware structure of a chip provided by an embodiment of the present application. The chip includes a neural network processor 400 (neural-network processing unit, NPU). The chip can be set in the execution device 210 as shown in FIG. 6 to complete the calculation work of the calculation module 211. The chip can also be set in the training device 220 shown in FIG. 6 to complete the training work of the training device 220 and output the target model/rule 201. The algorithms of each layer in the convolutional neural network as shown in FIG. 7 can be implemented in the chip as shown in FIG. 8.

The NPU 400 is mounted on a main central processing unit (CPU) as a coprocessor, and the main CPU allocates tasks. The core part of the NPU 400 is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to extract data from the memory (weight memory or input memory) and perform calculations.

In some implementations, the arithmetic circuit 403 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.

For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 403 fetches the data corresponding to matrix B from the weight memory 402 and caches it on each PE in the arithmetic circuit 403. The arithmetic circuit 403 fetches the matrix A data and matrix B from the input memory 401 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 408 (accumulator).

The vector calculation unit 407 can perform further processing on the output of the arithmetic circuit 403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on. For example, the vector calculation unit 407 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .

In some implementations, the vector calculation unit 407 can store the processed output vector to the unified memory 406. For example, the vector calculation unit 407 may apply a nonlinear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a combined value, or both.

In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 403, for example for use in subsequent layers in a neural network.

The unified memory 406 is used to store input data and output data. The weight data directly passes through the storage unit access controller 405 (direct memory access controller, DMAC) to store the input data in the external memory into the input memory 401 and/or unified memory 406, and the weight data in the external memory into the weight memory 402 , And store the data in the unified memory 406 into the external memory.

The bus interface unit 410 (bus interface unit, BIU) is used to implement interaction between the main CPU, the DMAC, and the fetch memory 409 through the bus.

An instruction fetch buffer 409 (instruction fetch buffer) connected to the controller 404 is used to store instructions used by the controller 404. The controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.

Generally, the unified memory 406, the input memory 401, the weight memory 402, and the instruction fetch memory 409 are all on-chip (On-Chip) memories. The external memory is a memory external to the NPU. The external memory can be a double data rate synchronous dynamic random access memory. Memory (double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.

Among them, the operations of each layer in the convolutional neural network shown in FIG. 7 may be executed by the arithmetic circuit 403 or the vector calculation unit 407.

The execution device 210 in FIG. 6 introduced above can execute each step of the image enhancement method of the embodiment of the present application. The CNN model shown in FIG. 7 and the chip shown in FIG. 8 can also be used to execute the image of the embodiment of the present application. Steps of the enhancement method.

FIG. 9 shows a system architecture 500 provided by an embodiment of the present application. The system architecture includes a local device 520, a local device 530, an execution device 510, and a data storage system 550. The local device 520 and the local device 530 are connected to the execution device 510 through a communication network.

Exemplarily, the execution device 510 may be implemented by one or more servers.

Optionally, the execution device 510 can be used in conjunction with other computing devices. For example: data storage, routers, load balancers and other equipment. The execution device 510 may be arranged on one physical site or distributed on multiple physical sites. The execution device 510 may use the data in the data storage system 550 or call the program code in the data storage system 550 to implement the image enhancement method of the embodiment of the present application.

It should be noted that the above-mentioned execution device 510 may also be referred to as a cloud device, and in this case, the execution device 510 may be deployed in the cloud.

Specifically, the execution device 510 may perform the following process: acquire a to-be-processed image and a first high dynamic range HDR image corresponding to the to-be-processed image, where the to-be-processed image is an image with a first resolution, and the first The HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; the image to be processed is input into a neural network model to obtain the color image characteristics of the image to be processed, wherein the color Image features are used to indicate areas of different brightness or areas of different color changes in the image to be processed; input the first HDR image into the neural network model for super-resolution processing; and perform super-resolution processing on the neural network model through the neural network model. The first HDR image after resolution processing and the color image feature are processed to obtain a second HDR image corresponding to the image to be processed, where the second HDR image is an HDR of the first resolution image.

In a possible implementation manner, the image enhancement method of the embodiment of the present application may be an offline method executed in the cloud. For example, the image enhancement method of the embodiment of the present application may be executed by the execution device 510 described above.

In a possible implementation manner, the image enhancement method in the embodiment of the present application may be executed by the local device 520 or the local device 530.

In the embodiments of the present application, image enhancement can be performed on the acquired image to be processed with poor image quality, so as to obtain the output image whose performance of the image to be processed is improved in terms of image details, image color, and image brightness, that is, the second image. HDR image.

For example, the user may operate respective user devices (for example, the local device 520 and the local device 530) to interact with the execution device 510. Each local device can represent any computing device, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.

Each user's local device can interact with the execution device 510 through a communication network of any communication mechanism/communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

In an implementation manner, the local device 520 and the local device 530 can obtain the relevant parameters of the aforementioned neural network model from the execution device 510, deploy the neural network model on the local device 520 and the local device 530, and use the neural network model to perform Image enhancement processing, etc.

In another implementation, the neural network model can be directly deployed on the execution device 510, and the execution device 510 obtains the image to be processed and the first HDR image corresponding to the image to be processed from the local device 520 and the local device 530, and according to the neural network model Perform image enhancement processing to obtain a second HDR image corresponding to the image to be processed.

For example, the aforementioned neural network model may be the superdivision network in the embodiment of the present application, for example, refer to the superdivision network 720 in the subsequent FIG. 11, FIG. 13 and FIG.

At present, in order to meet the real-time enhancement of high-resolution images, a "down-sampling+HDR+super-resolution" method is proposed to save computational overhead, so as to achieve the purpose of real-time image quality enhancement. The original resolution image and the low resolution image enhanced image can be input to the pre-trained convolutional neural network, so that the image is super-divided (that is, the original resolution of the image is restored), and the output original resolution enhanced image is obtained. High-resolution image enhancement is performed in the above manner. On the one hand, for the purpose of saving computational overhead, deep learning-based super-division methods usually only perform a complex super-division process on the brightness channel of the image, while the color channel of the image can be used Simple up-sampling results in the inability to guarantee the consistency of the original resolution enhanced image and the low resolution enhanced image in terms of color, brightness, contrast, saturation, and dynamic range; on the other hand, the above image The process of super-resolution processing for high-resolution images and low-resolution enhanced images has not been explained in detail; if simple super-resolution processing is used, there will be problems such as image blur and loss of texture details; if over-complex super-resolution processing is used, There will be an increase in the processing time of the entire convolutional neural network, and problems such as checkerboards and artifacts may be introduced.

In view of this, the embodiments of the present application provide an image enhancement method and an image enhancement device, by acquiring a to-be-processed image and a first HDR image of the to-be-processed image, where the first HDR image is the low resolution corresponding to the to-be-processed image To enhance the image, the first HDR image is super-resolution processed by the color image characteristics of the image to be processed in the neural network model, thereby obtaining an enhanced image with the same resolution as the image to be processed, that is, the second HDR image; in this application In the embodiment, a color attention mechanism is introduced on the basis of "downsampling + HDR + super resolution", that is, color image characteristics, so that in the process of image enhancement, the neural network model is improved for difficult areas in the image to be processed (over Bright and dark areas) restore effects in terms of color, brightness, contrast, and saturation, thereby improving the effect of image enhancement.

FIG. 10 shows a schematic flowchart of an image enhancement method provided by an embodiment of the present application. The image enhancement method shown in FIG. 10 may be executed by an image enhancement apparatus, which may be the execution device 210 in FIG. It may be the execution device 510 in FIG. 9 or a local device. The method shown in FIG. 10 includes steps 610 to 630, and steps 610 to 630 are respectively described in detail below.

Step 610: Acquire the first high dynamic range HDR image corresponding to the image to be processed and the color image characteristics of the image to be processed.

Wherein, the color image feature is used to indicate different brightness areas or different color change areas in the image to be processed, the image to be processed is an image with a first resolution, and the first HDR image is an image with a second resolution , The first resolution is greater than the second resolution.

For example, the above-mentioned image to be processed may refer to the image to be processed with image enhancement requirements; for example, the image to be processed may refer to the original image with higher resolution and poor image quality; for example, it may refer to the original image subject to weather, distance, shooting Due to the influence of environment and other factors, the acquired image to be processed has the problem of low image quality; low image quality includes but is not limited to: low image quality, including but not limited to image blur, or image color, brightness, saturation Degree, contrast, poor dynamic range, etc.

For example, the above-mentioned image to be processed may be an image captured by an electronic device through a camera, or the above-mentioned image to be processed may also be an image obtained from the inside of the electronic device (for example, an image stored in an album of an electronic device, or an electronic device from Images obtained from the cloud). For example, the electronic device may be any one of the local device or the execution device shown in FIG. 9.

Exemplarily, the first HDR image may be an image obtained by down-sampling and HDR enhancement of the image to be processed; among them, a high dynamic range (high dynamic range, HDR) image and a standard dynamic range (standard dynamic range, SDR) Compared with the image, it can provide more dynamic range and image color, that is, more image details can be included in the HDR image.

In the embodiments of the present application, the color image characteristics of the image to be processed can be acquired through target learning or traditional methods.

Exemplarily, a target learning method or a traditional method may be used to process the image to be processed to obtain the color image characteristics of the image to be processed.

Among them, the core of the target learning method is to provide such a learning target so that the difference between the input image and the true value image (learning target) can be measured. Such difference is usually reflected in the bright area and shadow (dark area) area of the image.

In addition, traditional methods can also be used to obtain the color image characteristics of the image to be processed. At this time, the true value image is no longer needed, and there is no learning process; for example, the color input image can be converted to a grayscale image or converted to a Other color domains of the luminance channel (such as YUV domain, LAB domain), and then set the lowest threshold and highest threshold, the channels (such as grayscale, Y channel in YUV domain, L channel in LAB domain) are lower and higher than the lowest threshold The part with the highest threshold is regarded as the area with larger weight, and the other areas are set as the area with lower weight.

In an example, the color image characteristics of the image to be processed can be obtained through the self-encoding and decoding sub-network of the image to be processed.

In an example, due to factors such as the operating speed, operating video memory, and power consumption of the terminal device, in order to save the amount of calculation; the image to be processed can be down-sampled first to obtain a low-resolution image, and the low-resolution image corresponding to the image to be processed can be The resolution image and the self-encoding and decoding sub-network obtain the color image characteristics; for example, refer to the following figure 12 to obtain the color image characteristics of the image to be processed through the low-resolution image of the image to be processed.

Step 620: Input the first HDR image into the neural network model for super-resolution processing.

The above-mentioned super-resolution processing may refer to a convolution operation and an up-sampling operation, so that the resolution of the first HDR image is the same as the resolution of the image to be processed.

Further, in order to make the neural network model pay more attention to the edge area and texture features when performing super-resolution processing on the first HDR image, a texture attention mechanism can be introduced when performing super-resolution processing on the first HDR image.

Optionally, in a possible implementation manner, the image to be processed can be input into the neural network model to obtain the texture image feature of the image to be processed, where the texture image feature can be used to indicate the edge area and texture of the image to be processed Region; inputting the first HDR image into a neural network model for super-resolution processing may refer to performing super-resolution processing on the first HDR image according to texture image features through the neural network model.

Further, in the process of super-division processing the first HDR image, multi-scale image features of the image to be processed can also be introduced; among them, multi-scale image features can refer to image features of different resolutions, by introducing multi-scale image features. Features can make the neural network model introduce more image detail information, which helps to ensure the restoration of the first HDR image in terms of details.

In the embodiment of the present application, in the process of performing super-resolution processing on the first HDR image, super-resolution reduction processing can be performed based on a dual attention mechanism, where the dual attention mechanism can include a texture attention mechanism and a color attention mechanism . By using the color attention mechanism and the texture attention mechanism in the super-division processing, the final enhanced image, that is, the second HDR image of the image to be processed, is more realistic in terms of color, brightness, saturation, contrast, and texture details. The value images are the same or close, which improves the effect of image enhancement.

Optionally, in a possible implementation manner, the image to be processed can be input into the neural network model to obtain the multi-scale image features of the image to be processed, where any one of the multi-scale image features has a different scale; The neural network model performs super-resolution processing on the first HDR image according to the texture image feature and the multi-scale image feature, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales. For example, refer to the schematic flowchart of the self-guided multi-level super-division processing shown in FIG. 16 below.

Exemplarily, performing super-resolution processing on the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model may include: performing scale adjustment processing on the texture image feature through the neural network model to obtain a texture image of the first scale The first-scale texture image feature has the same scale as the first-scale image feature in the multi-scale image; the first HDR image and the first-scale texture image feature are multiplied by the neural network model to obtain the first Three image features: the third image feature and the first-scale image feature are combined channel operation, convolution operation, and up-sampling operation through the neural network model.

In the embodiment of this application, a multi-level super-division processing method is proposed, that is, to extract multi-scale feature information from the original resolution input image through feature dimensionality reduction; the super-division processing can be based on the first The multi-scale features of the HDR image, ie the low-resolution HDR image and the image to be processed, that is the original input image, are restored by multi-scale super-division processing. The low-resolution HDR image and the extracted feature information of the corresponding scale of the original input image can be fused, thereby The second HDR image obtained from the image to be processed is the enhanced image after the image to be processed is enhanced.

Step 630: Process the super-resolution processed first HDR image and color image features through the neural network model to obtain a second HDR image corresponding to the image to be processed.

The second HDR image is an HDR image of the first resolution, that is, the second HDR image has the same resolution size as the image to be processed, and the second HDR image may refer to an enhanced image obtained after the image to be processed is enhanced.

Exemplarily, a point multiplication operation and a convolution operation may be performed on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain the second HDR image.

Wherein, the above-mentioned dot multiplication operation may refer to pixel-by-pixel multiplication.

FIG. 11 is a schematic diagram of a system architecture of an image enhancement method provided by an embodiment of the present application. As shown in FIG. 11, the system architecture may include an HDR network 710 and a super resolution (super resolution, SR) network 720; among them, the HDR network 710 may include a down-sampling unit 711 and an HDR enhancement unit 712; the SR network 720 may include The color attention unit 721, the super-division processing unit 722, and the HDR correction unit 723.

Exemplarily, the aforementioned down-sampling unit 711 may be used to perform down-sampling processing on an input image, such as a high-resolution image, to obtain a low-resolution image or a small-resolution image, for example, a 1080P resolution image.

Exemplarily, the above-mentioned HDR enhancement unit 712 may be used to process the down-sampling processed low-resolution image through an HDR increase method to obtain a low-resolution HDR enhanced image.

Among them, the HDR enhancement method can use any neural network model with image enhancement to perform image enhancement processing; for example, various neural networks based on Unet or High Dynamic Range Network (HDRNet) can be used. .

Exemplarily, the above-mentioned color attention unit 721 may be used to extract the color guide map of the input image by using a certain calculation method according to the input image (for example, the original resolution image); wherein, the color guide map may be the color corresponding to the input image. Image characteristics.

It should be understood that the above-mentioned color guide map has higher guidance information for difficult areas (for example, bright areas or dark areas) in the input image, so that the SR network 720 can pay more attention to the enhancement effect of difficult areas in the learning process.

Exemplarily, the HDR correction unit 723 can be used to combine the low-resolution enhanced image output by the HDR enhancement unit 712 and the image output by the super-division processing unit 722, combined with the color guide map obtained by the color attention unit 721, and use CNN for the super-resolution The divided image is corrected in terms of color, brightness, contrast, saturation, etc., so as to ensure the consistency of the HDR effect before and after the super-division processing unit 722 and strengthen the enhancement effect for the difficult area of the image.

The following describes the process of HDR correction through the color guide map in detail with reference to FIG. 12. FIG. 12 is a schematic diagram of HDR correction through a color guide map provided by an embodiment of the present application.

Among them, the color guide map shown in Figure 12, that is, the color image features corresponding to the input image can be extracted by using a pre-training network, for example, an Encoder-Decoder Network can be used as a network for extracting the color guide map The input image of the self-encoding and decoding network can refer to the low-resolution input image, and the color guide map can be obtained through the self-encoding and decoding network; among them, the self-encoding and decoding network can ensure that the input image is in the pre-training process by designing the objective function. Difficult areas (for example, bright areas or dark areas) correspond to more weight in the color guide map.

For example, the following objective function can be used to train the self-encoding and decoding network:

in,

Represents the maximum value of the channel dimension of the input image; A _map represents the color guide map; I _in represents the input image feature of the self-encoding and decoding network, for example, it can refer to the original resolution image feature or the low resolution image feature; I _target Represents a true value image with the same resolution as the input image.

The objective function shown above can make the self-encoding and decoding network have a greater difference between the input image and the target image in a certain area, and the color guide map will reflect a greater weight in this area.

In the training process of the SR network 720, the color guide map extraction network, such as the self-encoding and decoding network shown in Figure 12, can no longer participate in the training; in the darker area of the input image and the area where the color of the image changes more, the color guide map The medium correspondence can show greater weight, thereby guiding the back-end HDR correction unit 723 to focus on learning these areas.

In an example, as shown in FIG. 12, the input image may refer to the original resolution image; or, in order to save the amount of calculation, the input image may be a low-resolution image after down-sampling the original resolution image.

It should be noted that if the above-mentioned color guide map is obtained from an original resolution image, the color guide map may also not need to perform an upsampling operation.

Exemplarily, the above-mentioned HDR correction unit 723 may be a pre-trained CNN network, where the HDR correction unit 723 may receive three parts of input data, where the first input data is the first original resolution output by the super-division processing unit 722 Enhanced image, the first original resolution enhanced image may refer to an image obtained after upsampling the low-resolution enhanced image output by the HDR enhancement unit 712; the second input data is the low-resolution enhanced image output by the HDR enhancement unit 712; The third input data is the color attention guide map output by the color attention unit 721; the HDR correction unit 723 performs HDR correction according to the above three parts of input data, and outputs the second original resolution enhanced image.

Further, in order to make the super-division processing unit 722 pay more attention to learning texture and edges in the input image when performing super-division processing, a texture attention unit can be introduced into the system architecture, as shown in FIG. 13.

FIG. 13 may also include a texture attention unit 724, which may be used to extract the texture guide image (that is, the high frequency information of the image) using a certain calculation method according to the input image, that is, the original resolution image. When performing up-sampling and convolution operations, the super-division processing unit 722 can also enhance the restoration of image detail texture and edge regions in the super-division process according to the texture guide map output by the texture attention unit 724.

The process of obtaining the texture guide map will be described below in conjunction with FIG. 14. FIG. 14 is a schematic diagram of the extraction process of the texture guide map provided by the embodiment of the present application.

Exemplarily, the input data of the texture attention unit may be an input image without color information, for example, a grayscale image or Y channel of the YUV color gamut; for example, the input data is a Y channel image feature, and the image can be filtered through Gaussian filtering. Medium and high frequency information, after filtering, the output data and the input image are negatively residual so that the high frequency information of the image can be obtained, that is, the texture guide map. The texture and edges in the image usually exist in the high frequency information of the image as shown in Figure 14. Shown.

It should be noted that the acquisition process of the texture guide map is not limited to this type of method, and algorithms such as edge detection may also be used to acquire the texture guide map, which is not limited in this application.

In the embodiment of the present application, the texture attention unit 724 can be used to extract high-frequency information in the image as a texture guide map. The extracted texture guide map will act on the super-division processing unit 722, which can improve the SR network 200's ability to guide the map. The learning of areas with higher weights in the middle can enhance the learning of details such as image edges and textures by the SR network 720.

In an example, the above-mentioned super-resolution processing unit may adopt a self-guided multi-level super-division unit, that is, the super-division process may have multiple image sizes that progressively grow to the same size as the original resolution input image, as shown in Fig. 15 Shown.

FIG. 15 is a schematic diagram of a system architecture of an image enhancement method provided by an embodiment of the present application. As shown in FIG. 15, the system architecture may include an HDR network 710 and a super resolution (SR) network 720; among them, the HDR network 710 may include a down-sampling unit 711 and an HDR enhancement unit 712; the SR network 720 may It includes a color attention unit 721, a self-guided multi-level super-division unit 722, an HDR correction unit 723, a texture attention unit 724, and a multi-scale self-guided feature extraction unit 725.

Wherein, the aforementioned HDR network 710 is configured to input the image I according to the input original resolution (for example, 4K resolution), and perform HDR enhancement to obtain an HDR enhanced low-resolution HDR image (for example, 1080P resolution).

For example, the input image may be an original resolution image, such as a high-resolution image or a full-resolution image; the input image is processed by the down-sampling unit 711 to obtain a low-resolution image or a small-resolution image; The input HDR enhancement unit 712 is processed to obtain the output low-resolution HDR image.

The above-mentioned SR network 720 is used to perform super-division processing on the input image through the color attention unit 721 included in the above-mentioned SR network 720 to the multi-scale self-guided feature extraction unit 725 according to the input low-resolution HDR image and the original resolution image. Thereby, the original resolution of the image is restored, and the output original resolution enhanced image (for example, 4K resolution) is obtained.

Exemplarily, as shown in FIG. 15, the multi-scale self-guided feature extraction unit 725 in the SR network 720 can be used to perform feature extraction through a convolutional neural network according to the input original resolution image, so as to obtain multiple scales Self-guided map.

It should be noted that the above-mentioned self-guided images of multiple scales may refer to image features of different scales obtained through down-sampling and convolution processing of different depths through input graphics. The down-sampling method may include, but is not limited to, sampling interpolation methods. Or pixel rearrangement (space to depth) operations, etc., multiple scales may refer to multiple resolution sizes.

Exemplarily, the above-mentioned texture attention unit 724 may be used to extract the texture guide image (ie, high-frequency information of the image) using a certain calculation method according to the input image, that is, the original resolution image. The specific process can be seen in Figure 14 above, which will not be repeated here.

Exemplarily, as shown in FIG. 15, the input data in the self-guided multi-level super-division unit 722 may include image features of different scales input by the multi-scale self-guided feature extraction unit 725, and corresponding input images output by the texture attention unit 724. The texture image feature of the HDR enhancement unit 712 and the low-resolution HDR enhanced image input by the HDR enhancement unit 712, the self-guided multi-level super-division unit 722 can perform up-sampling and convolution operations on the above-mentioned input data to perform image resolution restoration processing, thereby obtaining a super-division restoration Image H _SR .

It should be noted that the introduction of texture attention image features in the above-mentioned super-division processing process can enhance the restoration of the image detail texture and edge area in the super-division process when performing image super-division processing; in addition, input a multi-scale image Features can make the input of more image details and information in the process of super-division processing.

Exemplarily, the above-mentioned HDR correction unit 723 may be used to correct the above-mentioned super-reduced image H _SR in terms of color, brightness, contrast, saturation, etc., so as to obtain an original resolution enhanced image, and make the original resolution enhanced image It is infinitely close to the true value image; among them, the input data of the HDR correction unit 723 can be a super- _{resolution restoration image H SR} , a low-resolution HDR image H _L and a color attention image feature, so as to perform a convolution operation on the input data to obtain The image feature of is the same as the true value image feature or the deviation is within the preset range.

For example, FIG. 16 is a schematic flowchart of the self-guided multi-level super-division processing provided by an embodiment of the present application.

As shown in Figure 16, assuming that the input original resolution image is W×H, the multi-scale self-guided image obtained through down-sampling and feature extraction processing is image features with a resolution of W/2×H/2 and a resolution of W /4×H/4 image features; the multi-scale guide image can be used as a priori information to be spliced with the output features of each level of super-division process to guide the super-division process; for example, for a resolution of W/2×H/2 Scale, first adjust the scale of the texture attention image feature to obtain a texture attention image feature with a resolution of W/4×H/4; set the texture attention image feature of W/4×H/4 to a resolution of W /4×H/4 HDR enhanced image is subjected to dot multiplication operation, which can refer to pixel-by-pixel multiplication; then, the resolution of the image feature obtained after dot multiplication and the original resolution of the input image is W/4× H/4 image features are connected, that is, channel processing is merged; in the same way, for a resolution of W/2×H/2 scale, first adjust the scale of the texture attention image feature to obtain a resolution of W/2× H/2 texture attention image feature; the texture attention image feature of W/2×H/2 and the HDR enhanced image with a resolution of W/2×H/2 are dot-multiplied, that is, it can be pixel by pixel Point multiplication; then connect the image feature obtained after the point multiplication process to the image feature of the original resolution input image with a resolution of W/2×H/2, that is, merge channel processing; finally, the resolution is W/ The image features processed by the 2×H/2 combined channel are up-sampled and convolved to obtain the original resolution-enhanced image.

It should be understood that the foregoing description is based on an example of a 3-level progressive super-division process with a resolution of W/4×H/4, a resolution of W/2×H/2, and a resolution of W×H. Others can also be used. This application does not make any restrictions on the multi-level progressive reduction of.

Exemplarily, FIG. 17 shows a schematic diagram of HDR correction through a color guide map after introducing multi-scale image features. As shown in Fig. 17, a detailed description is given as an example of super-division of a YUV domain image, where the low-resolution image can refer to the image characteristics of the U channel and the V channel of the low-resolution HDR image, and the original resolution image can refer to The image features of the Y channel extracted from the image output by the self-guided multi-level super-splitting unit 722; the image features of the low-resolution U channel and the V channel are up-sampled, so that the scale of the image feature is the same as the original resolution image scale ; Next, connect the image features of the U channel and the V channel after the upsampling process with the Y channel image features of the original resolution image, that is, the combined channel operation; and then convert the combined channel processed YUV image into an RGB image; Perform an up-sampling operation on the color guide image to make the scale of the color guide image the same as the original resolution image; perform a dot multiplication operation on the converted RGB image and the color guide image after the up-sampling operation, that is, multiply pixel by pixel; Perform HDR correction, that is, convolution operation, on the image after the dot multiplication operation, so that the difference between the original resolution-enhanced image and the true value image that is finally output is smaller than the preset threshold.

It should be understood that the self-encoding and decoding network in FIG. 17 may be the same as that in FIG. 12, and will not be repeated here; for the same reason, the HDR correction unit may also be the same as the HDR correction unit in FIG. 12, and see the description in FIG. 12 , I won’t repeat it here.

Table 1

Table 1 is the performance test result of the image enhancement method and the benchmark algorithm of the embodiment of the application. Table 1 shows the performance test results of the model in this application and several existing models when performing image enhancement processing. Among them, the test index is the peak signal to noise ratio (PSNR) and the structure is similar. Structural similarity index (SSIM) and calculation amount. The calculation amount can be used to represent the number of times of multiply accumulate (MAC), 1G=10^ ⁹ .

From the test results shown in Table 1, it can be seen that the quantitative indicators of the neural network model of this solution are PSNR and SSIM far exceeding the global scaling model (Range scaling global U-Net, RSGUnet) and self-guided network (Self-guided network, SGN). Compared with HDRNet, with PSNR and SSIM loss of 1.85% and 0.26%, the processing time of a single frame is reduced by 56.2%, and the calculation amount of a single frame is reduced by 86.6%. In the same test environment, it can be seen that the image enhancement method provided by the embodiment of the present application has better processing speed and computational overhead, and can simultaneously meet the requirements of image quality enhancement effects and real-time processing.

18 to 20 are schematic diagrams of the evaluation results of the visual quality image enhancement quality provided by the embodiments of the present application. Among them, (a) in FIG. 18 is the input image, that is, the image to be processed; (b) in FIG. 18 is the true value image corresponding to the input image; (c) in FIG. 18 is the input image through the image enhancement method of this application The output image obtained by image enhancement processing; Figure 18 (d) is the output image obtained by enhancing the input image through the HDRNet model; Figure 18 (e) is the output image obtained by enhancing the input image through the RSGUnet model Output image; Figure 18 (f) is the output image obtained by enhancing the input image through the SGN model; from Figure 18 (a) to Figure 18 (f), it can be seen that through the embodiment of the application The output image obtained is the closest to the true value image, and there are no artifacts, etc.; in the same way, it can be seen from (a) in FIG. 19 to (f) in FIG. 19 that the output image obtained by the embodiment of the present application Regarding the sky halo problem, that is, the image enhancement method of the embodiment of the present application can reduce the halo problem while ensuring the HDR effect; similarly, it can be seen from (a) in FIG. 20 to (f) in FIG. 20 In the output image obtained through the embodiment of the present application, for the highlight part, the color is closer to the true value image, and the texture details of the leaf veins are restored more clearly.

It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples given, and such modifications or changes also fall within the scope of the embodiments of the present application.

The image enhancement method provided by the embodiment of the present application is described in detail above with reference to Figs. 1 to 20, and the device embodiment of the present application will be described in detail below with reference to Figs. 21 and 22. It should be understood that the image enhancement device in the embodiment of the present application can execute the various image enhancement methods of the foregoing embodiment of the present application, that is, for the specific working processes of the following various products, reference may be made to the corresponding process in the foregoing method embodiment.

FIG. 21 is a schematic block diagram of an image enhancement device provided by an embodiment of the present application. It should be understood that the image enhancement apparatus 800 may execute the image enhancement method shown in FIG. 10. The image enhancement device 800 includes: an acquisition unit 810 and a processing unit 820.

Wherein, the acquiring unit 810 is configured to acquire the first high dynamic range HDR image corresponding to the image to be processed and the color image characteristics of the image to be processed, wherein the image to be processed is an image of the first resolution, so The first HDR image is an image with a second resolution, the first resolution is greater than the second resolution, and the color image feature is used to indicate areas of different brightness or areas of different color changes in the image to be processed The processing unit 820 is configured to input the first HDR image into a neural network model for super-resolution processing; use the neural network model to perform super-resolution processing on the first HDR image and the color The image feature is subjected to image enhancement processing to obtain a second HDR image corresponding to the image to be processed, where the second HDR image refers to an HDR image with a resolution of the first resolution.

Optionally, as an embodiment, the acquiring unit 810 is further configured to:

Acquiring a texture image feature of the image to be processed, where the texture image feature is used to indicate an edge area or a texture area of the image to be processed;

The processing unit 820 is specifically configured to:

Perform super-resolution processing on the first HDR image according to the texture image feature through the neural network model.

Optionally, as an embodiment, the acquiring unit 810 is further configured to:

The processing unit 820 is specifically configured to:

Optionally, as an embodiment, the acquiring unit 810 is further configured to:

Acquire the texture image feature of the image to be processed, wherein the texture image feature is used to indicate the edge area or the texture area of the image to be processed; acquire the multi-scale image feature of the image to be processed, wherein the multiple The scale image feature is used to indicate the image information of the image to be processed in the case of different scales, and the scale size of any one of the multi-scale image features is different;

The processing unit 820 is specifically configured to:

Optionally, as an embodiment, the acquiring unit 810 is specifically configured to:

The processing unit 820 is specifically configured to:

Use the neural network model to perform a dot product operation on the first HDR image and the texture image feature of the first scale to obtain a third image feature; use the neural network model to compare the third image feature with the The first-scale image features perform combined channel operation, convolution operation, and up-sampling operation.

Optionally, as an embodiment, the processing unit 820 is specifically configured to:

It should be noted that the above-mentioned image enhancement device 800 is embodied in the form of a functional unit. The term "unit" herein can be implemented in the form of software and/or hardware, which is not specifically limited.

For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that realizes the above-mentioned functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.

Therefore, the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

FIG. 22 is a schematic diagram of the hardware structure of an image enhancement device provided by an embodiment of the present application. The image enhancement apparatus 900 shown in FIG. 22 (the apparatus 900 may specifically be a computer device) includes a memory 901, a processor 902, a communication interface 903, and a bus 904. Among them, the memory 901, the processor 902, and the communication interface 903 implement communication connections between each other through the bus 904.

The memory 901 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 may store a program. When the program stored in the memory 901 is executed by the processor 902, the processor 902 is configured to execute each step of the image enhancement method of the embodiment of the present application, for example, execute each step shown in FIG. 10 to FIG. step.

It should be understood that the image enhancement device shown in the embodiment of the present application may be a server, for example, it may be a server in the cloud, or may also be a chip configured in a server in the cloud; or, the image enhancement device shown in the embodiment of the present application The device can be a smart terminal or a chip configured in the smart terminal.

The image enhancement method disclosed in the above embodiments of the present application may be applied to the processor 902 or implemented by the processor 902. The processor 902 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above-mentioned image enhancement method can be completed by hardware integrated logic circuits in the processor 902 or instructions in the form of software. For example, the processor 902 may be a chip including the NPU shown in FIG. 8.

The aforementioned processor 902 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor, a digital signal processor (digital signal processor, DSP), and an application specific integrated circuit (application integrated circuit). specific integrated circuit, ASIC), ready-made programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in random access memory (RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc. mature in the field Storage medium. The storage medium is located in the memory 901, and the processor 902 reads the instructions in the memory 901, and combines its hardware to complete the functions required by the units included in the image enhancement device shown in FIG. 21 in the implementation of this application, or execute the method of this application Each step of the image enhancement method shown in FIG. 10 to FIG. 17 of the embodiment.

The communication interface 903 uses a transceiving device such as but not limited to a transceiver to implement communication between the device 900 and other devices or a communication network.

The bus 904 may include a path for transferring information between various components of the image intensifying device 900 (for example, the memory 901, the processor 902, and the communication interface 903).

It should be noted that although the above-mentioned image enhancement device 900 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the image enhancement device 900 may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned image enhancement device 900 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned image enhancement device 900 may also only include the necessary devices for implementing the embodiments of the present application, and not necessarily all the devices shown in FIG. 22.

The embodiment of the present application also provides a chip, which includes a transceiver unit and a processing unit. Wherein, the transceiver unit may be an input/output circuit or a communication interface; the processing unit is a processor, microprocessor, or integrated circuit integrated on the chip. The chip can execute the image enhancement method in the above method embodiment.

The embodiment of the present application also provides a computer-readable storage medium with an instruction stored thereon, and the image enhancement method in the foregoing method embodiment is executed when the instruction is executed.

The embodiments of the present application also provide a computer program product containing instructions that, when executed, execute the image enhancement method in the foregoing method embodiments.

It should also be understood that, in the embodiments of the present application, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the processor may also include a non-volatile random access memory. For example, the processor may also store device type information.

It should be understood that the term "and/or" in this text is only an association relationship describing the associated objects, indicating that there can be three types of relationships, for example, A and/or B, which can mean: A alone exists, and both A and B exist. , There are three cases of B alone. In addition, the character "/" in this text generally indicates that the associated objects before and after are in an "or" relationship.

It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application. The implementation process constitutes any limitation.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device, and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

An image enhancement method, characterized in that it comprises:

Acquiring the first high dynamic range HDR image corresponding to the image to be processed and the color image characteristics of the image to be processed, wherein the color image characteristics are used to indicate areas of different brightness or areas of different color changes in the image to be processed, The image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution;

Inputting the first HDR image into a neural network model for super-division processing;

Perform image enhancement processing on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain a second HDR image corresponding to the image to be processed, wherein the first HDR image is The second HDR image refers to an HDR image with a resolution of the first resolution.
8. The image enhancement method of claim 1, further comprising:

Acquiring a texture image feature of the image to be processed, where the texture image feature is used to indicate an edge area or a texture area of the image to be processed;

The inputting the first HDR image into a neural network model for super-resolution processing includes:

Perform super-resolution processing on the first HDR image according to the texture image feature through the neural network model.
8. The image enhancement method of claim 1, further comprising:

Acquire the multi-scale image feature of the image to be processed, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;

The inputting the first HDR image into a neural network model for super-resolution processing:

Performing super-resolution processing on the first HDR image according to the multi-scale image feature through the neural network model.
8. The image enhancement method of claim 1, further comprising:

Acquiring a texture image feature of the image to be processed, where the texture image feature is used to indicate an edge area or a texture area of the image to be processed;

Acquire the multi-scale image feature of the image to be processed, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;

The inputting the first HDR image into a neural network model for super-resolution processing includes:

Perform super-resolution processing on the first HDR image according to the texture image feature and the multi-scale feature through the neural network model.
The image enhancement method according to claim 4, wherein the super-resolution processing of the first HDR image according to the texture image feature and the multi-scale image feature through the neural network model includes :

Acquiring a texture image feature of a first scale, wherein the texture image feature of the first scale has the same scale as the first scale image feature in the multi-scale image;

Performing a dot multiplication operation on the first HDR image and the texture image feature of the first scale through the neural network model to obtain a third image feature;

Perform a combined channel operation, a convolution operation, and an up-sampling operation on the third image feature and the first-scale image feature through the neural network model.
The image enhancement method according to any one of claims 1 to 5, wherein the first HDR image and the color image feature after the super-resolution processing are processed by the neural network model Performing image enhancement processing to obtain the second HDR image corresponding to the image to be processed includes:

Performing a dot multiplication operation and a convolution operation on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain the second HDR image.
An image enhancement device, characterized in that it comprises:

The acquiring unit is configured to acquire the first high dynamic range HDR image corresponding to the image to be processed and the color image feature of the image to be processed, wherein the color image feature is used to indicate different brightness regions in the image to be processed or Different color change areas, the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution;

A processing unit, configured to input the first HDR image into a neural network model for super-resolution processing; image the first HDR image after the super-resolution processing and the color image characteristics through the neural network model The enhancement processing is performed to obtain a second HDR image corresponding to the image to be processed, where the second HDR image refers to an HDR image with a resolution of the first resolution.
8. The image enhancement device according to claim 7, wherein the acquiring unit is further configured to:

Acquiring a texture image feature of the image to be processed, where the texture image feature is used to indicate an edge area or a texture area of the image to be processed;

The processing unit is specifically used for:

Perform super-resolution processing on the first HDR image according to the texture image feature through the neural network model.
8. The image enhancement device according to claim 7, wherein the acquiring unit is further configured to:

Acquire the multi-scale image feature of the image to be processed, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;

The processing unit is specifically used for:

Performing super-resolution processing on the first HDR image according to the multi-scale image feature through the neural network model.
8. The image enhancement device according to claim 7, wherein the acquiring unit is further configured to:

Acquiring a texture image feature of the image to be processed, where the texture image feature is used to indicate an edge area or a texture area of the image to be processed;

Acquire the multi-scale image feature of the image to be processed, where the multi-scale image feature is used to indicate the image information of the image to be processed under different scales, and any one of the multi-scale image features The size of the scale is different;

The processing unit is specifically used for:

Perform super-resolution processing on the first HDR image according to the texture image feature and the multi-scale feature through the neural network model.
The image enhancement device according to claim 10, wherein the acquiring unit is specifically configured to:

Acquiring a texture image feature of a first scale, wherein the texture image feature of the first scale has the same scale as the first scale image feature in the multi-scale image;

The processing unit is specifically used for:

Performing a dot multiplication operation on the first HDR image and the texture image feature of the first scale by using the neural network model to obtain a third image feature;

Perform a combined channel operation, a convolution operation, and an up-sampling operation on the third image feature and the first-scale image feature through the neural network model.
The image enhancement device according to any one of claims 7 to 11, wherein the processing unit is specifically configured to:

Performing a dot multiplication operation and a convolution operation on the super-resolution processed first HDR image and the color image feature through the neural network model to obtain the second HDR image.
An image enhancement device, characterized in that it comprises:

Memory, used to store programs;

The processor is configured to execute the program stored in the memory, and when the processor executes the program stored in the memory, the processor is configured to execute the image enhancement method according to any one of claims 1 to 6.
A computer-readable medium, wherein the computer-readable medium stores program code, and when the computer program code runs on a computer, the computer executes the method described in any one of claims 1 to 6 Image enhancement method.