CN112581379A

CN112581379A - Image enhancement method and device

Info

Publication number: CN112581379A
Application number: CN201910943355.7A
Authority: CN
Inventors: 宋风龙; 熊志伟; 王宪; 黄杰; 查正军
Original assignee: University of Science and Technology of China USTC; Huawei Technologies Co Ltd
Current assignee: University of Science and Technology of China USTC; Huawei Technologies Co Ltd
Priority date: 2019-09-30
Filing date: 2019-09-30
Publication date: 2021-03-30
Also published as: WO2021063341A1

Abstract

The application discloses an image enhancement method and an image enhancement device in the field of computer vision in the field of artificial intelligence, wherein the image enhancement method comprises the following steps: acquiring an image to be processed; performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, wherein the neural network comprises N convolutional layers, and N is a positive integer; and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image. According to the technical scheme, the performance of the image to be processed in the aspects of details, color and brightness is improved, and therefore the image enhancement processing effect is improved.

Description

Image enhancement method and device

Technical Field

The present application relates to the field of artificial intelligence, and more particularly, to an image enhancement method and apparatus in the field of computer vision.

Background

Computer vision is an integral part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, military and the like, and is a study on how to use cameras/video cameras and computers to acquire data and information of a photographed object which are required by us. In a descriptive sense, a computer is provided with eyes (camera/camcorder) and a brain (algorithm) to recognize, track, measure, etc. a target instead of human eyes, thereby enabling the computer to perceive the environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be viewed as the science of how to make an artificial system "perceive" from images or multidimensional data. Generally, computer vision is to use various imaging systems to obtain input information instead of visual organs, and then use computer to process and interpret the input information instead of brain. The ultimate research goal of computer vision is to make a computer have the ability to adapt to the environment autonomously by visually observing and understanding the world like a human.

Image enhancement is an important branch in the field of image processing, and the image quality can be improved by the image enhancement technology without data re-acquisition so as to meet more practical application requirements. For example, the image enhancement technology can make an original unclear image clear or emphasize some interesting features by purposefully emphasizing the overall or local characteristics of the image (or video), enlarge the difference between different object features in the image, inhibit the uninteresting features, improve the image quality, enrich the information content, enhance the image interpretation and recognition effect, and meet the needs of some special analyses.

In the field of computer vision, it is often necessary to acquire an image (or video) with an acquisition device and to identify or analyze the image. Under the influence of the shooting environment of the acquisition equipment or other unknown factors, the acquired image has the phenomena of blurring, low contrast and the like, the display effect of the image, the analysis and the identification of the image and the like are influenced due to low imaging quality, and the image or the video can be enhanced firstly and then identified or analyzed. However, at present, in many cases, the effect of image enhancement processing is not ideal. Therefore, how to improve the effect of image enhancement processing becomes an urgent problem to be solved.

Disclosure of Invention

The application provides an image enhancement method and an image enhancement device, which can enhance the performance of an image to be processed in the aspects of details, colors and brightness, so that the effect of image enhancement processing can be improved.

In a first aspect, an image enhancement method is provided, including: acquiring an image to be processed; performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, wherein the neural network comprises N convolutional layers, and N is a positive integer; and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.

For example, the image to be processed may be an original image with poor image quality; for example, the problem may be that the acquired image to be processed has a blurred image, a low image quality, a low color and brightness, or the like due to the influence of weather, distance, shooting environment, or the like.

It should be understood that the color enhancement processing described above can be used to improve the color distribution of the image to be processed, and improve the color saturation of the image to be processed; the brightness enhancement processing may refer to adjusting brightness of an image to be processed; feature enhancement processing may refer to enhancing details in an image such that more detailed information is included in the image; for example, the feature enhancement processing may refer to performing detail feature enhancement on the image to be processed.

According to the image enhancement method provided by the embodiment of the application, the image enhancement feature of the image to be processed is obtained by performing the feature enhancement processing on the image to be processed, and the color enhancement processing and the brightness enhancement processing are further performed on the image to be processed by utilizing the image enhancement feature, so that the detail feature of the image to be processed can be enhanced when the color enhancement and the brightness enhancement are performed, the performances of the image to be processed in the aspects of detail, color and brightness are enhanced, and the image enhancement processing effect is improved.

The image enhancement may also be referred to as image quality enhancement, and may refer to processing, for example, brightness, color, contrast, saturation, and/or dynamic range of an image so that each index of the image satisfies a preset condition.

With reference to the first aspect, in certain implementations of the first aspect, the performing, by a neural network, the feature enhancement processing on the image to be processed to obtain an enhanced image feature of the image to be processed includes: and performing the feature enhancement processing on the image to be processed by using a Laplace enhancement algorithm to obtain the enhanced image features of the image to be processed.

In the embodiment of the application, the feature enhancement of the image to be processed can be realized through the Laplace enhancement algorithm, wherein the Laplace enhancement algorithm can realize that new textures cannot be introduced when the feature enhancement is carried out on the image to be processed, so that the problem of introducing pseudo textures into an output image after the image enhancement processing is avoided to a certain extent, and the effect of the image enhancement processing can be improved.

In a possible implementation manner, the laplacian enhancement algorithm may be to enhance high-frequency features in the image to be processed, so as to obtain enhanced image features of the image to be processed.

The high-frequency feature of the image may refer to details, textures, and other information of the image to be processed.

With reference to the first aspect, in some implementations of the first aspect, the laplacian enhancement algorithm is configured to perform the feature enhancement processing on the input image feature of an ith convolutional layer according to a residual feature of an ith convolutional layer of the N convolutional layers, so as to obtain an enhanced image feature of the ith convolutional layer, where the residual feature represents a difference between the input image feature of the ith convolutional layer and an image feature of the ith convolutional layer after being processed by a convolution operation, the enhanced image feature of the ith convolutional layer is an input image feature of the (i + 1) th convolutional layer, the input image feature is obtained according to the image to be processed, and i is a positive integer.

In the embodiment of the present application, the laplacian enhancement algorithm may be an improved laplacian enhancement algorithm, and the improved laplacian enhancement algorithm according to the embodiment of the present application may use the image features in the previous convolution layer for enhancement of the subsequent image features, so as to realize progressive gradual enhancement of the image features of different convolution layers, and improve the effect of image enhancement processing.

With reference to the first aspect, in certain implementations of the first aspect, the enhanced image feature of the image to be processed is an image feature output by an nth convolutional layer of the N convolutional layers, and the enhanced image feature of the image to be processed is obtained by the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

wherein, L (F)_N) Features of the enhanced image representing said Nth convolution layer, F_NRepresenting input image characteristics of said Nth convolution layer, [ phi ] representing convolution kernel of said Nth convolution layer, [ s ]_lDenotes a scaling parameter obtained by learning, and N is a positive integer.

In the embodiment of the present application, the parameter s can be learned through_lReplacing the fixed scaling factor s in the conventional Laplace enhancement algorithm_c(ii) a Meanwhile, the residual characteristics of adjacent layers are adopted for characteristic enhancement, and the residual characteristics can be used for representing any information needing to be emphasized. Therefore, the laplacian algorithm in the embodiment of the application can not only enhance the high-frequency information of the image, but also realize the gradual and gradual enhancement of the image features of different convolution layers, thereby improving the effect of image enhancement processing.

With reference to the first aspect, in some implementation manners of the first aspect, the performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image includes: obtaining a confidence image feature and an illumination compensation image feature of the image to be processed according to the enhanced image feature of the image to be processed, wherein the confidence image feature is used for performing color enhancement on the image to be processed, and the illumination compensation image feature is used for performing brightness enhancement on the image to be processed; and obtaining the output image according to the image to be processed, the confidence image characteristic and the illumination compensation image characteristic.

It should be noted that the above-mentioned confidence image feature may represent a mapping relationship or a mapping function for performing color enhancement processing on the image to be processed.

For example, the confidence image feature may correspond to an image feature of the image to be processed, for example, one element in the confidence image feature may be used to indicate a zoom degree of the corresponding element in the image feature of the image to be processed; color enhancement of the image to be processed can be achieved by scaling different regions in the image to be processed.

It should be understood that the enhanced image features of the image to be processed may include more detailed features and textures; the color enhancement processing and the brightness enhancement processing are carried out on the image to be processed according to the enhanced image characteristics of the image to be processed, so that the detail enhancement of the output image can be realized, and meanwhile, the brightness and the color of the output image can also be improved.

In the embodiment of the application, the confidence image features for color enhancement processing and the illumination compensation image features for brightness enhancement processing can be obtained through the enhanced image features of the image to be processed, and compared with a traditional scaling method that the colors and the brightness of all pixel points in the image to be processed are directly enhanced through a mapping function, the confidence image features and the illumination compensation image features in the embodiment of the application can not only enhance the colors and the brightness, but also enhance the details of the image to be processed, so that the effect of image enhancement processing is improved.

With reference to the first aspect, in certain implementations of the first aspect, the method further includes: performing convolution operation on the enhanced image features of the image to be processed to obtain the confidence image features and the illumination compensation image features; obtaining the color enhancement image characteristic of the image to be processed by multiplying the image characteristic of the image to be processed with the confidence image characteristic; and fusing the color enhanced image characteristics and the illumination compensation image characteristics to obtain the output image.

In a possible implementation manner, the confidence image feature and the illumination compensation image feature may be obtained in parallel through an enhanced image feature of an image to be processed; for example, the confidence image feature may be obtained by performing convolution operation on the enhanced image feature of the image to be processed through the first branch in the network model; and carrying out convolution operation on the enhanced image characteristics of the image to be processed through a second branch in the network model to obtain the illumination compensation image characteristics.

In a possible implementation manner, the blending of the color enhanced image feature and the illumination compensation image feature may refer to adding the color enhanced image feature and the illumination compensation image feature.

In a second aspect, an image enhancement method is provided, including: detecting a first operation of a user for turning on a camera; in response to the first operation, displaying a shooting interface on the display screen, wherein a view frame is included on the shooting interface, and a first image is included in the view frame; detecting a second operation of the user-indicated camera; and responding to the second operation, displaying a second image in the view frame, or saving the second image in the electronic equipment, wherein the second image is obtained by performing color enhancement processing and brightness enhancement processing on the first image according to the enhanced image characteristics of the first image, the enhanced image characteristics of the first image are obtained by performing feature enhancement processing on the first image through a neural network, the neural network comprises N convolutional layers, and N is a positive integer.

The specific flow of the feature enhancement processing on the first image may be obtained according to any one of the implementation manners of the first aspect and the first aspect.

In a possible implementation manner, the image enhancement method provided in the embodiment of the present application may be applied to the field of photographing of an intelligent terminal, and the image enhancement method in the embodiment of the present application may perform image enhancement on an original image with poor image quality acquired by the intelligent terminal to obtain an output image with improved image quality, for example, when the intelligent terminal performs real-time photographing, the acquired original image may be subjected to image enhancement, and the output image after the image enhancement is displayed on a screen of the intelligent terminal, or the acquired original image may be subjected to image enhancement, and the output image after the image enhancement is stored in an album of the intelligent terminal.

In a third aspect, an image enhancement method is provided, including: acquiring a road picture to be processed; carrying out feature enhancement processing on the road picture to be processed through a neural network to obtain enhanced image features of the road picture to be processed, wherein the neural network comprises N convolutional layers, and N is a positive integer; carrying out color enhancement processing and brightness enhancement processing on the road picture to be processed according to the enhanced image characteristics to obtain a processed output road picture; and identifying information in the output road picture according to the processed output road picture.

The specific flow of the feature enhancement processing on the road picture may be obtained according to any one of the implementation manners of the first aspect and the first aspect.

In a possible implementation manner, the image enhancement method provided by the embodiment of the application can be applied to the field of automatic driving. For example, the method can be applied to a navigation system of an automatic driving vehicle, and the image enhancement method in the application can perform image enhancement processing on an original road picture with low image quality obtained in the process of navigating the automatic driving vehicle on the road to obtain the road picture after the image enhancement processing, so that the safety of the automatic driving vehicle is realized.

In a fourth aspect, an image enhancement method is provided, including: obtaining a street view picture; carrying out feature enhancement processing on the street view picture through a neural network to obtain enhanced image features of the street view picture, wherein the neural network comprises N convolution layers, and N is a positive integer; carrying out color enhancement processing and brightness enhancement processing on the street view picture according to the enhanced image characteristics to obtain a processed output street view picture; and identifying information in the output street view picture according to the processed output street view picture.

The specific flow of the feature enhancement processing on the street view picture may be obtained according to any one of the first aspect and the implementation manner of the first aspect.

In a possible implementation manner, the image enhancement method provided by the embodiment of the application can be applied to the field of security protection. For example, the image enhancement method of the embodiment of the application can be applied to the enhancement of the monitoring image of a safe city, for example, the image (or video) collected by the monitoring equipment in a public place is often influenced by factors such as weather and distance, and the problems of image blur, low image quality and the like exist. The image enhancement method can be used for carrying out image enhancement on the collected original image, thereby recovering important information such as license plate numbers, clear human faces and the like for public security personnel and providing important clue information for case detection.

It is to be understood that extensions, definitions, explanations and explanations of relevant matters in the above-described first aspect also apply to the same matters in the second, third and fourth aspects.

In a fifth aspect, an image enhancement apparatus is provided, which includes means for performing the image enhancement method in any one implementation manner of the first to fourth aspects and the first to fourth aspects.

In a sixth aspect, there is provided an image enhancement apparatus comprising: a memory for storing a program; a processor for executing the memory-stored program, the processor for performing, when the memory-stored program is executed: acquiring an image to be processed; performing feature enhancement processing on the image to be processed to obtain enhanced image features of the image to be processed; and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.

In a possible implementation manner, the processor included in the image enhancement apparatus is further used in the image enhancement method in any one implementation manner of the first to fourth aspects and the first to fourth aspects.

In a seventh aspect, a computer-readable medium is provided, which stores program code for execution by a device, where the program code includes instructions for performing the image enhancement method in any one of the implementations of the first to fourth aspects and the first to fourth aspects.

In an eighth aspect, a computer program product containing instructions is provided, which when run on a computer causes the computer to perform the image enhancement method in any one of the implementations of the first to fourth aspects and the first to fourth aspects.

In a ninth aspect, a chip is provided, where the chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface and executes the image enhancement method in any implementation manner of the first to fourth aspects and the first to fourth aspects.

Optionally, as an implementation manner, the chip may further include a memory, where instructions are stored in the memory, and the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the processor is configured to execute the image enhancement method in any one implementation manner of the first to fourth aspects and the first to fourth aspects.

Drawings

FIG. 1 is a schematic diagram of an artificial intelligence agent framework provided by an embodiment of the present application;

fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application;

fig. 3 is a schematic diagram of another application scenario provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of another application scenario provided by an embodiment of the present application;

fig. 5 is a schematic diagram of another application scenario provided in an embodiment of the present application

FIG. 6 is a block diagram of a system architecture according to an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of a convolutional neural network provided in an embodiment of the present application;

fig. 8 is a schematic diagram of a chip hardware structure according to an embodiment of the present disclosure;

FIG. 9 is a diagram illustrating a system architecture according to an embodiment of the present application;

FIG. 10 is a schematic flow chart diagram of an image enhancement method provided by an embodiment of the present application;

FIG. 11 is a schematic structural diagram of an image enhancement model provided in an embodiment of the present application;

fig. 12 is a schematic diagram of a laplacian enhancement unit and a hybrid enhancement unit provided in an embodiment of the present application;

FIG. 13 is a schematic diagram of a feature enhancement process provided by an embodiment of the present application;

FIG. 14 is a diagram illustrating a color enhancement process and a brightness enhancement process according to an embodiment of the present disclosure;

FIG. 15 is a diagram illustrating a color enhancement process and a brightness enhancement process according to an embodiment of the present disclosure;

FIG. 16 is a schematic illustration of a visual quality assessment provided by an embodiment of the present application;

FIG. 17 is a schematic flow chart diagram of an image enhancement method provided by an embodiment of the present application;

FIG. 18 is a schematic diagram of a set of display interfaces provided by embodiments of the present application;

FIG. 19 is a schematic diagram of another set of display interfaces provided by embodiments of the present application;

fig. 20 is a schematic block diagram of an image enhancement apparatus provided by an embodiment of the present application;

fig. 21 is a schematic hardware structure diagram of an image enhancement apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be understood that the image in the embodiment of the present application may be a still image (or referred to as a still picture) or a dynamic image (or referred to as a dynamic picture), for example, the image in the present application may be a video or a dynamic picture, or the image in the present application may also be a still picture or a photo. For convenience of description, the present application collectively refers to a still image or a moving image as an image in the following embodiments.

FIG. 1 shows a schematic diagram of an artificial intelligence body framework that describes the overall workflow of an artificial intelligence system, applicable to the general artificial intelligence field requirements.

The artificial intelligence theme framework 100 described above is described in detail below in terms of two dimensions, the "intelligent information chain" (horizontal axis) and the "Information Technology (IT) value chain" (vertical axis).

The "smart information chain" reflects a list of processes processed from the acquisition of data. For example, the general processes of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision making and intelligent execution and output can be realized. In this process, the data undergoes a "data-information-knowledge-wisdom" refinement process.

The 'IT value chain' reflects the value of the artificial intelligence to the information technology industry from the bottom infrastructure of the human intelligence, information (realization of providing and processing technology) to the industrial ecological process of the system.

(1) Infrastructure 110

The infrastructure provides computing power support for the artificial intelligent system, realizes communication with the outside world, and realizes support through a foundation platform.

The infrastructure may communicate with the outside through sensors, and the computing power of the infrastructure may be provided by a smart chip.

The intelligent chip may be a hardware acceleration chip such as a Central Processing Unit (CPU), a neural-Network Processing Unit (NPU), a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The infrastructure platform may include distributed computing framework and network, and may include cloud storage and computing, interworking network, and the like.

For example, for an infrastructure, data may be obtained through sensors and external communications and then provided to an intelligent chip in a distributed computing system provided by the base platform for computation.

(2) Data 120

Data at the upper level of the infrastructure is used to represent the data source for the field of artificial intelligence. The data relates to graphics, images, voice and text, and also relates to internet of things data of traditional equipment, including service data of an existing system and sensing data such as force, displacement, liquid level, temperature, humidity and the like.

(3) Data processing 130

The data processing generally includes processing modes such as data training, machine learning, deep learning, searching, reasoning, decision making and the like.

The machine learning and the deep learning can perform symbolized and formalized intelligent information modeling, extraction, preprocessing, training and the like on data.

Inference means a process of simulating an intelligent human inference mode in a computer or an intelligent system, using formalized information to think about and solve a problem by a machine according to an inference control strategy, and a typical function is searching and matching.

The decision-making refers to a process of making a decision after reasoning intelligent information, and generally provides functions of classification, sequencing, prediction and the like.

(4) General capabilities 140

After the above-mentioned data processing, further based on the result of the data processing, some general capabilities may be formed, such as algorithms or a general system, e.g. translation, analysis of text, computer vision processing, speech recognition, recognition of images, etc.

(5) Intelligent product and industry applications 150

The intelligent product and industry application refers to the product and application of an artificial intelligence system in various fields, and is the encapsulation of an artificial intelligence integral solution, the intelligent information decision is commercialized, and the landing application is realized, and the application field mainly comprises: intelligent manufacturing, intelligent transportation, intelligent home, intelligent medical treatment, intelligent security, automatic driving, safe city, intelligent terminal and the like.

Fig. 2 is a schematic diagram of an application scenario of an image enhancement method provided in an embodiment of the present application.

As shown in fig. 2, the technical solution of the embodiment of the present application may be applied to an intelligent terminal, and the image enhancement method in the embodiment of the present application may perform image enhancement processing on an input image to obtain an output image of the input image after image enhancement. The smart terminal may be mobile or fixed, for example, the smart terminal may be a mobile phone with an image enhancement function, a Tablet Personal Computer (TPC), a media player, a smart tv, a Laptop Computer (LC), a Personal Digital Assistant (PDA), a Personal Computer (PC), a camera, a camcorder, a smart watch, an Augmented Reality (AR)/Virtual Reality (VR), a Wearable Device (WD), or an autonomous vehicle, which is not limited in the embodiments.

It should be noted that, in the embodiment of the present application, image enhancement may also be referred to as image quality enhancement, and specifically, brightness, color, contrast, saturation, dynamic range, and/or the like of an image may be processed so that each index of the image satisfies a preset condition. In the embodiments of the present application, image enhancement and image quality enhancement have the same meaning.

Specific application scenarios of the embodiments of the present application are illustrated below.

The application scene one: intelligent terminal photographing field

In one embodiment, as shown in fig. 3, the image enhancement method of the embodiment of the present application may be applied to shooting of a smart terminal device (e.g., a mobile phone). According to the image enhancement method, the obtained original image with poor quality can be subjected to image enhancement processing, and an output image with improved image quality can be obtained.

In fig. 3, a color image portion is indicated by diagonal filling in order to be distinguished from a grayscale image portion.

Illustratively, when the intelligent terminal takes a picture in real time, the image enhancement method of the embodiment of the application performs image enhancement processing on the acquired original image, and displays the output image after the image enhancement processing on the screen of the intelligent terminal.

For example, the image enhancement method according to the embodiment of the application can be used for performing image enhancement processing on the acquired original image, and saving the output image after the image enhancement processing into an album of the intelligent terminal.

The application provides an image enhancement method, which is applied to an electronic device with a display screen and a camera, and comprises the following steps: detecting a first operation of a user for turning on a camera; in response to the first operation, displaying a shooting interface on the display screen, wherein a view frame is included on the shooting interface, and a first image is included in the view frame; detecting a second operation of the user-indicated camera; and responding to the second operation, displaying a second image in the view frame, or saving the second image in the electronic equipment, wherein the second image is obtained by performing color enhancement processing and brightness enhancement processing on the first image according to the enhanced image characteristics of the first image, the enhanced image characteristics of the first image are obtained by performing feature enhancement processing on the first image through a neural network, the neural network comprises N convolutional layers, and N is a positive integer.

It should be noted that the image enhancement method provided in the embodiment of the present application is also applicable to the extension, limitation, explanation and explanation of the related content of the image enhancement method in the related embodiments in fig. 6 to fig. 16, which are not described herein again.

Application scenario two: field of autonomous driving

In one embodiment, as shown in fig. 4, the image enhancement method of the embodiment of the present application may be applied to the field of automatic driving. For example, the method can be applied to a navigation system of an automatic driving vehicle, and the image enhancement method in the application can perform image enhancement processing on an original road picture with low image quality obtained in the process of navigating the automatic driving vehicle on the road to obtain the road picture after the image enhancement processing, so that the safety of the automatic driving vehicle is realized.

Illustratively, the present application provides an image enhancement method, comprising: acquiring a road picture to be processed; carrying out feature enhancement processing on the road picture to be processed through a neural network to obtain enhanced image features of the road picture to be processed, wherein the neural network comprises N convolutional layers, and N is a positive integer; carrying out color enhancement processing and brightness enhancement processing on the road picture to be processed according to the enhanced image characteristics to obtain a processed output road picture; and identifying information in the output road picture according to the processed output road picture.

Application scenario three: the field of security protection

In one embodiment, as shown in fig. 5, the image enhancement method of the embodiment of the present application may be applied to the security field. For example, the image enhancement method of the embodiment of the application can be applied to the enhancement of the monitoring image of a safe city, for example, the image (or video) collected by the monitoring equipment in a public place is often influenced by factors such as weather and distance, and the problems of image blur, low image quality and the like exist. The image enhancement method can be used for enhancing the image of the collected image, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information is provided for case detection.

Illustratively, the present application provides an image enhancement method, comprising: obtaining a street view picture; carrying out feature enhancement processing on the street view picture through a neural network to obtain enhanced image features of the street view picture, wherein the neural network comprises N convolution layers, and N is a positive integer; carrying out color enhancement processing and brightness enhancement processing on the street view picture according to the enhanced image characteristics to obtain a processed output street view picture; and identifying information in the output street view picture according to the processed output street view picture.

And an application scene four: film source enhancement

In one embodiment, the image enhancement method of the embodiment of the present application can also be applied to a film source enhanced scene. For example, when a movie is played by using an intelligent terminal (e.g., an intelligent television, an intelligent screen, etc.), in order to display better image quality (image quality), the image enhancement method of the embodiment of the present application may be used to perform image enhancement processing on an original film source of the movie, so as to improve the image quality of the film source and obtain better visual impression.

For example, when an old movie (the source of the old movie is relatively early in time and the image quality of the source is poor) is played by using a smart television or a smart screen, the image enhancement method of the embodiment of the application can be used for performing image enhancement processing on the source of the old movie, so that the visual sense of the modern movie can be displayed. For example, the film source of the old movie can be enhanced into a high-dynamic range (HDR) 10 or a high-quality video of Dolby vision (Dolby vision) standard by the image enhancement method of the embodiment of the present application.

Illustratively, the present application provides an image enhancement method, comprising: acquiring an original image (e.g., an original film source of a movie); performing feature enhancement processing on the original image through a neural network to obtain enhanced image features of the original image, wherein the neural network comprises N convolutional layers, and N is a positive integer; and performing color enhancement processing and brightness enhancement processing on the original image according to the enhanced image characteristics to obtain a processed output image (for example, a film source for improving the image quality).

It should be understood that the above description is illustrative of the application scenario and does not limit the application scenario of the present application in any way.

Since the embodiments of the present application relate to the application of a large number of neural networks, for the sake of understanding, the following description will be made first of all with respect to terms and concepts of the neural networks to which the embodiments of the present application may relate.

(1) Neural network

The neural network may be composed of neural units, which may be referred to as x_sAnd an arithmetic unit with intercept 1 as input, the output of which may be:

wherein s is 1, 2, … … n, n is a natural number greater than 1, and W is_sIs x_sB is the bias of the neural unit. f is an activation function (activation functions) of the neural unit for introducing a nonlinear characteristic into the neural network to convert an input signal in the neural unit into an output signal. The output signal of the activation function may be used as an input for the next convolutional layer, and the activation function may be a sigmoid function. A neural network is a network formed by connecting together a plurality of the above-mentioned single neural units, i.e. oneThe output of a neural unit may be an input to another neural unit. The input of each neural unit can be connected with the local receiving domain of the previous layer to extract the characteristics of the local receiving domain, and the local receiving domain can be a region composed of a plurality of neural units.

(2) Deep neural network

Deep Neural Networks (DNNs), also called multi-layer neural networks, can be understood as neural networks with multiple hidden layers. The DNNs are divided according to the positions of different layers, and the neural networks inside the DNNs can be divided into three categories: input layer, hidden layer, output layer. Generally, the first layer is an input layer, the last layer is an output layer, and the middle layers are hidden layers. The layers are all connected, that is, any neuron of the ith layer is necessarily connected with any neuron of the (i + 1) th layer.

Although DNN appears complex, it is not really complex in terms of the work of each layer, simply the following linear relational expression:

wherein the content of the first and second substances,

is the input vector of the input vector,

is the output vector of the output vector,

is an offset vector, W is a weight matrix (also called coefficient), and α () is an activation function. Each layer is only for the input vector

Obtaining the output vector through such simple operation

Due to the large number of DNN layers, the coefficient W and the offset vector

The number of the same is also large. The definition of these parameters in DNN is as follows: taking coefficient W as an example: assume that in a three-layer DNN, the linear coefficients of the 4 th neuron of the second layer to the 2 nd neuron of the third layer are defined as

The superscript 3 represents the number of layers in which the coefficient W is located, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

In summary, the coefficients from the kth neuron at layer L-1 to the jth neuron at layer L are defined as

Note that the input layer is without the W parameter. In deep neural networks, more hidden layers make the network more able to depict complex situations in the real world. Theoretically, the more parameters the higher the model complexity, the larger the "capacity", which means that it can accomplish more complex learning tasks. The final goal of the process of training the deep neural network, i.e., learning the weight matrix, is to obtain the weight matrix (the weight matrix formed by the vectors W of many layers) of all the layers of the deep neural network that is trained.

(3) Convolutional neural network

A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network comprises a feature extractor consisting of convolutional layers and sub-sampling layers, which can be regarded as a filter. The convolutional layer is a neuron layer for performing convolutional processing on an input signal in a convolutional neural network. In convolutional layers of convolutional neural networks, one neuron may be connected to only a portion of the neighbor neurons. In a convolutional layer, there are usually several characteristic planes, and each characteristic plane may be composed of several neural units arranged in a rectangular shape. The neural units of the same feature plane share weights, where the shared weights are convolution kernels. Sharing weights may be understood as the way image information is extracted is location independent. The convolution kernel can be initialized in the form of a matrix of random size, and can be learned to obtain reasonable weights in the training process of the convolutional neural network. In addition, sharing weights brings the direct benefit of reducing connections between layers of the convolutional neural network, while reducing the risk of overfitting.

(4) Loss function

In the process of training the deep neural network, because the output of the deep neural network is expected to be as close to the value really expected to be predicted as possible, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the really expected target value (of course, an initialization process is usually carried out before the first updating, namely parameters are preset for each layer in the deep neural network), for example, if the predicted value of the network is high, the weight vector is adjusted to be lower, and the adjustment is continuously carried out until the deep neural network can predict the really expected target value or the value which is very close to the really expected target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which are loss functions (loss functions) or objective functions (objective functions), which are important equations for measuring the difference between the predicted value and the target value. Taking the loss function as an example, if the higher the output value (loss) of the loss function indicates the larger the difference, the training of the deep neural network becomes the process of reducing the loss as much as possible.

(5) Back propagation algorithm

The neural network can adopt a Back Propagation (BP) algorithm to correct the size of parameters in the initial neural network model in the training process, so that the reconstruction error loss of the neural network model is smaller and smaller. Specifically, the error loss is generated by transmitting the input signal in the forward direction until the output, and the parameters in the initial neural network model are updated by reversely propagating the error loss information, so that the error loss is converged. The back propagation algorithm is a back propagation motion with error loss as a dominant factor, aiming at obtaining the optimal parameters of the neural network model, such as a weight matrix.

Fig. 6 illustrates a system architecture 200 provided by an embodiment of the present application.

In fig. 6, a data acquisition device 260 is used to acquire training data. For the image enhancement method of the embodiment of the present application, the image enhancement model (also referred to as an image enhancement network) may be further trained through training data, that is, the training data acquired by the data acquisition device 260 may be training images.

For example, the training data for training the image enhancement model in the embodiment of the present application may include an original image and a sample enhanced image.

For example, the original image may be an image with a low image quality, the sample enhanced image may be an image with a high image quality, and for example, the original image may be an image with a brightness, a color, a detail, or one or more aspects thereof improved with respect to the sample image.

It should be understood that image enhancement may also be referred to as image quality enhancement, and specifically may refer to processing brightness, color, contrast, saturation, and/or dynamic range of an image, so that each index of the image satisfies a preset condition. In the embodiments of the present application, image enhancement and image quality enhancement have the same meaning.

After the training data is collected, the data collection device 260 stores the training data in the database 230, and the training device 220 trains the target model/rule 201 (i.e., the image enhancement model in the embodiment of the present application) based on the training data maintained in the database 230. The training device 220 inputs training data to the image enhancement model until a difference between a predicted enhanced image output by the training image enhancement model and a sample enhanced image satisfies a preset condition (e.g., the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold, or the difference between the predicted enhanced image and the sample enhanced image remains unchanged or no longer decreases), thereby completing the training of the target model/rule 201.

For example, the image enhancement model for performing the image enhancement method in the embodiment of the present application may implement end-to-end training, for example, for the image enhancement model shown in fig. 12, end-to-end training may be implemented by a sample enhancement image (e.g., a true value image) of the input image corresponding to the input image.

In the embodiment provided by the present application, the target model/rule 201 is obtained by training an image enhancement model. It should be noted that, in practical applications, the training data maintained in the database 230 may not necessarily all come from the collection of the data collection device 260, and may also be received from other devices.

It should be noted that, the training device 220 does not necessarily perform the training of the target model/rule 201 based on the training data maintained by the database 230, and may also obtain the training data from the cloud or other places for performing the model training, and the above description should not be taken as a limitation to the embodiments of the present application. It should also be noted that at least a portion of the training data maintained in the database 230 may also be used to execute the process of the device 210 on the to-be-processed process.

The target model/rule 201 obtained by training according to the training device 220 may be applied to different systems or devices, for example, the execution device 210 shown in fig. 6, where the execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an AR/VR, a vehicle-mounted terminal, and may also be a server or a cloud.

In fig. 6, the execution device 210 configures an input/output (I/O) interface 212 for data interaction with an external device, and a user may input data to the I/O interface 212 through the client device 240, where the input data may include: the image to be processed is input by the client device.

The preprocessing module 213 and the preprocessing module 214 are configured to perform preprocessing according to input data (such as an image to be processed) received by the I/O interface 212, and in this embodiment of the application, the preprocessing module 213 and the preprocessing module 214 may not be provided (or only one of them may be provided), and the computing module 211 is directly used to process the input data.

In the process that the execution device 210 preprocesses the input data or in the process that the calculation module 211 of the execution device 210 executes the calculation or other related processes, the execution device 210 may call the data, the code, and the like in the data storage system 250 for corresponding processes, and may store the data, the instruction, and the like obtained by corresponding processes in the data storage system 250.

Finally, the I/O interface 212 provides the processing result, i.e., the image-enhanced image to be processed is obtained as described above, i.e., the obtained output image is returned to the client device 240, thereby being provided to the user.

It should be noted that the training device 220 may generate corresponding target models/rules 201 for different targets or different tasks based on different training data, and the corresponding target models/rules 201 may be used to achieve the targets or complete the tasks, so as to provide the user with the required results.

In the case shown in fig. 6, in one case, the user may manually give the input data, which may be operated through an interface provided by the I/O interface 212.

Alternatively, the client device 240 may automatically send the input data to the I/O interface 212, and if the client device 240 is required to automatically send the input data to obtain authorization from the user, the user may set the corresponding permissions in the client device 240. The user can view the result output by the execution device 210 at the client device 240, and the specific presentation form can be display, sound, action, and the like. The client device 240 may also serve as a data collection terminal, collecting input data of the input I/O interface 212 and output results of the output I/O interface 212 as new sample data, and storing the new sample data in the database 230. Of course, the input data input to the I/O interface 212 and the output result output from the I/O interface 212 as shown in the figure may be directly stored in the database 230 as new sample data by the I/O interface 212 without being collected by the client device 240.

It should be noted that fig. 6 is only a schematic diagram of a system architecture provided in the embodiment of the present application, and the position relationship between the devices, modules, and the like shown in the diagram does not constitute any limitation, for example, in fig. 6, the data storage system 250 is an external memory with respect to the execution device 210, and in other cases, the data storage system 250 may also be disposed in the execution device 210.

As shown in fig. 6, a target model/rule 201 is obtained by training according to a training device 220, where the target model/rule 201 may be an image enhancement model in this embodiment, specifically, the image enhancement model provided in this embodiment may be a deep neural network, a convolutional neural network, or may be a deep convolutional neural network.

The structure of the convolutional neural network will be described in detail below with emphasis on fig. 7. As described in the introduction of the basic concept above, the convolutional neural network is a deep neural network with a convolutional structure, and is a deep learning (deep learning) architecture, where the deep learning architecture refers to performing multiple levels of learning at different abstraction levels through a machine learning algorithm. As a deep learning architecture, a convolutional neural network is a feed-forward artificial neural network in which individual neurons can respond to an image input thereto.

The structure of the image enhancement model in the embodiment of the present application may be as shown in fig. 7. In fig. 7, convolutional neural network 300 may include an input layer 310, a convolutional/pooling layer 320 (where pooling layers are optional), a fully-connected layer 330, and an output layer 340. The input layer 310 may obtain an image to be processed, and deliver the obtained image to be processed to the convolutional layer/pooling layer 320 and the fully-connected layer 330 for processing, so as to obtain a processing result of the image. The following describes the internal layer structure in CNN 300 in fig. 7 in detail.

Convolutional layer/pooling layer 320:

the convolutional layer/pooling layer 320 as shown in fig. 7 may include layers such as examples 321 and 326, for example: in one implementation, 321 layers are convolutional layers, 322 layers are pooling layers, 323 layers are convolutional layers, 324 layers are pooling layers, 325 layers are convolutional layers, 326 layers are pooling layers; in another implementation, 321, 322 are convolutional layers, 323 is a pooling layer, 324, 325 are convolutional layers, and 326 is a pooling layer, i.e., the output of a convolutional layer can be used as the input of a subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.

The inner working principle of one convolution layer will be described below by taking convolution layer 321 as an example.

Convolution layer 321 may include a plurality of convolution operators, also called kernels, whose role in image processing is equivalent to a filter for extracting specific information from the input image matrix, and the convolution operator may be essentially a weight matrix, which is usually predefined, and during the convolution operation on the image, the weight matrix is usually processed on the input image pixel by pixel (or two pixels by two pixels, etc., depending on the value of step size stride) in the horizontal direction, so as to complete the task of extracting specific features from the image. The size of the weight matrix should be related to the size of the image, and it should be noted that the depth dimension (depth dimension) of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Thus, convolving with a single weight matrix will produce a single depth dimension of the convolved output, but in most cases not a single weight matrix is used, but a plurality of weight matrices of the same size (row by column), i.e. a plurality of matrices of the same type, are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image, where the dimension is understood to be determined by "plurality" as described above.

Different weight matrices may be used to extract different features in the image, e.g., one weight matrix to extract image edge information, another weight matrix to extract a particular color of the image, yet another weight matrix to blur unwanted noise in the image, etc. The plurality of weight matrices have the same size (row × column), the sizes of the convolution feature maps extracted by the plurality of weight matrices having the same size are also the same, and the extracted plurality of convolution feature maps having the same size are combined to form the output of the convolution operation.

The weight values in these weight matrices need to be obtained through a large amount of training in practical application, and each weight matrix formed by the trained weight values can be used to extract information from the input image, so that the convolutional neural network 300 can make correct prediction.

When convolutional neural network 300 has multiple convolutional layers, the initial convolutional layer (e.g., 321) tends to extract more general features, which may also be referred to as low-level features; as the depth of convolutional neural network 300 increases, the more convolutional layers (e.g., 326) later extract more complex features, such as features with high levels of semantics, the more highly semantic features are suitable for the problem to be solved.

A pooling layer:

since it is often desirable to reduce the number of training parameters, it is often desirable to periodically introduce pooling layers after the convolutional layer, where the layers 321-326 as illustrated by 320 in fig. 7 may be one convolutional layer followed by one pooling layer, or multiple convolutional layers followed by one or more pooling layers. The purpose of the pooling layer is to reduce the spatial size of the image during image processing. The pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to smaller sized images. The average pooling operator may calculate pixel values in the image over a certain range to produce an average as a result of the average pooling. The max pooling operator may take the pixel with the largest value in a particular range as the result of the max pooling.

In addition, just as the size of the weighting matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after the processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel point in the image output by the pooling layer represents an average value or a maximum value of a corresponding sub-region of the image input to the pooling layer.

Full connection layer 330:

after processing by convolutional layer/pooling layer 320, convolutional neural network 300 is not sufficient to output the required output information. Because, as previously described, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, to generate the final output information (required class information or other relevant information), the convolutional neural network 300 needs to generate one or a set of the required number of classes of output using the fully-connected layer 330. Therefore, a plurality of hidden layers (331, 332 to 33n shown in fig. 7) and an output layer 340 may be included in the fully-connected layer 330, and parameters included in the hidden layers may be obtained by pre-training according to related training data of a specific task type, for example, the task type may include image enhancement, image recognition, image classification, image detection, image super-resolution reconstruction, and the like.

After the hidden layers in the fully-connected layer 330, i.e., the last layer of the whole convolutional neural network 300 is the output layer 340, the output layer 340 has a loss function similar to the classification cross entropy, and is specifically used for calculating the prediction error, once the forward propagation (i.e., the propagation from the direction 310 to the direction 340 in fig. 7 is the forward propagation) of the whole convolutional neural network 300 is completed, the backward propagation (i.e., the propagation from the direction 340 to the direction 310 in fig. 7 is the backward propagation) starts to update the weight values and the bias of the aforementioned layers, so as to reduce the loss of the convolutional neural network 300, and the error between the result output by the convolutional neural network 300 through the output layer and the ideal result.

It should be noted that the convolutional neural network shown in fig. 7 is only one example of the structure of the image enhancement model in the embodiment of the present application, and in a specific application, the convolutional neural network adopted in the image enhancement method in the embodiment of the present application may also exist in the form of other network models.

In an embodiment of the present application, the image enhancement apparatus may include the convolutional neural network 300 shown in fig. 7, and the image enhancement apparatus may perform image enhancement processing on the image to be processed, so as to obtain a processed output image.

Fig. 8 is a hardware structure of a chip provided in an embodiment of the present application, where the chip includes a neural-Network Processing Unit (NPU) 400. The chip may be provided in the execution device 210 shown in fig. 6 to complete the calculation work of the calculation module 211. The chip may also be disposed in the training device 220 as shown in fig. 6 to complete the training work of the training device 220 and output the target model/rule 201. The algorithm for each layer in the convolutional neural network shown in fig. 7 can be implemented in a chip as shown in fig. 8.

The NPU 400 is mounted as a coprocessor on a main processing unit (CPU), and tasks are allocated by the main CPU. The core portion of the NPU 400 is an arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to extract data in a memory (weight memory or input memory) and perform arithmetic.

In some implementations, the arithmetic circuit 403 includes a plurality of processing units (PEs) therein. In some implementations, the operational circuitry 403 is a two-dimensional systolic array. The arithmetic circuit 403 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 403 is a general-purpose matrix processor.

For example, assume that there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 403 fetches the data corresponding to the matrix B from the weight memory 402 and buffers it in each PE in the arithmetic circuit 403. The arithmetic circuit 403 takes the matrix a data from the input memory 401 and performs matrix operation with the matrix B, and partial or final results of the obtained matrix are stored in an accumulator 408 (accumulator).

The vector calculation unit 407 may further process the output of the operation circuit 403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, and the like. For example, the vector calculation unit 407 may be used for network calculations of non-convolution/non-FC layers in a neural network, such as pooling (pooling), batch normalization (batch normalization), local response normalization (local response normalization), and the like.

In some implementations, the vector calculation unit 407 can store the processed output vector to the unified memory 406. For example, the vector calculation unit 407 may apply a non-linear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 407 generates normalized values, combined values, or both.

In some implementations, the vector of processed outputs can be used as activation inputs to the arithmetic circuitry 403, for example for use in subsequent layers in a neural network.

The unified memory 406 is used to store input data as well as output data. The weight data directly passes through a memory unit access controller 405 (DMAC) to store the input data in the external memory into the input memory 401 and/or the unified memory 406, store the weight data in the external memory into the weight memory 402, and store the data in the unified memory 406 into the external memory.

A bus interface unit 410 (BIU) for implementing interaction between the main CPU, the DMAC, and the instruction fetch memory 409 through a bus.

An instruction fetch buffer 409(instruction fetch buffer) connected to the controller 404 is used for storing instructions used by the controller 404. And the controller 404 is configured to call the instruction cached in the instruction fetch memory 409 to implement controlling of the working process of the operation accelerator.

Generally, the unified memory 406, the input memory 401, the weight memory 402, and the instruction fetch memory 409 are On-Chip (On-Chip) memories, and the external memory is a memory external to the NPU, and the external memory may be a double data rate synchronous dynamic random access memory (DDR SDRAM), a High Bandwidth Memory (HBM), or other readable and writable memories.

The operation of each layer in the convolutional neural network shown in fig. 7 may be performed by the operation circuit 403 or the vector calculation unit 407.

The execution device 210 in fig. 6 described above can execute the steps of the image enhancement method in the embodiment of the present application, and the CNN model shown in fig. 7 and the chip shown in fig. 8 can also be used to execute the steps of the image enhancement method in the embodiment of the present application.

Fig. 9 illustrates a system architecture 500 according to an embodiment of the present application. The system architecture includes a local device 520, a local device 530, and an execution device 510 and a data storage system 550, wherein the local device 520 and the local device 530 are connected to the execution device 510 through a communication network.

Illustratively, the execution device 510 may be implemented by one or more servers.

Alternatively, the execution device 510 may be used with other computing devices. For example: data storage, routers, load balancers, and the like. The enforcement devices 510 may be disposed on one physical site or distributed across multiple physical sites. The execution device 510 may use data in the data storage system 550 or call program code in the data storage system 550 to implement the image enhancement method of the embodiments of the present application.

It should be noted that the execution device 510 may also be referred to as a cloud device, and at this time, the execution device 510 may be deployed in the cloud.

Specifically, the execution device 510 may perform the following process: acquiring an image to be processed; performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, wherein the neural network comprises N convolutional layers, and N is a positive integer; and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.

In a possible implementation manner, the image enhancement method according to the embodiment of the present application may be an offline method executed in a cloud, for example, the image enhancement method according to the embodiment of the present application may be executed in the execution device 510.

In one possible implementation, the image enhancement method of the embodiment of the present application may be performed by the local device 520 or the local device 530.

In the embodiment of the application, the acquired image to be processed with poor image quality can be subjected to image enhancement, so that an output image with the performance of the image to be processed in the aspects of image details, image colors, image brightness and the like is improved.

For example, a user may operate respective user devices (e.g., local device 520 and local device 530) to interact with the execution device 510. Each local device may represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, and so forth.

The local devices of each user may interact with the enforcement device 510 via a communication network of any communication mechanism/standard, such as a wide area network, a local area network, a peer-to-peer connection, etc., or any combination thereof.

In one implementation, the

local devices

520 and 530 may obtain relevant parameters of the target neural network from the execution device 510, deploy the target neural network on the

local devices

520 and 530, and perform image enhancement processing and the like using the target neural network.

In another implementation, the execution device 510 may directly deploy a target neural network, and the execution device 510 performs image enhancement processing and the like on the image to be processed by acquiring the image to be processed from the local device 520 and the local device 530 according to the target neural network.

For example, the target neural network may be an image enhancement model in the embodiment of the present application.

At present, photographing imaging and video of an intelligent terminal are limited by hardware performance of an optical sensor of the intelligent terminal, the quality of the photographed pictures and video is still not high enough, and the problems of high noise, low resolving power, missing details, color cast and the like exist. Image (or picture) enhancement is the basis of various image processing applications, and computer vision often involves a problem of how to perform image enhancement on an acquired image. Currently existing image enhancement methods can be divided into two categories: the first type, Scaling Method, is to Scale (Scale) pixels or features of an input image or video by learning a mapping function of a conversion relationship from an original image or video to a target image or video; the second type is a Generative Method (generic Method), which extracts features from an input image or video through a Generative Adaptive Network (GAN), generates new elements, and reconstructs an output image or video. However, in the scaling method, if the details of the original input image are unclear, the image obtained by the scaling method after image enhancement cannot realize detail enhancement; pseudo-texture is introduced for the generative method when image enhancement is performed.

In view of this, an embodiment of the present application provides an image enhancement method, which performs feature enhancement on an image to be processed to obtain an image enhancement feature of the image to be processed, and further performs color enhancement and brightness enhancement on the image to be processed by using the image enhancement feature, so that the detail feature of the image to be processed can be enhanced when performing color enhancement and brightness enhancement, so that the performance of the image to be processed in detail, color, and brightness is enhanced, and the effect of the image enhancement processing is improved.

Fig. 10 shows a schematic flowchart of an image enhancement method provided in an embodiment of the present application, and the image enhancement method shown in fig. 10 may be executed by an image enhancement apparatus, which may specifically be the execution device 210 in fig. 6, or the execution device 510 or a local device in fig. 9. The method shown in fig. 10 includes steps 610 through 630, and the steps 610 through 630 are described in detail below.

And step 610, acquiring an image to be processed.

The image to be processed can be an original image with poor image quality; for example, the problem may be that the acquired image to be processed has a blurred image, a low image quality, a low color and brightness, or the like due to the influence of weather, distance, shooting environment, or the like.

For example, the to-be-processed image may be an image captured by the electronic device through a camera, or the to-be-processed image may be an image obtained from the inside of the electronic device (for example, an image stored in an album of the electronic device, or a picture obtained by the electronic device from a cloud). For example, the electronic device may be any one of the local device or the execution device shown in fig. 9.

And 620, performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, wherein the neural network can comprise N convolutional layers, and N is a positive integer.

The feature enhancement processing may be detail feature enhancement performed on the image to be processed.

It should be understood that the neural network described above may refer to the feature extraction portion in the image enhancement model shown in fig. 11; the image enhancement model may include a plurality of neural networks, the feature extraction part may be a first neural network, and the feature reconstruction part may include a second neural network. For example, the first neural network may be configured to perform feature enhancement processing on the image to be processed to obtain an enhanced image feature of the image to be processed; the second neural network can be used for performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image. Fig. 13 may be a first neural network, and fig. 14 may be a second neural network.

It should be noted that the feature enhancement processing may be to purposefully emphasize the overall or local characteristics of the image, to make the original unclear image clear or emphasize some interesting features, to enlarge the difference between different object features in the image, to suppress the uninteresting features, to improve the image quality, to enrich the information content, to enhance the image interpretation and recognition effect, to meet the needs of recognition and analysis.

And 630, performing color enhancement processing and brightness enhancement processing on the image to be processed according to the features of the enhanced image to obtain an output image.

The color enhancement processing can be used for improving the color distribution of the image to be processed and improving the color saturation of the image to be processed. The brightness enhancement processing may refer to adjusting brightness of an image to be processed. The definition and the quality of the image can be improved through the characteristic enhancement processing, the color enhancement processing and the intensity enhancement processing, so that the outline of an object in the image is clearer and the details are more obvious.

It should be noted that, in the embodiment of the present application, the output image may refer to an image obtained by performing image enhancement on an acquired image to be processed, where the image enhancement may be referred to as image quality enhancement, and specifically may refer to processing brightness, color, contrast, saturation, dynamic range, and/or the like of the image, so that one or more indexes of the image satisfy a preset condition.

For example, the feature enhancement processing may be performed on the image to be processed by using a laplacian enhancement algorithm, so as to obtain an enhanced image feature of the image to be processed.

It should be understood that the enhanced image feature of the image to be processed may refer to an enhanced image feature obtained by enhancing details, textures, or the like in the image to be processed.

In the embodiment of the application, the enhancement of the detail features in the image to be processed can be realized through the Laplace enhancement algorithm, wherein the Laplace enhancement algorithm does not introduce new textures when used for feature enhancement of the image to be processed, so that the problem that pseudo textures are introduced into an output image obtained after image enhancement processing is avoided, and the effect of image enhancement processing can be improved.

For example, the laplacian enhancement algorithm may refer to a conventional laplacian enhancement algorithm, and the enhanced image features are obtained by performing fusion processing on the original image features of the image to be processed and the high-frequency features of the image to be processed.

For example, the feature enhancement processing is performed on the image to be processed by the following equation:

E＝I+s_c·[I-h(I)]；

wherein, I can represent the original image characteristic of the image to be processed, E can represent the enhanced image characteristic of the image to be processed, h can represent the fuzzy kernel function, s_cA scaling factor of a constant value may be represented, and I-h (I) may represent obtaining high frequency characteristics of the image to be processed.

It should be noted that the high-frequency features of the image may refer to details, textures, and other information of the image; the low frequency features of the image may refer to contour information of the image.

Further, in order to improve the feature enhancement processing effect of the image to be processed, an improved laplacian enhancement algorithm is provided in the embodiment of the present application, and by using the image features in the previous convolution layer for enhancement of the subsequent image features, the image features of different convolution layers can be gradually and gradually enhanced.

Optionally, the laplacian enhancement algorithm proposed in this embodiment may be configured to perform feature enhancement processing on an input image feature of an ith convolutional layer according to a residual feature of the ith convolutional layer in the N convolutional layers, to obtain an enhanced image feature of the ith convolutional layer, where the residual feature may represent a difference between the input image feature of the ith convolutional layer and an image feature after convolution processing in the ith convolutional layer, the enhanced image feature of the ith convolutional layer is an input image feature of an i +1 th convolutional layer, the input image feature is obtained according to an image to be processed, and i is a positive integer.

For example, the enhanced image feature of the image to be processed may be an image feature output by an nth convolutional layer of the N convolutional layers, and the enhanced image feature of the image to be processed is obtained by the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

wherein, L (F)_N) Can represent the enhanced image feature of the Nth convolution layer, F_NMay represent the input image characteristics of the Nth convolution layer, and Φ may represent the convolution kernel of the Nth convolution layer, s_lMay represent a parameter obtained by learning, and N is a positive integer.

According to the Laplace enhancement algorithm provided by the embodiment of the application, a fixed scaling coefficient in the traditional Laplace enhancement algorithm is replaced by a learnable parameter; meanwhile, the residual characteristics of adjacent layers are adopted for characteristic enhancement, and the residual characteristics can be used for representing any information needing to be emphasized. Therefore, the laplacian algorithm of the embodiment of the present application can enhance not only the high-frequency information of the image, but also the image features of different convolution layers progressively, thereby improving the effect of enhancing the image features.

Further, in the embodiment of the application, the color enhancement and the brightness enhancement can be performed on the image to be processed through the obtained enhanced image features, and as the enhanced image features of the image to be processed include more detailed features and textures, the color enhancement and the brightness enhancement can be performed on the image to be processed according to the enhanced image features of the image to be processed, so that the detail enhancement of the output image can be realized, and meanwhile, the brightness and the color of the output image can also be improved.

For example, performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image, may include: obtaining a confidence image feature and an illumination compensation image feature of the image to be processed according to the enhanced image feature of the image to be processed, wherein the confidence image feature is used for performing color enhancement on the image to be processed, and the illumination compensation image feature is used for performing brightness enhancement on the image to be processed; and obtaining the output image according to the image to be processed, the confidence image characteristic and the illumination compensation image characteristic.

For example, the confidence image feature and the illumination compensation image feature may be obtained by performing a convolution operation on the enhanced image feature of the image to be processed; obtaining the color enhancement image characteristic of the image to be processed by multiplying the image characteristic of the image to be processed with the confidence image characteristic; and fusing the color enhanced image characteristics and the illumination compensation image characteristics to obtain the output image.

It should be noted that the confidence image feature and the illumination compensation image feature are obtained according to the enhanced image feature of the image to be processed, so that the confidence image feature also enhances the image feature of the image to be processed to a certain extent when the confidence image feature is used for performing color enhancement on the image to be processed; similarly, the illumination compensation image features can also enhance the image features of the image to be processed when the illumination compensation image features are used for enhancing the brightness of the image to be processed, so that the details, the color and the brightness of the image to be processed are enhanced.

In the embodiment of the application, the confidence image features for color enhancement processing and the image features for brightness enhancement processing are obtained through the enhanced image features of the image to be processed, and compared with a traditional scaling method that the colors and the brightness of all pixel points in the image to be processed are enhanced directly through a mapping function, the confidence image features and the illumination compensation image features in the embodiment of the application not only can enhance the colors and the brightness, but also can enhance the details of the image to be processed.

In one possible implementation, the output image may be derived from the image to be processed, the confidence image feature, and the illumination compensation image feature.

For example, the color enhancement of different regions in the image to be processed can be realized by multiplying the image features of the image to be processed (e.g., the original image features of the image to be processed) with the confidence image features, so as to obtain color enhanced image features; and then fusing the color enhanced image features and the illumination compensation image features to obtain an output image after image enhancement processing.

Exemplarily, the color enhanced image feature and the illumination compensation image feature are fused, which may mean that the color enhanced image feature and the illumination compensation image feature are added to obtain an output image.

In another possible implementation manner, the output image after the image enhancement processing can be obtained through the enhanced image feature, the confidence image feature and the illumination compensation feature of the image to be processed.

For example, the enhanced image features of the image to be processed may be multiplied by the confidence image features, so as to realize color enhancement of different areas in the image to be processed, and obtain color enhanced image features; and then fusing the color enhanced image features and the illumination compensation image features to obtain an output image after image enhancement processing.

Fig. 11 is a schematic structural diagram of a model for image enhancement according to an embodiment of the present application. The model shown in fig. 11 may be deployed in an image enhancement apparatus that performs the image enhancement method described above.

The model shown in fig. 11 may include four parts, namely an input part, a feature extraction part, a feature reconstruction part, and an output part. Wherein, the feature extraction part can comprise a Laplacian Enhancing Unit (LEU); the feature reconstruction part may include a Hybrid Enhancing Module (HEM).

It is to be understood that a plurality of neural networks may be included in the image enhancement model shown in fig. 11, the feature extraction part may be a first neural network, and the feature reconstruction part may include a second neural network. The first neural network can be used for performing feature enhancement processing on the image to be processed to obtain enhanced image features of the image to be processed; the second neural network can be used for performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image. For example, fig. 13 may show a first neural network, and fig. 14 may show a second neural network.

Wherein, in the feature extraction part, a Laplace enhancement unit can be embedded into the convolution layer, and the extracted features are enhanced by using the Laplace enhancement unit; specifically, the laplacian enhancement unit may be configured to perform feature enhancement processing on the image to be processed by using a laplacian enhancement algorithm.

For example, the extracted features are progressively enhanced by the laplacian enhancement units in several layers, the image features of the previous layer can be used for enhancing the image features of the next layer, and the image features of different convolution layers are progressively enhanced by overlapping the residual errors of the image features of the previous layer, so that the performance of image enhancement can be improved. For the feature reconstruction part, the advantages of both the scaled method and the generated method of image enhancement can be achieved using the hybrid enhancement unit.

The hybrid enhancement unit may use, as input data, the image features of the input image processed by the laplacian enhancement unit, that is, the hybrid enhancement unit uses, as input data, the enhanced image features of the output image output by the laplacian enhancement unit. The laplacian enhancement unit may be configured to perform feature enhancement processing on the output image through a laplacian algorithm, and the hybrid enhancement unit may be configured to perform color enhancement processing and brightness enhancement processing on the output image.

For example, the image enhancement model for performing the image enhancement method provided by the embodiment of the present application shown in fig. 11 may be referred to as a hybrid progressive enhancement U-net (HPEU), and by using LEU and HEM, the receptive field is increased layer by layer; the LEU can realize feature enhancement processing on the output image based on more image information.

In addition, the laplacian enhancement unit may be used to perform different degrees of enhancement processing on different levels of image features. For example, in a convolutional layer with a shallow image enhancement model, the LEU may be mainly used to perform local region feature enhancement processing according to edge information of an input image; in the convolution layer with a deeper image enhancement model, since the receptive field is larger, the LEU can be used for enhancement processing of the global features.

It should be noted that the above-mentioned receptive field is a term of the deep neural network field in the computer vision field, and is used to represent the size of the receptive range of the neurons at different positions inside the neural network to the original image. The larger the value of the neuron receptive field is, the larger the range of the original image which can be contacted with the neuron is, which also means that the neuron possibly contains more global and higher semantic level features; while smaller values indicate that the neuron contains more local and detailed features, and the value of the receptive field can be used approximately to determine the level of abstraction at each level.

Fig. 12 is a schematic diagram of a laplacian enhancement unit and a hybrid enhancement unit provided in an embodiment of the present application. As shown in fig. 12, the image enhancement apparatus may include one or more laplacian enhancement units and a hybrid enhancement unit, where features of an input image are extracted through a convolutional layer, and the extracted features are enhanced by the laplacian enhancement unit to obtain enhanced image features; further, the enhanced image features are used as input data of a subsequent convolution layer or a subsequent Laplace enhancement unit layer; until the enhanced image features are input to the hybrid enhancement unit, the feature channel is divided into two parts, the scaling component and the generation component are respectively calculated, and then the two components are fused to obtain a final enhanced image.

The zooming component can be used for realizing color enhancement, and the color enhancement degree of different areas of the input image can be restricted through a confidence map (also called confidence image characteristic); the above-described generation assembly may be used for illumination compensation, thereby achieving contrast and brightness enhancement.

For example, fig. 13 is a schematic diagram of a processing flow of the laplacian enhancement unit (i.e., a schematic diagram of the feature enhancement processing) provided in the embodiment of the present application. As shown in fig. 13, the laplacian enhancement unit processing procedure includes the following steps:

the method comprises the following steps: suppose that the current network (first neural network) includes N convolutional layers, and the output data of the Nth convolutional layer is the image feature F_NThat is, F can be substituted_NAs input data for the feature enhancement process.

Step two: extracting F with Nth convolution layer_NIs characterized by phi (F)_N)。

Step three: calculating phi (F)_N) And F_NThe residual between, the residual can be written as: phi (F)_N)-F_NOr, F_N-Φ(F_N) And the residual error is enhanced by learnable parameters.

Step four: the residual error after the enhancement processing and phi (F)_N) And superposing to obtain the enhanced image characteristics of the Nth convolution layer after the characteristic enhancement processing is carried out on the Nth convolution layer by the Laplace enhancement unit.

Illustratively, the image characteristics output by the nth convolutional layer are obtained by the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

wherein, L (F)_N) Can represent the enhanced image feature of the Nth convolution layer, F_NMay represent the input image characteristics of the Nth convolution layer, and Φ may represent the convolution kernel of the Nth convolution layer, s_lParameters obtained through learning can be expressed and can be used for expressing the enhancement degree of the image feature enhanced by the LEU each time.

It should be understood that the first neural network may include N convolutional layers, and the image feature output by the nth convolutional layer may be an enhanced image feature of the image to be processed, which is obtained by performing feature enhancement processing on the image to be processed.

E.g. s_lMay represent the learned scaling parameters that may perform scaling operations on different regions of the image to achieve color enhancement of the different regions of the image.

For example, fig. 14 is a schematic diagram of a processing flow of a hybrid enhancement unit (i.e., a schematic diagram of a color enhancement process and a brightness enhancement process) provided in the embodiment of the present application. As shown in fig. 14, the processing procedure of the hybrid enhancement unit includes the following steps:

the method comprises the following steps: the plurality of feature channels of the enhanced image features output by the laplacian enhancement unit (that is, the enhanced image features output by the first neural network) are divided into two branches, which are referred to as a first branch and a second branch.

Step two: and performing convolution operation through a first branch in the mixing enhancement unit to obtain a confidence map, and applying the confidence map to the input image to perform pixel-level scaling to obtain a scaling component.

The confidence map, i.e., the confidence image feature in the enhancement processing method shown in fig. 10, is used to perform color enhancement on the input image. For example, the confidence map may represent a mapping relationship, or a mapping function, for color enhancement processing of the input image. The scaling component, i.e., the color enhancement image feature in the enhancement processing method shown in fig. 10.

For example, the confidence map may correspond to the input image, for example, one element in the image feature of the confidence map may be used to indicate the degree of scaling of the corresponding element in the image feature of the input image; the color enhancement of the image to be input can be achieved by scaling different regions in the input image.

Wherein the confidence map may comprise a number of channels corresponding to the channels of the input image.

Applying the confidence map to the input image may illustratively refer to multiplying the confidence map with the input image, e.g., pixel-by-pixel multiplying the input image with the confidence map.

Step three: the generation component for illumination compensation is obtained by performing a convolution operation through the second branch in the hybrid enhancement unit.

The generation component is used for performing brightness enhancement on the input image; the generation component is the illumination compensation image feature in the enhancement processing method shown in fig. 10 described above.

Step four: and performing fusion processing on the image characteristics of the two branches to obtain an output image after image enhancement processing.

For example, if the scaling component is s [ i, j ] ═ x [ i, j ] · c [ i, j ], where x denotes the input image, c denotes the confidence map, and i and j denote the pixel coordinates, the resulting output image is y [ i, j ] ═ x [ i, j ] · c [ i, j ] + g [ i, j ].

In one possible implementation, assuming that the enhanced image features of the input image output by the laplacian enhancement unit have N channels, the N channels may be divided into two branches, where M channel features are convolved to obtain a confidence map, the confidence map may be used to calculate a scaling component, and the M channels may respectively correspond to R, G, B three channels of the image; the other N-M channels may be convolved to generate components for illumination compensation to achieve contrast and brightness enhancement.

It will be appreciated that since the generation component is used for illumination compensation, the illumination compensated image features used for illumination enhancement processing can be obtained by performing a convolution operation with one channel feature.

For example, in an embodiment of the present application, a confidence image feature and an illumination compensation image feature of an input image may be obtained according to a hybrid enhancement module and an enhanced image feature of the input image, where the hybrid enhancement module includes a first branch and a second branch, the first branch is used for obtaining the confidence image feature according to the enhanced image feature, the second branch is used for obtaining the illumination compensation image feature according to the enhanced image feature, the confidence image feature may be used for representing a mapping relationship (e.g., a mapping function) for performing color enhancement on an image to be processed, and the illumination compensation image feature may be used for performing brightness enhancement on the input image; and obtaining an output image according to the input image, the confidence image characteristic and the illumination compensation image characteristic.

For example, fig. 15 is a schematic diagram of a processing flow of a hybrid enhancement unit provided in an embodiment of the present application. The scale sizes of the image to be processed and the confidence map can be the same, and different areas of the input image can be zoomed in different degrees through the confidence map, so that color enhancement is realized; and then the superposition generation component is used for realizing illumination compensation, realizing the enhancement of contrast and brightness, and finally obtaining the output image after the characteristic enhancement, the color enhancement and the brightness enhancement of the input image.

It should be noted that the confidence map may represent a mapping relationship or a mapping function for performing color enhancement processing on the image to be processed.

For example, the confidence map and the image to be processed may correspond to each other, for example, one element in the image feature of the confidence map may be used to indicate the scaling degree of the corresponding element in the image feature of the image to be processed; color enhancement of the image to be processed can be achieved by scaling different regions in the image to be processed.

In the embodiment of the application, the confidence map for color enhancement processing and the illumination compensation image feature for brightness enhancement processing may be obtained through an enhanced image feature of an image to be processed, and compared with a conventional scaling method in which the color and the brightness of each pixel point in the image to be processed are directly enhanced through a mapping function, the confidence map and the illumination compensation image feature in the embodiment of the application not only can enhance the color and enhance the brightness, but also can enhance the details of the image to be processed.

TABLE 1

Model (model)	PSNR	MS-SSIM	Time(ms)	Parameters
					Baseline	22.67	0.9355	371	3758879
+LEU	22.94	0.9352	383	3758903
					+HEM	22.79	0.9405	390	3758828
This application	23.29	0.9431	408	3758852

Table 1 shows the quantitative performance evaluation results on MIT-Adobe FiveK data set provided in the examples of the present application. Wherein, Baseline represents a basic model architecture, that is, it may be a Unet model excluding the hybrid enhancement unit (HEM) and the Laplacian Enhancement Unit (LEU); + LEU denotes an image enhancement uet model including the laplacian enhancement unit; + HEM denotes an image enhancement uet model including the above hybrid enhancement unit; peak signal-to-noise ratio (PSNR) is generally used as a method for measuring signal reconstruction quality in the field of image processing and the like, and can be defined by mean square error. Multi-scale structural similarity (MS-SSIM) can be used as an index for measuring the similarity between two images, and is used for evaluating the quality of an output image processed by an algorithm. The structural similarity index defines structural information from the perspective of image composition as attributes reflecting the structure of objects in a scene independent of brightness, contrast, and models distortion as a combination of three different factors of brightness, contrast, and structure. For example, the mean value is used as the luminance estimate, the standard deviation is used as the contrast estimate, and the covariance is used as the measure of the structural similarity. Time is used to represent the time at which the model performs image enhancement on the input image. Parameters (parameters) can be used to describe the number of parameters that the neural network contains for evaluating the size of the model.

As can be seen from the performance evaluation results shown in table 1, the image enhancement model (HPEU) provided in the embodiment of the present application improves PSNR by 0.62dB, and in addition, excessive parameter amount is not increased, and processing delay is increased by 37 ms; namely, the image enhancement method provided by the embodiment of the application can effectively improve the image enhancement effect.

TABLE 2

Table 2 shows the quantitative performance evaluation results of different image enhancement models on the data set provided in the embodiments of the present application. Wherein, the data set 1 can represent an MIT-Adobe FiveK data set; data set 2 may represent the DPED-iPhone data set; the model tested included: digital camera Photo Enhancer (weather super Photo Enhancer for Digital camera, WESPE), Context Aggregation Network (CAN), and global U-network (rsgun) for regional scaling.

As can be seen from the performance evaluation results shown in table 2, the PSNR and SSIM of the image enhancement model proposed in the present application are inferior to the WESPE model in the DPED-iPhone dataset, mainly due to the non-pixel level alignment of the DPED-iPhone dataset, and for the DPED-iPhone dataset, a single-lens reflex camera and a mobile phone are adopted in the same scene, and an obtained image pair is simultaneously captured at the same angle, and since the sensors of the single-lens reflex camera and the mobile phone are different, the image captured by the single-lens reflex camera and the image captured by the camera are not aligned pixel by pixel, so that the HPEU generates a relative deviation in the downsampling process; the image enhancement model (HPEU) proposed by the embodiment of the application has the highest PSNR and SSIM on the MIT-Adobe FiveK data set, and the visual effect is closer to the true value (Ground Truth).

TABLE 3

Model (model)	PSNR	MS-SSIM
			GDF	22.64	0.9353
HDR	22.27	0.9391
			HPEU + guided filtering	22.59	0.9383

At present, there is a method for performing image enhancement fast processing on a small-resolution picture, for example, an original image is used as a guide (guide) to perform upsampling on an output image to obtain an enhanced image.

Table 3 shows the quantitative performance evaluation result of the MIT-Adobe five dataset by adding Guided Filter (Guided Filter) to the model according to the embodiment of the present application. The model tested included: guiding filtering is added to a trainable guided filter (GDF) model, a high dynamic range network (HDR) model, and an image enhancement model (HPEU) provided by an embodiment of the present application.

As can be seen from the performance evaluation results shown in table 3, the performance of the network after the guiding filtering is added to the HPEU model proposed in the embodiment of the present application is close to that of the high dynamic range network model.

TABLE 4

Table 4 shows evaluation results of the run time of the image enhancement process provided in the embodiment of the present application. As can be seen from the run-time evaluation results shown in table 4, the image enhancement model (HPEU) proposed in the embodiment of the present application has the shortest run-time and the calculation efficiency of HPEU + Guided Filter is higher under the same processing conditions. After the Guided Filter is added in the HPEU model, the speed of the HPEU + Guided Filter is faster than that of the GDF model and the HDR model, and objective quantitative indexes are close to each other.

TABLE 5

Model (model)	PSNR	MS-SSIM	Perceptual Index
				Input	17.08	0.8372	4.10
WESPE	22.57	0.9186	3.67
				RSGUnet	22.83	0.9254	3.77
This application	23.02	0.9255	3.50

Table 5 is a quantitative assessment of the DPED-iphone data set based on the results of the quantitative assessment after the introduction of visual perception loss provided in the examples of the present application. The evaluation indexes include objective evaluation indexes PSNR, SSIM and perception evaluation indexes (Perceptual Index).

As can be seen from the performance evaluation results shown in table 5, the image enhancement model (HPEU) proposed in the embodiment of the present application performs well in both detail enhancement and noise suppression.

TABLE 6

Table 6 shows that each network model is trained based on the MIT-Adobe five dataset provided in this embodiment, and the dataset a (DPED-iphone), the dataset B (DPED-Sony), and the dataset C (DPED-Blackberry) are used as test datasets for verifying the generalization capability of the model.

As can be seen from the performance evaluation results shown in table 6, the PSNR of the image enhancement model (HPEU) provided in the example of the present application is relatively best compared with that of the other comparative models, and the generalization ability is relatively best.

Fig. 16 is a schematic diagram of a visual quality assessment result provided by an embodiment of the present application. In fig. 16, a color image portion is indicated by diagonal filling in order to be distinguished from a grayscale image portion.

Wherein (a) of fig. 16 represents an input image (e.g., an image to be processed); fig. 16 (b) shows a prediction output image obtained by using a base model, i.e., a Unet model that does not include the mixture enhancement unit (HEM) and the Laplacian Enhancement Unit (LEU); fig. 16 (c) shows a prediction output image obtained by using the Unet model of the laplacian enhancement unit; fig. 16 (d) shows a predicted output image obtained by using the Unet model of the hybrid enhancement unit; fig. 16 (e) shows a prediction output image obtained by using a model according to an embodiment of the present application (for example, the model shown in fig. 11 or 12); fig. 16 (f) shows a true value image (Ground Truth) corresponding to the input image, where the true value image may represent a sample enhanced image corresponding to the input image; fig. 16 (e) shows an error map 1, i.e., a residual between a predicted output image output from the base model and a true value image; fig. 16 (h) shows an error map 2, which is a residual between a predicted output image and a true value image output by using the Unet model of the laplacian enhancement unit; fig. 16 (i) shows an error map 3, which is a residual between a predicted output image and a true value image output by the Unet model using the hybrid enhancement unit described above; fig. 16 (j) shows an error map 4, which is a residual between a predicted output image and a true value image output by using the model according to the embodiment of the present application.

As can be seen from the schematic diagram of the visual quality evaluation result shown in fig. 16, the output image obtained by using the laplacian enhancement unit and the hybrid enhancement unit, i.e., the image enhancement model provided in the embodiment of the present application, is visually closest to the true value image.

Fig. 17 is a schematic flowchart of an image enhancement method provided in an embodiment of the present application. The method 700 shown in fig. 17 includes steps 710 to 740, and the steps 710 to 740 are described in detail below.

Step 710, a first operation of a user to turn on a camera is detected.

Step 720, responding to the first operation, displaying a shooting interface on the display screen, and displaying the shooting interface on the display screen, wherein the shooting interface comprises a view frame, and the view frame comprises a first image.

In one example, the photographing behavior of the user may include a first operation of the user to turn on the camera; and responding to the first operation, and displaying a shooting interface on a display screen.

Fig. 18 (a) shows a Graphical User Interface (GUI) of a mobile phone, which is a desktop 810 of the mobile phone. When the electronic device detects an operation of a user clicking an icon 820 of a camera Application (APP) on the desktop 810, the camera application may be started, and another GUI, which may be referred to as a photographing interface 830, may be displayed as shown in (b) of fig. 18. A viewfinder 840 may be included on the capture interface 830. In the preview state, a preview image can be displayed in real time in the finder frame 840.

Note that, in the embodiment of the present application, in order to distinguish from a grayscale image portion, a color image portion is indicated by diagonal filling.

For example, referring to (b) in fig. 18, after the electronic device starts the camera, a first image may be displayed in the view frame 840, where the first image is a color image. Controls 850 for indicating a photographing mode, as well as other photographing controls, may also be included on the photographing interface.

In one example, the photographing behavior of the user may include a first operation of the user to turn on the camera; and responding to the first operation, and displaying a shooting interface on a display screen. For example, after detecting a first operation of a user clicking an icon of a camera Application (APP) on a desktop, the electronic device may start the camera application and display a shooting interface. A viewfinder frame may be included on the camera interface, and it is understood that the size of the viewfinder frame may be different in the camera mode and the video mode. For example, the finder frame may be a finder frame in a photographing mode. In the video mode, the viewing frame may be the entire display screen. In the preview state, i.e. before the user turns on the camera and does not press the photo/video button, the preview image can be displayed in real time in the view finder.

In one example, the preview image may be a color image, and the preview image may be an image displayed in a case where the camera is set to the automatic photographing mode.

Step 730, detecting a second operation of the user indicated camera.

In one possible implementation, a second operation of the user indicating the first processing mode may be detected. Wherein the first processing mode may be a professional shooting mode (e.g., an image-enhanced shooting mode). Referring to (a) of fig. 19, a shooting option 860 is included on the shooting interface, and after the electronic device detects that the user clicks the shooting option 860, referring to (b) of fig. 19, the electronic device displays a shooting mode interface. After the electronic device detects that the user clicks on the shooting mode interface to indicate the professional shooting mode 861, the mobile phone enters the professional shooting mode, for example, the mobile phone performs an image enhancement shooting mode.

In one possible implementation, a second operation of the user for instructing shooting may be detected, where the second operation is an operation for shooting a distant object, or shooting a tiny object, or instructing shooting in a case where the shooting environment is poor.

Exemplarily, referring to (c) in fig. 19, a second operation 870 for instructing photographing by the user is detected.

In another possible implementation, it may be that the second operation for instructing photographing is detected, that is, the second operation 870 for instructing photographing shown in (c) in fig. 19 may be directly performed without performing the operations shown in (a) in fig. 19 and (b) in fig. 19.

It should be understood that the second operation of the user for instructing the shooting behavior may include pressing a shooting button in a camera of the electronic device, or may include the user device instructing the electronic device to perform the shooting behavior through voice, or may also include the user instructing the electronic device to perform the shooting behavior. The foregoing is illustrative and not limiting of the present application.

Step 740, responding to the second operation, displaying a second image in the view finder, or saving the second image in the electronic device, where the second image is obtained by performing color enhancement processing and brightness enhancement processing on the first image according to the enhanced image feature of the first image acquired by the camera, the enhanced image feature of the first image is obtained by performing feature enhancement processing on the first image through a neural network, the neural network includes N convolutional layers, and N is a positive integer.

It should be understood that the flow of the image enhancement method for the first image may refer to the image enhancement method shown in fig. 10, and the image enhancement model for performing the image enhancement method may adopt the model shown in fig. 11.

In one possible implementation, referring to fig. 19, the second image is displayed in the frame in (d) of fig. 19, the first image is displayed in the frame in (c) of fig. 19, and the second image and the first image have the same or substantially the same content, but the second image has better image quality than the first image. For example, the second image shows better detail than the first image; alternatively, the second image has a luminance superior to the first image; alternatively, the second image may have a luminance superior to that of the first image.

In another possible implementation manner, the second image shown in (d) of fig. 19 may not be displayed in the view box, but the second image may be saved in an album of the electronic device.

Optionally, in a possible implementation manner, the enhanced image feature of the first image is obtained by performing the feature enhancement processing on the first image according to a laplacian enhancement algorithm.

Optionally, in a possible implementation manner, the laplacian enhancement algorithm is configured to perform the feature enhancement processing on the input image feature of the ith convolutional layer according to a residual feature of an ith convolutional layer of the N convolutional layers to obtain an enhanced image feature of the ith convolutional layer, where the residual feature represents a difference between the input image feature of the ith convolutional layer and an image feature after convolution processing in the ith convolutional layer, the enhanced image feature of the ith convolutional layer is an input image feature of the (i + 1) th convolutional layer, the input image feature is obtained according to the first image, and i is a positive integer.

Optionally, in a possible implementation manner, the enhanced image feature of the first image is an image feature output by an nth convolutional layer of the N convolutional layers, and the enhanced image feature of the first image is obtained by the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

wherein, L (F)_N) Features of the enhanced image representing said Nth convolution layer, F_NRepresenting input image characteristics of said Nth convolution layer, [ phi ] representing convolution kernel of said Nth convolution layer, [ s ]_lThe zoom parameters obtained by learning are indicated.

Optionally, in a possible implementation, the output image is obtained according to a first image, a confidence image feature and an illumination compensation image feature, the confidence image feature and the illumination compensation image feature are obtained according to an enhanced image feature of the first image, the confidence image feature is used for performing color enhancement on the first image, and the illumination compensation image feature is used for performing brightness enhancement on the first image.

Optionally, in a possible implementation manner, the output image is obtained by fusing a color enhanced image feature and the illumination compensation image feature, the color enhanced image feature is obtained by multiplying an image feature of the first image by the confidence image feature, the confidence image feature is obtained by performing a convolution operation on the enhanced image feature of the first image, and the illumination compensation image feature is obtained by performing a convolution operation on the enhanced image feature of the first image.

It is to be understood that the above description is intended to assist those skilled in the art in understanding the embodiments of the present application and is not intended to limit the embodiments of the present application to the particular values or particular scenarios illustrated. It will be apparent to those skilled in the art from the foregoing description that various equivalent modifications or changes may be made, and such modifications or changes are intended to fall within the scope of the embodiments of the present application.

The image enhancement method provided by the embodiment of the present application is described in detail above with reference to fig. 1 to 19, and the device embodiment of the present application is described in detail below with reference to fig. 20 and 21. It should be understood that the image enhancement device in the embodiment of the present application may perform the various image enhancement methods of the embodiments of the present application, that is, the following specific working processes of various products, and reference may be made to the corresponding processes in the embodiments of the foregoing methods.

Fig. 20 is a schematic block diagram of an image enhancement apparatus provided in an embodiment of the present application. It is understood that the image enhancement apparatus 900 may perform the image enhancement method shown in fig. 10. The image enhancement apparatus 900 includes: an acquisition unit 910 and a processing unit 920.

The acquiring unit 910 is configured to acquire an image to be processed; the processing unit 920 is configured to perform feature enhancement processing on the image to be processed through a neural network to obtain an enhanced image feature of the image to be processed, where the neural network includes N convolutional layers, and N is a positive integer; and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.

Optionally, as an embodiment, the processing unit 920 is specifically configured to:

and performing the feature enhancement processing on the image to be processed by using a Laplace enhancement algorithm to obtain the enhanced image features of the image to be processed.

Optionally, as an embodiment, the laplacian enhancement algorithm is configured to perform the feature enhancement processing on the input image feature of the i-th convolutional layer according to a residual feature of the i-th convolutional layer of the N convolutional layers to obtain an enhanced image feature of the i-th convolutional layer, where the residual feature represents a difference between the input image feature of the i-th convolutional layer and an image feature of the i-th convolutional layer after being processed by a convolution operation, the enhanced image feature of the i-th convolutional layer is an input image feature of the i + 1-th convolutional layer, the input image feature is obtained according to the image to be processed, and i is a positive integer smaller than or equal to N.

Optionally, as an embodiment, the enhanced image feature of the image to be processed is an image feature output by an nth convolutional layer of the N convolutional layers, and the processing unit 920 is specifically configured to: obtaining the enhanced image characteristics of the image to be processed by the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

wherein, L (F)_N) Features of the enhanced image representing said Nth convolution layer, F_NRepresenting input image characteristics of said Nth convolution layer, [ phi ] representing convolution kernel of said Nth convolution layer, [ s ]_lThe parameters obtained by learning are shown.

obtaining a confidence image feature and an illumination compensation image feature of the image to be processed according to the enhanced image feature of the image to be processed, wherein the confidence image feature is used for performing color enhancement on the image to be processed, and the illumination compensation image feature is used for performing brightness enhancement on the image to be processed;

and obtaining the output image according to the image to be processed, the confidence image characteristic and the illumination compensation image characteristic.

Optionally, as an embodiment, the processing unit 920 is further configured to:

performing convolution operation on the enhanced image features of the image to be processed to obtain the confidence image features and the illumination compensation image features;

obtaining the color enhancement image characteristic of the image to be processed by multiplying the image characteristic of the image to be processed with the confidence image characteristic;

and fusing the color enhanced image characteristics and the illumination compensation image characteristics to obtain the output image.

In one embodiment, the enhancement apparatus 900 shown in fig. 20 can also be used to perform the image enhancement methods shown in fig. 17-19.

The image enhancement apparatus 900 is embodied as a functional unit. The term "unit" herein may be implemented in software and/or hardware, and is not particularly limited thereto.

For example, a "unit" may be a software program, a hardware circuit, or a combination of both that implement the above-described functions. The hardware circuitry may include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group of processors) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that support the described functionality.

Accordingly, the units of the respective examples described in the embodiments of the present application can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Fig. 21 is a schematic hardware structure diagram of an image enhancement apparatus according to an embodiment of the present application. An image enhancement apparatus 1000 (the apparatus 1000 may be specifically a computer device) as shown in fig. 21 includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, the processor 1002, and the communication interface 1003 are communicatively connected to each other via a bus 1004.

The memory 1001 may be a Read Only Memory (ROM), a static memory device, a dynamic memory device, or a Random Access Memory (RAM). The memory 1001 may store a program, and when the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 is configured to perform the steps of the image enhancement method according to the embodiment of the present application, for example, perform the steps illustrated in fig. 10 to 15, or perform the steps illustrated in fig. 17 to 19.

It should be understood that the image enhancement device shown in the embodiment of the present application may be a server, for example, a server in a cloud, or may also be a chip configured in the server in the cloud; alternatively, the image enhancement device shown in the embodiment of the present application may be an intelligent terminal, or may be a chip configured in the intelligent terminal.

The image enhancement method disclosed in the embodiment of the present application can be applied to the processor 1002 or implemented by the processor 1002. The processor 1002 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the image enhancement method described above may be performed by instructions in the form of hardware integrated logic circuits or software in the processor 1002. For example, the processor 1002 may be a chip containing the NPU shown in fig. 8.

The processor 1002 may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a Random Access Memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM, an electrically erasable programmable memory, a register, or other storage media that are well known in the art. The storage medium is located in the memory 1001, and the processor 1002 reads the instructions in the memory 1001, and completes, in combination with hardware thereof, functions required to be executed by units included in the image enhancement apparatus shown in fig. 20 in the embodiment of the present application, or executes the image enhancement methods shown in fig. 10 to 15 in the embodiment of the method of the present application, or executes the steps shown in fig. 17 to 19 in the embodiment of the method of the present application.

The communication interface 1003 enables communication between the apparatus 1000 and other devices or communication networks using transceiver means such as, but not limited to, a transceiver.

Bus 1004 may include a pathway to transfer information between various components of image enhancement device 1000 (e.g., memory 1001, processor 1002, communication interface 1003).

It should be noted that although the image enhancement apparatus 1000 described above only shows a memory, a processor, and a communication interface, in a specific implementation process, a person skilled in the art should understand that the image enhancement apparatus 1000 may also include other devices necessary for normal operation. Meanwhile, it will be understood by those skilled in the art that the image enhancement apparatus 1000 may further include hardware components for implementing other additional functions according to specific needs. Furthermore, it should be understood by those skilled in the art that the image enhancement apparatus 1000 may also include only the components necessary to implement the embodiments of the present application, and not necessarily all of the components shown in fig. 21.

The embodiment of the application also provides a chip, which comprises a transceiver unit and a processing unit. The transceiver unit can be an input/output circuit and a communication interface; the processing unit is a processor or a microprocessor or an integrated circuit integrated on the chip. The chip can execute the image enhancement method in the method embodiment.

Embodiments of the present application further provide a computer-readable storage medium, on which instructions are stored, and when executed, the instructions perform the image enhancement method in the above method embodiments.

Embodiments of the present application further provide a computer program product containing instructions, which when executed, perform the image enhancement method in the foregoing method embodiments.

It will also be appreciated that in embodiments of the present application, the memory may comprise both read-only memory and random access memory, and may provide instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store information of the device type.

It should be understood that the term "and/or" herein is merely one type of association relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional modules in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image enhancement method, comprising:

acquiring an image to be processed;

performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, wherein the neural network comprises N convolutional layers, and N is a positive integer;

and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.

2. The image enhancement method of claim 1, wherein the performing the feature enhancement processing on the image to be processed through the neural network to obtain the enhanced image features of the image to be processed comprises:

3. The image enhancement method according to claim 2, wherein the laplacian enhancement algorithm is configured to perform the feature enhancement processing on the input image features of an i-th convolutional layer of the N convolutional layers according to residual features of the i-th convolutional layer, so as to obtain enhanced image features of the i-th convolutional layer, where the residual features represent differences between the input image features of the i-th convolutional layer and image features of the i-th convolutional layer processed by a convolution operation, the enhanced image features of the i-th convolutional layer are input image features of the i + 1-th convolutional layer, the input image features are obtained according to the image to be processed, and i is a positive integer smaller than or equal to N.

4. The image enhancement method according to any one of claims 1 to 3, wherein the enhanced image features of the image to be processed are enhanced image features of an Nth convolutional layer of the N convolutional layers, and are obtained according to the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

5. The image enhancement method according to any one of claims 1 to 4, wherein the performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image features to obtain an output image comprises:

6. The image enhancement method of claim 5, further comprising:

7. An image enhancement apparatus, comprising:

the acquisition unit is used for acquiring an image to be processed;

the processing unit is used for performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, the neural network comprises N convolutional layers, and N is a positive integer; and performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.

8. The image enhancement device according to claim 7, wherein the processing unit is specifically configured to:

9. The image enhancement device according to claim 8, wherein the laplacian enhancement algorithm is configured to perform the feature enhancement processing on the input image features of an i-th convolutional layer of the N convolutional layers according to residual features of the i-th convolutional layer, so as to obtain enhanced image features of the i-th convolutional layer, where the residual features represent differences between the input image features of the i-th convolutional layer and image features of the i-th convolutional layer processed by a convolution operation, the enhanced image features of the i-th convolutional layer are input image features of the i + 1-th convolutional layer, the input image features are obtained according to the image to be processed, and i is a positive integer smaller than or equal to N.

10. The image enhancement device according to any one of claims 7 to 9, wherein the enhanced image feature of the image to be processed is an image feature output by an nth convolutional layer of the N convolutional layers, and the processing unit is specifically configured to:

obtaining the enhanced image characteristics of the image to be processed according to the following equation:

L(F_N)＝Φ(F_N)+s_l·(Φ(F_N)-F_N)；

11. The image enhancement device according to any one of claims 7 to 10, wherein the processing unit is specifically configured to:

12. The image enhancement device of claim 11, wherein the processing unit is further to:

13. An image enhancement apparatus, comprising:

a memory for storing a program;

a processor for executing the memory-stored program, the processor for performing the image enhancement method of any one of claims 1 to 6 when the processor executes the memory-stored program.

14. A computer-readable medium, characterized in that the computer-readable medium stores program code which, when run on a computer, causes the computer to perform the image enhancement method according to any one of claims 1 to 6.

15. A chip, comprising: a processor interfacing with data, the processor reading instructions stored on a memory through the data interface to perform the image enhancement method of any one of claims 1 to 6.