WO2021063341A1 - Procédé et appareil d'amélioration d'image - Google Patents

Procédé et appareil d'amélioration d'image Download PDF

Info

Publication number
WO2021063341A1
WO2021063341A1 PCT/CN2020/118721 CN2020118721W WO2021063341A1 WO 2021063341 A1 WO2021063341 A1 WO 2021063341A1 CN 2020118721 W CN2020118721 W CN 2020118721W WO 2021063341 A1 WO2021063341 A1 WO 2021063341A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
feature
processed
enhancement
enhanced
Prior art date
Application number
PCT/CN2020/118721
Other languages
English (en)
Chinese (zh)
Inventor
宋风龙
熊志伟
王宪
黄杰
查正军
Original Assignee
华为技术有限公司
中国科学技术大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司, 中国科学技术大学 filed Critical 华为技术有限公司
Publication of WO2021063341A1 publication Critical patent/WO2021063341A1/fr

Links

Images

Classifications

    • G06T5/90—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20192—Edge enhancement; Edge preservation

Definitions

  • This application relates to the field of artificial intelligence, and more specifically, to an image enhancement method and device in the field of computer vision.
  • Computer vision is an inseparable part of various intelligent/autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain What we need is the knowledge of the data and information of the subject. To put it vividly, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data.
  • computer vision is to use various imaging systems to replace the visual organs to obtain input information, and then the computer replaces the brain to complete the processing and interpretation of the input information.
  • the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.
  • Image enhancement is an important branch in the field of image processing.
  • Image enhancement technology can improve image quality without re-collecting data to meet more practical application requirements.
  • image enhancement technology can purposely emphasize the overall or local characteristics of an image (or video), make an unclear image clear or emphasize certain features of interest, and expand the difference between the features of different objects in the image. , Suppress uninteresting features, make it improve the image quality, enrich the amount of information, strengthen the image interpretation and recognition effect, and meet the needs of some special analysis.
  • the present application provides an image enhancement method and device, which can enhance the performance of the image to be processed in terms of details, color, and brightness, thereby improving the effect of image enhancement processing.
  • an image enhancement method including: acquiring an image to be processed; performing feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, and the neural network includes N A convolutional layer, where N is a positive integer; performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image.
  • the above-mentioned image to be processed may be an original image with poor image quality; for example, it may mean that the image to be processed is affected by factors such as weather, distance, and shooting environment, and the acquired image to be processed is blurred or has low image quality. , Or image color and brightness are low and other issues.
  • the above-mentioned color enhancement processing can be used to improve the color distribution of the image to be processed, and increase the color saturation of the image to be processed;
  • the brightness enhancement processing can refer to adjusting the brightness of the image to be processed;
  • the feature enhancement processing can refer to enhancing the Details, so that the image includes more detailed information; for example, feature enhancement processing can refer to the feature enhancement of the image to be processed.
  • the image enhancement method provided by the embodiments of the application obtains the image enhancement feature of the image to be processed by performing feature enhancement processing on the image to be processed, and uses the image enhancement feature to further perform color enhancement processing and brightness enhancement processing on the image to be processed, thereby performing color enhancement processing. Enhancement and brightness enhancement can also enhance the details of the image to be processed, so that the performance of the image to be processed in terms of detail, color, and brightness can be enhanced, thereby enhancing the effect of image enhancement processing.
  • image enhancement can also be referred to as image quality enhancement.
  • image quality enhancement can refer to processing the brightness, color, contrast, saturation, and/or dynamic range of the image, so that the various indicators of the image meet the preset values. condition.
  • the performing the feature enhancement processing on the image to be processed through a neural network to obtain the enhanced image feature of the image to be processed includes:
  • the Lass enhancement algorithm performs the feature enhancement processing on the image to be processed to obtain the enhanced image feature of the image to be processed.
  • the feature enhancement of the image to be processed can be realized by the Laplacian enhancement algorithm, where the Laplacian enhancement algorithm can realize that the feature enhancement of the image to be processed does not introduce new textures, so that To a certain extent, the problem of introducing false textures into the output image after image enhancement processing can be avoided, and the effect of image enhancement processing can be improved.
  • the aforementioned Laplacian enhancement algorithm may enhance the high-frequency features in the image to be processed, so as to obtain the enhanced image features of the image to be processed.
  • the high-frequency features of the image may refer to the details, texture, and other information of the image to be processed.
  • the Laplacian enhancement algorithm is used to compare the i-th convolutional layer based on the residual characteristics of the i-th convolutional layer among the N convolutional layers.
  • the input image feature of the convolutional layer is subjected to the feature enhancement processing to obtain the enhanced image feature of the i-th convolutional layer, wherein the residual feature represents the input image feature of the i-th convolutional layer and the
  • the difference between the image features processed by the convolution operation in the i-th convolutional layer, and the enhanced image feature of the i-th convolutional layer is the input image feature of the i+1th convolutional layer ,
  • the input image feature is obtained according to the image to be processed, and i is a positive integer.
  • the Laplacian enhancement algorithm may be an improved Laplacian enhancement algorithm
  • the improved Laplacian enhancement algorithm according to the embodiment of the present application may be obtained by combining the image in the previous convolutional layer Features are used to enhance subsequent image features to achieve progressive enhancement of image features of different convolutional layers, which can improve the effect of image enhancement processing.
  • the enhanced image feature of the image to be processed is the image feature output by the Nth convolutional layer among the N convolutional layers, which is obtained by the following equation
  • the enhanced image feature of the image to be processed is the image feature output by the Nth convolutional layer among the N convolutional layers, which is obtained by the following equation
  • the enhanced image feature of the image to be processed is the image feature output by the Nth convolutional layer among the N convolutional layers, which is obtained by the following equation
  • the enhanced image feature of the image to be processed is the image feature output by the Nth convolutional layer among the N convolutional layers
  • L(F N ) represents the enhanced image feature of the Nth convolutional layer
  • F N represents the input image feature of the Nth convolutional layer
  • ⁇ represents the convolution of the Nth convolutional layer Core
  • s l represents the scaling parameter obtained through learning
  • N is a positive integer.
  • Example embodiments of the present application can be replaced by the parameter s l learning enhancement algorithm conventional Laplacian fixed scaling factor s c; at the same time, the residual layer is characterized by contiguous enhanced features, may be used wherein the residual Indicates any information that needs to be emphasized. Therefore, the Laplacian algorithm of the embodiment of the present application can not only enhance the high-frequency information of the image, but also realize the progressive enhancement of the image features of different convolutional layers, thereby improving the effect of the image enhancement processing.
  • the performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image includes: The enhanced image feature of the image is used to obtain the confidence image feature and the illumination compensation image feature of the image to be processed, where the confidence image feature is used to enhance the color of the image to be processed, and the illumination compensation image feature is used to The brightness of the to-be-processed image is enhanced; and the output image is obtained according to the to-be-processed image, the features of the confidence image, and the features of the illumination compensation image.
  • the above-mentioned confidence image feature may represent a mapping relationship or a mapping function for performing color enhancement processing on the image to be processed.
  • the feature of the confidence image can correspond to the image feature of the image to be processed.
  • an element in the feature of the confidence image can be used to indicate the zoom degree of the corresponding element in the image feature of the image to be processed;
  • the zooming of the area can realize the color enhancement of the image to be processed.
  • the above-mentioned enhanced image features of the image to be processed may include more detailed features and textures; performing color enhancement and brightness enhancement processing on the image to be processed according to the enhanced image features of the image to be processed can enable the output image to achieve detail enhancement, and at the same time It can also improve the brightness and color of the output image.
  • the confidence image feature used for color enhancement processing and the illumination compensation image feature used for brightness enhancement processing can be obtained by the enhanced image feature of the image to be processed, which is compared with the traditional zoom method.
  • the confidence image feature and the illumination compensation image feature in the embodiment of this application can not only perform color enhancement and brightness enhancement, but also achieve the processing The details of the image are enhanced to improve the effect of image enhancement processing.
  • the method further includes: obtaining the confidence image feature and the illumination compensation image feature by performing a convolution operation on the enhanced image feature of the image to be processed;
  • the image feature of the image to be processed is multiplied by the confidence image feature to obtain the color-enhanced image feature of the image to be processed; the color-enhanced image feature and the illumination compensation image feature are fused to obtain the output image .
  • the above-mentioned confidence image feature and the above-mentioned illumination compensation image feature can be obtained through the enhanced image feature of the image to be processed in parallel; for example, the enhanced image feature of the image to be processed can be performed through the first branch in the network model.
  • a convolution operation is performed to obtain the above-mentioned confidence image feature; the second branch in the network model is used to perform a convolution operation on the enhanced image feature of the image to be processed to obtain the above-mentioned illumination compensation image feature.
  • the fusion of the above-mentioned color-enhanced image feature and the illumination compensation image feature may refer to the addition of the color-enhanced image feature and the illumination compensation image feature.
  • an image enhancement method including: detecting a first operation for opening a camera by a user; in response to the first operation, displaying a shooting interface on the display screen, the shooting interface including a viewfinder A frame, the viewfinder frame includes a first image; a second operation instructed by the user to the camera is detected; in response to the second operation, a second image is displayed in the viewfinder frame, or in the electronic device Save a second image, the second image is obtained by performing color enhancement processing and brightness enhancement processing on the first image according to the enhanced image feature of the first image, and the enhanced image feature of the first image is obtained through neural It is obtained by performing feature enhancement processing on the first image by a network, and the neural network includes N convolutional layers, and N is a positive integer.
  • the foregoing specific process of performing feature enhancement processing on the first image may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.
  • the image enhancement method provided by the embodiments of this application can be applied to the field of photographing of smart terminals.
  • the image enhancement method of the embodiments of this application can perform processing on the original images with poor image quality acquired by the smart terminal.
  • the image enhancement process obtains an output image with improved image quality.
  • it can be an image enhancement process on the acquired original image when the smart terminal is taking a real-time photo, and the output image after the image enhancement process is displayed on the screen of the smart terminal, or , You can also save the output image after the image enhancement processing to the album of the smart terminal by performing image enhancement processing on the acquired original image.
  • an image enhancement method including: acquiring a road image to be processed; performing feature enhancement processing on the road image to be processed through a neural network to obtain enhanced image features of the road image to be processed, so
  • the neural network includes N convolutional layers, where N is a positive integer; according to the enhanced image characteristics, the road image to be processed is subjected to color enhancement processing and brightness enhancement processing to obtain a processed output road image; according to the processing After the output road screen, the information in the output road screen is recognized.
  • the foregoing specific process of performing feature enhancement processing on the road image may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.
  • the image enhancement method provided in the embodiment of the present application may be applied to the field of automatic driving.
  • it can be applied to the navigation system of an autonomous vehicle.
  • the image enhancement method in this application can enable the autonomous vehicle to perform image enhancement processing through the acquired original road images with lower image quality during the navigation process of the autonomous vehicle while driving on the road. Obtain the enhanced road picture, so as to realize the safety of self-driving vehicles.
  • an image enhancement method including: acquiring a street view image; performing feature enhancement processing on the street view image through a neural network to obtain enhanced image features of the street view image, the neural network including N convolutional layers , N is a positive integer; perform color enhancement processing and brightness enhancement processing on the street view image according to the enhanced image characteristics to obtain a processed output street view image; identify the output street view image according to the processed output street view image Information in.
  • the foregoing specific process of performing feature enhancement processing on the street view image may be obtained according to the foregoing first aspect and any one of the implementation manners of the first aspect.
  • the image enhancement method provided in the embodiment of the present application can be applied to the security field.
  • the image enhancement method of the embodiment of the present application can be applied to the surveillance image enhancement of a safe city.
  • the image (or video) collected by the surveillance equipment in public places is often affected by factors such as weather and distance, and the image is blurred. Problems such as low image quality.
  • the image enhancement method of the present application can perform image enhancement on the collected original images, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information can be provided for case detection.
  • an image enhancement device which includes a module for executing the image enhancement method in any one of the foregoing first to fourth aspects and the first to fourth aspects.
  • an image enhancement device including: a memory for storing a program; a processor for executing the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute: Processing an image; performing feature enhancement processing on the image to be processed to obtain an enhanced image feature of the image to be processed; performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image.
  • the processor included in the above-mentioned image enhancement device is also used in the image enhancement method in any one of the above-mentioned first aspect to the fourth aspect and the first aspect to the fourth aspect.
  • a computer-readable medium stores program code for device execution, and the program code includes the program code for executing the first aspect to the fourth aspect and the first aspect to the fourth aspect.
  • the image enhancement method in any one of the implementations.
  • a computer program product containing instructions is provided.
  • the computer program product runs on a computer, the computer executes any one of the first to fourth aspects and the first to fourth aspects.
  • the image enhancement method in the implementation mode.
  • a chip in a ninth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes the first to fourth aspects and the first aspect. To the image enhancement method in any one of the implementation manners of the fourth aspect.
  • the chip may further include a memory in which instructions are stored, and the processor is configured to execute instructions stored on the memory.
  • the processor is configured to execute the image enhancement method in any one of the foregoing first aspect to the fourth aspect and the first aspect to the fourth aspect.
  • FIG. 1 is a schematic diagram of an artificial intelligence main body framework provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of still another application scenario provided by an embodiment of the present application.
  • Figure 5 is a schematic diagram of yet another application scenario provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a convolutional neural network structure provided by an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a chip hardware structure provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of an image enhancement model provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of a Laplacian enhancement unit and a hybrid enhancement unit provided by an embodiment of the present application.
  • FIG. 13 is a schematic diagram of feature enhancement processing provided by an embodiment of the present application.
  • FIG. 14 is a schematic diagram of color enhancement processing and brightness enhancement processing provided by an embodiment of the present application.
  • 15 is a schematic diagram of color enhancement processing and brightness enhancement processing provided by an embodiment of the present application.
  • FIG. 16 is a schematic diagram of a visual quality evaluation result provided by an embodiment of the present application.
  • FIG. 17 is a schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • FIG. 18 is a schematic diagram of a set of display interfaces provided by an embodiment of the present application.
  • FIG. 19 is a schematic diagram of another set of display interfaces provided by an embodiment of the present application.
  • FIG. 20 is a schematic block diagram of an image enhancement device provided by an embodiment of the present application.
  • FIG. 21 is a schematic diagram of the hardware structure of an image enhancement device provided by an embodiment of the present application.
  • the images in the embodiments of the present application may be static images (or referred to as static pictures) or dynamic images (or referred to as dynamic pictures).
  • the images in the present application may be videos or dynamic pictures, or the present application
  • the images in can also be static pictures or photos.
  • static images or dynamic images are collectively referred to as images.
  • Figure 1 shows a schematic diagram of an artificial intelligence main framework, which describes the overall workflow of the artificial intelligence system and is suitable for general artificial intelligence field requirements.
  • Intelligent Information Chain reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has gone through the condensing process of "data-information-knowledge-wisdom".
  • the "IT value chain” is the industrial ecological process from the underlying infrastructure and information (providing and processing technology realization) of human intelligence to the system, reflecting the value that artificial intelligence brings to the information technology industry.
  • the infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the basic platform.
  • the infrastructure can communicate with the outside through sensors, and the computing power of the infrastructure can be provided by smart chips.
  • the smart chip here can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), and an application specific integrated circuit (application specific).
  • Hardware acceleration chips such as integrated circuit (ASIC) and field programmable gate array (FPGA).
  • the basic platform of infrastructure can include distributed computing framework and network and other related platform guarantees and support, and can include cloud storage and computing, interconnection networks, etc.
  • data can be obtained through sensors and external communication, and then these data can be provided to the smart chip in the distributed computing system provided by the basic platform for calculation.
  • the data in the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence.
  • the data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and sensory data such as force, displacement, liquid level, temperature, and humidity.
  • the above-mentioned data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.
  • machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, training, etc.
  • Reasoning refers to the process of simulating human intelligent reasoning in a computer or intelligent system, using formal information to conduct machine thinking and solving problems based on reasoning control strategies.
  • the typical function is search and matching.
  • Decision-making refers to the process of making decisions based on intelligent information after reasoning, and usually provides functions such as classification, sorting, and prediction.
  • some general capabilities can be formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image Recognition and so on.
  • Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is an encapsulation of the overall solution of artificial intelligence, productizing intelligent information decision-making and realizing landing applications. Its application fields mainly include: intelligent manufacturing, intelligent transportation, Smart home, smart medical, smart security, autonomous driving, safe city, smart terminal, etc.
  • Fig. 2 is a schematic diagram of an application scenario of an image enhancement method provided by an embodiment of the present application.
  • the technical solution of the embodiment of the present application can be applied to a smart terminal.
  • the image enhancement method in the embodiment of the present application can perform image enhancement processing on an input image to obtain an output image of the input image after image enhancement.
  • the smart terminal may be mobile or fixed.
  • the smart terminal may be a mobile phone with image enhancement function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer, LC), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, augmented reality (AR)/virtual reality (VR), wearable Wearable device (WD) or self-driving vehicle, etc., which are not limited in the embodiment of the present application.
  • image enhancement can also be referred to as image quality enhancement. Specifically, it can refer to processing the brightness, color, contrast, saturation, and/or dynamic range of the image to make the image The indicators meet the preset conditions.
  • image enhancement and image quality enhancement have the same meaning.
  • Application scenario 1 Smart terminal camera field
  • the image enhancement method of the embodiment of the present application may be applied to the shooting of a smart terminal device (for example, a mobile phone).
  • the image enhancement method of the embodiment of the present application can perform image enhancement processing on the acquired original image of poor quality to obtain an output image with improved image quality.
  • the color image portion is represented by oblique line filling.
  • the image enhancement method of the embodiment of the present application can perform image enhancement processing on the acquired original image when the smart terminal is taking real-time photos, and display the output image after the image enhancement processing on the screen of the smart terminal.
  • the image enhancement method of the embodiment of the present application may be used to perform image enhancement processing on the acquired original image, and the output image after the image enhancement processing can be saved in the album of the smart terminal.
  • this application proposes an image enhancement method, which is applied to an electronic device with a display screen and a camera, including: detecting a first operation of a user to turn on the camera; in response to the first operation, A photographing interface is displayed on the display screen, the photographing interface includes a viewfinder frame, and the first image is included in the viewfinder frame; a second operation instructed by the user to the camera is detected; in response to the second operation, in the viewfinder Display a second image in a frame, or save a second image in the electronic device, where the second image is obtained by performing color enhancement processing and brightness enhancement processing on the first image according to the enhanced image characteristics of the first image
  • the enhanced image feature of the first image is obtained by performing feature enhancement processing on the first image through a neural network, and the neural network includes N convolutional layers, and N is a positive integer.
  • image enhancement method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the following related embodiments in FIGS. 6 to 16, which will not be repeated here.
  • the image enhancement method of the embodiment of the present application can be applied to the field of automatic driving.
  • it can be applied to the navigation system of an autonomous vehicle.
  • the image enhancement method in this application can enable the autonomous vehicle to perform image enhancement processing through the acquired original road images with lower image quality during the navigation process of the autonomous vehicle while driving on the road. Obtain the enhanced road picture, so as to realize the safety of self-driving vehicles.
  • this application provides an image enhancement method.
  • the method includes: acquiring a road image to be processed; performing feature enhancement processing on the road image to be processed through a neural network to obtain an image of the road image to be processed
  • the neural network includes N convolutional layers, where N is a positive integer; according to the enhanced image features, perform color enhancement processing and brightness enhancement processing on the road image to be processed to obtain a processed output road image ; According to the processed output road screen, identify the information in the output road screen.
  • image enhancement method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the following related embodiments in FIGS. 6 to 16, which will not be repeated here.
  • the image enhancement method of the embodiment of the present application can be applied to the security field.
  • the image enhancement method of the embodiment of the present application can be applied to the surveillance image enhancement of a safe city.
  • the image (or video) collected by the surveillance equipment in public places is often affected by factors such as weather and distance, and the image is blurred. Problems such as low image quality.
  • the image enhancement method of the present application can perform image enhancement on the collected pictures, so that important information such as license plate numbers and clear human faces can be recovered for public security personnel, and important clue information can be provided for case detection.
  • the present application provides an image enhancement method, the method includes: acquiring a street view image; performing feature enhancement processing on the street view image through a neural network to obtain an enhanced image feature of the street view image, and the neural network includes N convolutional layers, where N is a positive integer; perform color enhancement processing and brightness enhancement processing on the street view image according to the enhanced image characteristics to obtain a processed output street view image; according to the processed output street view image, identify Said outputting the information in the street view picture.
  • image enhancement method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the following related embodiments in FIGS. 6 to 16, which will not be repeated here.
  • the image enhancement method of the embodiment of the present application may also be applied to a source enhancement scene.
  • a source enhancement scene For example, when using smart terminals (such as smart TVs, smart screens, etc.) to play movies, in order to display better image quality (picture quality), the original source of the movie can be enhanced by using the image enhancement method of the embodiment of the application. Processing to improve the image quality of the film source and obtain a better visual impression.
  • the image enhancement method of the embodiment of the application can be used to compare the old movies.
  • the source performs image enhancement processing, which can show the visual sense of modern movies.
  • the source of an old movie can be enhanced to a high-dynamic range (HDR) 10 or a high-quality video of the Dolby Vision (Dolby Vision) standard through the image enhancement method of the embodiment of the present application.
  • HDR high-dynamic range
  • Dolby Vision Dolby Vision
  • the present application provides an image enhancement method, which includes: obtaining an original image (for example, the original film source of a movie); performing feature enhancement processing on the original image through a neural network to obtain the original image
  • the neural network includes N convolutional layers, where N is a positive integer; according to the enhanced image features, the original image is subjected to color enhancement processing and brightness enhancement processing to obtain a processed output image (for example, enhanced Picture quality film source).
  • image enhancement method provided by the embodiment of the present application is also applicable to the extension, limitation, explanation and description of the related content of the image enhancement method in the following related embodiments in FIGS. 6 to 16, which will not be repeated here.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes x s and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • W s is the weight of x s
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • the DNN is divided according to the positions of different layers.
  • the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated as far as the work of each layer is concerned. Simply put, it is the following linear relationship expression: among them, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • DNN The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.
  • the coefficient from the kth neuron of the L-1 layer to the jth neuron of the Lth layer is defined as
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can only be connected to a part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way of extracting image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network.
  • the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.
  • Important equation taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, then the training of the deep neural network becomes a process of reducing this loss as much as possible.
  • the neural network can use an error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forwarding the input signal until the output will cause error loss, and the parameters in the initial neural network model are updated by backpropagating the error loss information, so that the error loss is converged.
  • the backpropagation algorithm is a backpropagation motion dominated by error loss, and aims to obtain the optimal parameters of the neural network model, such as the weight matrix.
  • Fig. 6 shows a system architecture 200 provided by an embodiment of the present application.
  • a data collection device 260 is used to collect training data.
  • the image enhancement model also referred to as an image enhancement network
  • the training data collected by the data collection device 260 can be training images.
  • the training data of the training image enhancement model in the embodiment of the present application may include original images and sample enhanced images.
  • the original image may refer to an image with lower image quality
  • the sample-enhanced image may refer to an image with higher image quality.
  • it may refer to one or more of the brightness, color, details, etc., relative to the sample image. The image after all aspects have been improved.
  • image enhancement can also be referred to as image quality enhancement, which can specifically refer to processing the brightness, color, contrast, saturation, and/or dynamic range of an image, so that various indicators of the image meet preset conditions .
  • image enhancement and image quality enhancement have the same meaning.
  • the data collection device 260 stores the training data in the database 230, and the training device 220 trains based on the training data maintained in the database 230 to obtain the target model/rule 201 (that is, the image enhancement model in the embodiment of the present application) .
  • the training device 220 inputs the training data to the image enhancement model until the difference between the predicted enhanced image output by the training image enhancement model and the sample enhanced image meets a preset condition (for example, the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold , Or predict that the difference between the enhanced image and the sample enhanced image remains unchanged or no longer decreases), thereby completing the training of the target model/rule 201.
  • a preset condition for example, the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold , Or predict that the difference between the enhanced image and the sample enhanced image remains unchanged or no longer decreases
  • the image enhancement model used to perform the image enhancement method in the embodiment of the present application can realize end-to-end training.
  • the image can be enhanced by the input image and the sample corresponding to the input image ( For example, true-value images) realize end-to-end training.
  • the target model/rule 201 is obtained by training an image enhancement model. It should be noted that in actual applications, the training data maintained in the database 230 may not all come from the collection of the data collection device 260, and may also be received from other devices.
  • the training device 220 does not necessarily perform the training of the target model/rule 201 completely based on the training data maintained by the database 230. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of the embodiment. It should also be noted that at least part of the training data maintained in the database 230 may also be used to execute the process of the processing to be processed by the execution device 210.
  • the target model/rule 201 trained according to the training device 220 can be applied to different systems or devices, such as the execution device 210 shown in FIG. 6, the execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, car terminals, etc., can also be servers or cloud, etc.
  • the execution device 210 may be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, car terminals, etc., can also be servers or cloud, etc.
  • the execution device 210 is configured with an input/output (input/output, I/O) interface 212 for data interaction with external devices.
  • the user can input data to the I/O interface 212 through the client device 240.
  • the input data in this embodiment of the present application may include: a to-be-processed image input by the client device.
  • the preprocessing module 213 and the preprocessing module 214 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 212.
  • the preprocessing module 213 and the preprocessing module may not be provided. 214 (there may only be one of the preprocessing modules), and the calculation module 211 is directly used to process the input data.
  • the execution device 210 When the execution device 210 preprocesses input data, or when the calculation module 211 of the execution device 210 performs calculations and other related processing, the execution device 210 can call data, codes, etc. in the data storage system 250 for corresponding processing. , The data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 250.
  • the I/O interface 212 returns the processing result to the image enhancement image to be processed as described above, and returns the resulting output image to the client device 240 to provide the user.
  • the training device 220 can generate corresponding target models/rules 201 based on different training data for different goals or tasks, and the corresponding target models/rules 201 can be used to achieve the above goals or complete The above tasks provide users with the desired results.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 212.
  • the client device 240 can automatically send input data to the I/O interface 212. If the client device 240 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 240. The user can view the result output by the execution device 210 on the client device 240, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 240 can also be used as a data collection terminal to collect the input data of the input I/O interface 212 and the output result of the output I/O interface 212 as new sample data, and store it in the database 230 as shown in the figure. Of course, it is also possible not to collect through the client device 240. Instead, the I/O interface 212 directly uses the input data input to the I/O interface 212 and the output result of the output I/O interface 212 as a new sample as shown in the figure. The data is stored in the database 230.
  • FIG. 6 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data The storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210.
  • the target model/rule 201 is trained according to the training device 220.
  • the target model/rule 201 may be an image enhancement model in the embodiment of the present application.
  • the image enhancement model provided in the embodiment of the present application may be Deep neural network, convolutional neural network, or, it can be deep convolutional neural network, etc.
  • a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture.
  • the deep learning architecture refers to the algorithm of machine learning. Multi-level learning is carried out on the abstract level of the system.
  • a convolutional neural network is a feed-forward artificial neural network, and each neuron in the feed-forward artificial neural network can respond to the input image.
  • the convolutional neural network 300 may include an input layer 310, a convolutional layer/pooling layer 320 (wherein the pooling layer is optional), a fully connected layer 330 and an output layer 340.
  • the input layer 310 can obtain the image to be processed, and pass the obtained image to be processed to the convolutional layer/pooling layer 320 and the fully connected layer 330 for processing, and the processing result of the image can be obtained.
  • the convolutional layer/pooling layer 320 may include layers 321-326, for example: in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, and layer 323 is a convolutional layer. Layers, 324 is a pooling layer, 325 is a convolutional layer, and 326 is a pooling layer; in another implementation, 321 and 322 are convolutional layers, 323 is a pooling layer, and 324 and 325 are convolutional layers. Layer, 326 is the pooling layer, that is, the output of the convolutional layer can be used as the input of the subsequent pooling layer, or can be used as the input of another convolutional layer to continue the convolution operation.
  • the convolution layer 321 can include many convolution operators.
  • the convolution operator is also called a kernel. Its role in image processing is equivalent to a filter that extracts specific information from the input image matrix.
  • the convolution operator is essentially It can be a weight matrix. This weight matrix is usually pre-defined. In the process of convolution on the image, the weight matrix is usually one pixel after one pixel (or two pixels after two pixels) along the horizontal direction on the input image. And so on, it depends on the value of stride) to complete the process of extracting specific features from the image.
  • the size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix and the depth dimension of the input image are the same.
  • the weight matrix will extend to Enter the entire depth of the image. Therefore, convolution with a single weight matrix will produce a single depth dimension convolution output, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row ⁇ column) are applied. That is, multiple homogeneous matrices.
  • the output of each weight matrix is stacked to form the depth dimension of the convolutional image, where the dimension can be understood as determined by the "multiple" mentioned above.
  • Different weight matrices can be used to extract different features in the image. For example, one weight matrix is used to extract the edge information of the image, another weight matrix is used to extract the specific color of the image, and the other weight matrix is used to correct the unwanted images in the image. The noise is blurred and so on.
  • the multiple weight matrices have the same size (row ⁇ column), the size of the convolution feature maps extracted by the multiple weight matrices of the same size are also the same, and then the multiple extracted convolution feature maps of the same size are merged to form The output of the convolution operation.
  • weight values in these weight matrices need to be obtained through a lot of training in practical applications.
  • Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, so that the convolutional neural network 300 can make correct predictions. .
  • the initial convolutional layer (such as 321) often extracts more general features, and the general features can also be called low-level features; with the convolutional neural network With the 300-depth deepening, the features extracted by the subsequent convolutional layer (for example, 326) become more and more complex, for example, features such as high-level semantics, and features with higher semantics are more suitable for the problem to be solved.
  • the pooling layer can be a convolutional layer followed by a layer.
  • the pooling layer can also be a multi-layer convolutional layer followed by one or more pooling layers.
  • the purpose of the pooling layer is to reduce the size of the image space.
  • the pooling layer may include an average pooling operator and/or a maximum pooling operator for sampling the input image to obtain an image with a smaller size.
  • the average pooling operator can calculate the pixel values in the image within a specific range to generate an average value as the result of the average pooling.
  • the maximum pooling operator can take the pixel with the largest value within a specific range as the result of the maximum pooling.
  • the operators in the pooling layer should also be related to the image size.
  • the size of the image output after processing by the pooling layer can be smaller than the size of the image of the input pooling layer, and each pixel in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.
  • the convolutional neural network 300 After processing by the convolutional layer/pooling layer 320, the convolutional neural network 300 is not enough to output the required output information. Because as mentioned above, the convolutional layer/pooling layer 320 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (required class information or other related information), the convolutional neural network 300 needs to use the fully connected layer 330 to generate one or a group of required classes of output. Therefore, the fully connected layer 330 may include multiple hidden layers (331, 332 to 33n as shown in FIG. 7) and an output layer 340. The parameters contained in the multiple hidden layers can be based on specific task types. Relevant training data of, for example, the task type can include image enhancement, image recognition, image classification, image detection, and image super-resolution reconstruction, etc.
  • the output layer 340 After the multiple hidden layers in the fully connected layer 330, that is, the final layer of the entire convolutional neural network 300 is the output layer 340.
  • the output layer 340 has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error.
  • the convolutional neural network shown in FIG. 7 is only used as an example of the structure of the image enhancement model of the embodiment of the present application.
  • the convolutional neural network used in the image enhancement method of the embodiment of the present application It can also exist in the form of other network models.
  • the image enhancement device may include the convolutional neural network 300 shown in FIG. 7, and the image enhancement device may perform image enhancement processing on the image to be processed to obtain a processed output image.
  • FIG. 8 is a hardware structure of a chip provided by an embodiment of the present application.
  • the chip includes a neural network processor 400 (neural-network processing unit, NPU).
  • the chip can be set in the execution device 210 as shown in FIG. 6 to complete the calculation work of the calculation module 211.
  • the chip can also be set in the training device 220 shown in FIG. 6 to complete the training work of the training device 220 and output the target model/rule 201.
  • the algorithms of each layer in the convolutional neural network as shown in FIG. 7 can be implemented in the chip as shown in FIG. 8.
  • the NPU 400 is mounted on a main central processing unit (CPU) as a coprocessor, and the main CPU allocates tasks.
  • the core part of the NPU 400 is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 403 includes multiple processing units (process engines, PE). In some implementations, the arithmetic circuit 403 is a two-dimensional systolic array. The arithmetic circuit 403 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.
  • the arithmetic circuit 403 fetches the data corresponding to matrix B from the weight memory 402 and caches it on each PE in the arithmetic circuit 403.
  • the arithmetic circuit 403 fetches the matrix A data and matrix B from the input memory 401 to perform matrix operations, and the partial result or final result of the obtained matrix is stored in an accumulator 408 (accumulator).
  • the vector calculation unit 407 can perform further processing on the output of the arithmetic circuit 403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 407 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 407 can store the processed output vector to the unified memory 406.
  • the vector calculation unit 407 may apply a nonlinear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 407 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 403, for example for use in a subsequent layer in a neural network.
  • the unified memory 406 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller 405 (direct memory access controller, DMAC) to store the input data in the external memory into the input memory 401 and/or unified memory 406, and the weight data in the external memory into the weight memory 402 , And store the data in the unified memory 406 into the external memory.
  • DMAC direct memory access controller
  • the bus interface unit 410 (bus interface unit, BIU) is used to implement interaction between the main CPU, the DMAC, and the fetch memory 409 through the bus.
  • An instruction fetch buffer 409 (instruction fetch buffer) connected to the controller 404 is used to store instructions used by the controller 404.
  • the controller 404 is used to call the instructions cached in the instruction fetch memory 409 to control the working process of the computing accelerator.
  • the unified memory 406, the input memory 401, the weight memory 402, and the instruction fetch memory 409 are all on-chip (On-Chip) memories.
  • the external memory is a memory external to the NPU.
  • the external memory can be a double data rate synchronous dynamic random access memory.
  • Memory double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • each layer in the convolutional neural network shown in FIG. 7 may be executed by the arithmetic circuit 403 or the vector calculation unit 407.
  • the execution device 210 in FIG. 6 introduced above can execute each step of the image enhancement method of the embodiment of the present application.
  • the CNN model shown in FIG. 7 and the chip shown in FIG. 8 can also be used to execute the image of the embodiment of the present application. Steps of the enhancement method.
  • FIG. 9 shows a system architecture 500 provided by an embodiment of the present application.
  • the system architecture includes a local device 520, a local device 530, an execution device 510, and a data storage system 550.
  • the local device 520 and the local device 530 are connected to the execution device 510 through a communication network.
  • the execution device 510 may be implemented by one or more servers.
  • the execution device 510 can be used in conjunction with other computing devices.
  • data storage for example: data storage, routers, load balancers and other equipment.
  • the execution device 510 may be arranged on one physical site or distributed on multiple physical sites.
  • the execution device 510 may use the data in the data storage system 550 or call the program code in the data storage system 550 to implement the image enhancement method of the embodiment of the present application.
  • execution device 510 may also be referred to as a cloud device, and in this case, the execution device 510 may be deployed in the cloud.
  • the execution device 510 may perform the following process: obtain an image to be processed; perform feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, and the neural network includes N convolutions. Layer, N is a positive integer; perform color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image characteristics to obtain an output image.
  • the image enhancement method of the embodiment of the present application may be an offline method executed in the cloud.
  • the image enhancement method of the embodiment of the present application may be executed by the execution device 510 described above.
  • the image enhancement method in the embodiment of the present application may be executed by the local device 520 or the local device 530.
  • image enhancement can be performed on the acquired image to be processed with poor image quality, so as to obtain an output image with improved performance of the image to be processed in terms of image details, image color, and image brightness.
  • the user can operate respective user devices (for example, the local device 520 and the local device 530) to interact with the execution device 510.
  • Each local device can represent any computing device, for example, a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.
  • the local device of each user can interact with the execution device 510 through a communication network of any communication mechanism/communication standard.
  • the communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.
  • the local device 520 and the local device 530 can obtain the relevant parameters of the target neural network from the execution device 510, deploy the target neural network on the local device 520 and the local device 530, and use the target neural network to perform image processing. Enhanced processing, etc.
  • the target neural network can be directly deployed on the execution device 510.
  • the execution device 510 obtains the image to be processed from the local device 520 and the local device 530, and performs image enhancement processing on the image to be processed according to the target neural network.
  • the above-mentioned target neural network may be the image enhancement model in the embodiment of the present application.
  • the camera imaging and video of smart terminals are limited by the hardware performance of the optical sensors of smart terminals, and the quality of photos and videos taken by them is still not high enough, and there are problems such as high noise, low resolution, lack of detail, and color cast.
  • Image (or picture) enhancement is the basis of various image processing applications, and computer vision often involves the issue of how to enhance the acquired images.
  • the existing image enhancement methods can be divided into two categories: the first category, scaling method, that is, by learning the mapping function from the original image or video to the target image or video, the pixels of the input image or video Or feature scaling (Scale) operation; the second category is Generative Method, which uses generative adversarial networks (GAN) to extract features from the input image or video, generate new elements, and reconstruct the output image Or video.
  • GAN generative adversarial networks
  • the embodiments of the present application provide an image enhancement method, by performing feature enhancement processing on the image to be processed, the image enhancement feature of the image to be processed is obtained, and the image enhancement feature is used to further perform color enhancement and brightness enhancement on the image to be processed Processing, so that when performing color enhancement and brightness enhancement, the details of the image to be processed can be enhanced, so that the performance of the image to be processed in terms of detail, color, and brightness can be enhanced, thereby improving the effect of image enhancement processing.
  • FIG. 10 shows a schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • the image enhancement method shown in FIG. 10 may be executed by an image enhancement device, which specifically may be the execution device 210 in FIG. It may be the execution device 510 in FIG. 9 or a local device.
  • the method shown in FIG. 10 includes steps 610 to 630, and steps 610 to 630 are respectively described in detail below.
  • Step 610 Obtain an image to be processed.
  • the image to be processed may be an original image with poor image quality; for example, it may mean that the image to be processed is affected by factors such as weather, distance, and shooting environment, and the image to be processed is blurred, or the image quality is low, or the image Problems such as low color and brightness.
  • the above-mentioned image to be processed may be an image captured by an electronic device through a camera, or the above-mentioned image to be processed may also be an image obtained from the inside of the electronic device (for example, an image stored in an album of an electronic device, or an electronic device).
  • the electronic device may be any one of the local device or the execution device shown in FIG. 9.
  • Step 620 Perform feature enhancement processing on the image to be processed through the neural network to obtain the enhanced image feature of the image to be processed.
  • the neural network may include N convolutional layers, and N is a positive integer.
  • the feature enhancement processing mentioned above may refer to the feature enhancement of the details of the image to be processed.
  • the aforementioned neural network may refer to the feature extraction part in the image enhancement model shown in FIG. 11; the image enhancement model may include multiple neural networks, the feature extraction part may be the first neural network, and the feature reconstruction part may include the first neural network.
  • Neural network For example, the first neural network can be used to perform feature enhancement processing on the image to be processed to obtain the enhanced image feature of the image to be processed; the second neural network can be used to perform feature enhancement on the image to be processed according to the enhanced image feature. Perform color enhancement processing and brightness enhancement processing to obtain an output image.
  • FIG. 13 may be the first neural network
  • FIG. 14 may be the second neural network.
  • feature enhancement processing can refer to purposefully emphasizing the overall or local characteristics of the image, making the original unclear image clear or emphasizing certain features of interest, and expanding the difference between the features of different objects in the image , Suppress uninteresting features, improve image quality, enrich the amount of information, strengthen image interpretation and recognition effects, and meet the needs of recognition and analysis.
  • Step 630 Perform color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image.
  • the color enhancement processing can be used to improve the color distribution of the image to be processed and increase the color saturation of the image to be processed.
  • the brightness enhancement processing may refer to adjusting the brightness of the image to be processed.
  • the output image may refer to the image obtained by performing image enhancement on the acquired image to be processed.
  • the image enhancement may be referred to as image quality enhancement, which may specifically refer to the brightness and color of the image. , Contrast, saturation, and/or dynamic range, etc., to make one or more indicators of the image meet preset conditions.
  • the feature enhancement processing of the image to be processed may be performed by using the Laplacian enhancement algorithm to obtain the enhanced image feature of the image to be processed.
  • the enhanced image feature of the image to be processed may refer to the enhanced image feature obtained by enhancing details or textures in the image to be processed.
  • the Laplacian enhancement algorithm can be used to achieve the enhancement of the detailed features in the image to be processed.
  • the Laplacian enhancement algorithm does not introduce new features when it is used to enhance the features of the image to be processed. Texture, thereby avoiding the problem of introducing pseudo texture in the output image obtained after image enhancement processing, and can improve the effect of image enhancement processing.
  • the aforementioned Laplacian enhancement algorithm may refer to a traditional Laplacian enhancement algorithm, which is an enhanced image feature obtained by fusing the original image feature of the image to be processed and the high frequency feature of the image to be processed.
  • I may represent the original image feature image to be processed
  • E may represent an enhanced image wherein image to be processed
  • h may represent the blur kernel
  • s c may represent a scaling factor constant values
  • Ih (I) may represent obtained to be processed The high-frequency characteristics of the image.
  • the high-frequency features of the image may refer to information such as details and texture of the image; the low-frequency features of the image may refer to the contour information of the image.
  • an improved Laplacian enhancement algorithm is proposed in the embodiment of the present application, and the improved Laplacian enhancement algorithm of the embodiment of the present application can be obtained by comparing the previous volume
  • the image features in the build-up layer are used to enhance subsequent image features to achieve the progressive enhancement of the image features of different convolutional layers.
  • the Laplacian enhancement algorithm proposed in this embodiment of the application can be used to compare the input image characteristics of the i-th convolutional layer according to the residual characteristics of the i-th convolutional layer in the N convolutional layers.
  • Perform feature enhancement processing to obtain the enhanced image feature of the i-th convolutional layer, where the residual feature can represent the input image feature of the i-th convolutional layer and the image feature processed by the convolution operation in the i-th convolutional layer
  • the difference between, the enhanced image feature of the i-th convolutional layer is the input image feature of the i+1th convolutional layer, the input image feature is obtained from the image to be processed, and i is a positive integer.
  • the enhanced image feature of the image to be processed may be the image feature output by the Nth convolutional layer among the N convolutional layers, and the enhanced image feature of the image to be processed is obtained by the following equation:
  • L(F N ) can represent the enhanced image feature of the Nth convolutional layer
  • F N can represent the input image feature of the Nth convolutional layer
  • ⁇ can represent the convolution kernel of the Nth convolutional layer
  • s l can represent the parameters obtained through learning
  • N is a positive integer.
  • the Laplacian enhancement algorithm proposed in the embodiment of this application replaces the fixed scaling factor in the traditional Laplacian enhancement algorithm with learnable parameters; at the same time, the residual feature of the adjacent layer is used for feature enhancement, and the residual feature Can be used to represent any information that needs to be emphasized. Therefore, the Laplacian algorithm of the embodiment of the present application can not only enhance the high-frequency information of the image, but also can gradually enhance the image features of different convolutional layers, thereby improving the effect of image feature enhancement.
  • the color enhancement and brightness enhancement of the image to be processed can be performed through the acquired enhanced image features. Since the enhanced image features of the image to be processed include more detailed features and textures, the The enhanced image feature of the processed image Performing color enhancement and brightness enhancement processing on the image to be processed can enhance the details of the output image, and also improve the brightness and color of the output image.
  • performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain the output image may include: obtaining the image of the image to be processed according to the enhanced image feature of the image to be processed Confidence image feature and illumination compensation image feature, wherein the confidence image feature is used for color enhancement of the image to be processed, and the illumination compensation image feature is used for brightness enhancement of the image to be processed; The image, the confidence image feature, and the illumination compensation image feature are used to obtain the output image.
  • the above-mentioned confidence image feature may represent a mapping relationship or a mapping function for performing color enhancement processing on the image to be processed.
  • the feature of the confidence image can correspond to the image feature of the image to be processed.
  • an element in the feature of the confidence image can be used to indicate the zoom degree of the corresponding element in the image feature of the image to be processed;
  • the zooming of the area can realize the color enhancement of the image to be processed.
  • the confidence image feature used for color enhancement processing and the illumination compensation image feature used for brightness enhancement processing can be obtained by the enhanced image feature of the image to be processed, which is compared with the traditional zoom method.
  • the confidence image feature and the illumination compensation image feature in the embodiment of this application can not only perform color enhancement and brightness enhancement, but also achieve the processing The details of the image are enhanced to improve the effect of image enhancement processing.
  • the confidence image feature and the illumination compensation image feature can be obtained; by multiplying the image feature of the image to be processed and the confidence image feature Obtain the color-enhanced image feature of the image to be processed; fuse the color-enhanced image feature and the illumination compensation image feature to obtain the output image.
  • the above-mentioned confidence image feature and illumination compensation image feature are obtained based on the enhanced image feature of the image to be processed. Therefore, the confidence image feature will also enhance the image to be processed to a certain extent when it is used to enhance the color of the image to be processed.
  • the image features of the image; similarly, the illumination compensation image feature will also enhance the image feature of the image to be processed when it is used to enhance the brightness of the image to be processed, so as to achieve the enhancement of the details, color and brightness of the image to be processed.
  • the confidence image feature used for color enhancement processing and the image feature used for brightness enhancement processing are obtained through the enhanced image feature of the image to be processed.
  • the confidence image feature and the illumination compensation image feature in the embodiments of the present application can not only perform color enhancement and brightness enhancement, but also achieve detail enhancement of the image to be processed.
  • the above-mentioned confidence image feature and the above-mentioned illumination compensation image feature can be obtained through the enhanced image feature of the image to be processed in parallel; for example, the enhanced image feature of the image to be processed can be performed through the first branch in the network model.
  • a convolution operation is performed to obtain the above-mentioned confidence image feature; the second branch in the network model is used to perform a convolution operation on the enhanced image feature of the image to be processed to obtain the above-mentioned illumination compensation image feature.
  • the output image can be obtained from the image to be processed, the feature of the confidence image, and the feature of the illumination compensation image.
  • the image feature of the image to be processed (for example, the original image feature of the image to be processed) can be multiplied by the feature of the confidence image to realize the color enhancement of different regions in the image to be processed to obtain the color enhancement image feature; and then the color enhancement The image feature and the illumination compensation image feature are fused to obtain the output image after image enhancement processing.
  • the fusion of the aforementioned color-enhanced image feature and the illumination compensation image feature may refer to the addition of the color-enhanced image feature and the illumination compensation image feature to obtain an output image.
  • the output image after image enhancement processing can be obtained through the enhanced image feature, the confidence image feature, and the illumination compensation feature of the image to be processed.
  • the enhanced image feature of the image to be processed can be multiplied with the confidence image feature to realize the color enhancement of different areas in the image to be processed to obtain the color enhanced image feature; then the color enhanced image feature and the illumination compensation image feature are fused to Obtain the output image after image enhancement processing.
  • Fig. 11 is a schematic diagram of a model structure for image enhancement provided by an embodiment of the present application.
  • the model shown in FIG. 11 can be deployed in an image enhancement device that executes the above-mentioned image enhancement method.
  • the model shown in FIG. 11 may include four parts, namely an input part, a feature extraction part, a feature reconstruction part, and an output part.
  • the feature extraction part may include a Laplacian enhancing unit (LEU); the feature reconstruction part may include a hybrid enhancing module (HEM).
  • LEU Laplacian enhancing unit
  • HEM hybrid enhancing module
  • the image enhancement model shown in FIG. 11 may include multiple neural networks, the feature extraction part may be a first neural network, and the feature reconstruction part may include a second neural network.
  • the first neural network can be used to perform feature enhancement processing on the image to be processed to obtain the enhanced image feature of the image to be processed; the second neural network can be used to color the image to be processed according to the enhanced image feature Enhance processing and brightness enhancement processing to get the output image.
  • FIG. 13 may be the first neural network
  • FIG. 14 may be the second neural network.
  • the Laplacian enhancement unit can be embedded in the convolutional layer, and the extracted features can be enhanced by using the Laplacian enhancement unit; specifically, the Laplacian enhancement unit can be used to pass the Laplacian enhancement unit.
  • the Rass enhancement algorithm performs feature enhancement processing on the image to be processed.
  • the extracted features are gradually enhanced through several layers of Laplacian enhancement units, the image features of the previous layer can be used to enhance the image features of the next layer, and the residuals of the image features of the previous layer can be superimposed.
  • the image features of different convolutional layers are gradually enhanced to improve the performance of image enhancement.
  • a hybrid enhancement unit can be used to achieve the advantages of the zooming method and the generative method of image enhancement.
  • the hybrid enhancement unit may use the image feature of the input image processed by the Laplacian enhancement unit as the input data, that is, the hybrid enhancement unit uses the enhanced image feature of the output image output by the Laplacian unit as the input data. Input data.
  • the Laplacian enhancement unit can be used to perform feature enhancement processing on the output image through the Laplacian algorithm
  • the hybrid enhancement unit can be used to perform color enhancement processing and brightness enhancement processing on the output image.
  • the image enhancement model for performing the image enhancement method provided by the embodiment of the application shown in FIG. 11 may be called a hybrid progressive enhancing u-net (HPEU), which uses LEU and HEM, so that the receptive field increases layer by layer; among them, LEU can achieve feature enhancement processing on the output image based on more image information.
  • HPEU hybrid progressive enhancing u-net
  • the Laplacian enhancement unit can be used to perform different levels of enhancement processing on image features of different levels.
  • LEU in the shallow convolutional layer of the image enhancement model, LEU can be mainly used to perform local area feature enhancement processing based on the edge information of the input image; in the deep convolutional layer of the image enhancement model, the receptive field is larger, so LEU Can be used to enhance processing of global features.
  • the above-mentioned receptive field is a term in the field of deep neural network in the field of computer vision, and is used to indicate the size of the receptive range of neurons in different positions within the neural network to the original image.
  • Fig. 12 is a schematic diagram of a Laplacian enhancement unit and a hybrid enhancement unit provided by an embodiment of the present application. As shown in FIG. 12, it may include one or more Laplacian enhancement units and hybrid enhancement units.
  • the input image is extracted by the convolutional layer, and the extracted features are enhanced by the Laplacian enhancement unit to obtain the enhanced Further, the enhanced image features are used as the input data of the subsequent convolutional layer or the subsequent Laplacian enhancement unit layer; until the enhanced image features are input to the hybrid enhancement unit, the feature channel can be divided In two parts, the zoom component and the generated component are calculated separately, and then the two components are merged to obtain the final enhanced image.
  • the above-mentioned scaling component can be used to achieve color enhancement, and the degree of color enhancement in different areas of the input image can be constrained by the confidence map (also called the confidence image feature); the above-mentioned generating component can be used for illumination compensation, thereby achieving contrast and brightness enhancement .
  • FIG. 13 is a schematic diagram of the processing flow of the Laplacian enhancement unit proposed by an embodiment of the application (ie, a schematic diagram of feature enhancement processing).
  • the processing procedure of the Laplace enhancement unit includes the following steps:
  • Step a Suppose the current network (first neural network) comprises N layers convolution, the convolution output data of the N-th layer image feature F N, i.e. F N may be characterized as an enhanced input data processing.
  • Step 2 Extract the features of F N through the Nth convolutional layer, denoted as ⁇ (F N ).
  • Step 3 Calculate the residual error between ⁇ (F N ) and F N.
  • the residual error can be recorded as: ⁇ (F N )-F N , or F N - ⁇ (F N ) and pass the learnable parameter pair The above residual error is enhanced.
  • Step 4 Superimpose the above-mentioned enhancement processing residual error with ⁇ (F N ) to obtain the enhanced image feature obtained after the Nth convolutional layer is subjected to feature enhancement processing by the Laplacian enhancement unit.
  • the image features output by the Nth convolutional layer are obtained by the following equation:
  • L(F N ) can represent the enhanced image feature of the Nth convolutional layer
  • F N can represent the input image feature of the Nth convolutional layer
  • ⁇ can represent the convolution kernel of the Nth convolutional layer
  • s l can represent the parameters obtained through learning, and can be used to represent the enhancement degree of LEU enhancement of image features each time.
  • the first neural network may include N convolutional layers, and the image feature output by the Nth convolutional layer may be the enhanced image feature of the image to be processed by performing feature enhancement processing on the image to be processed.
  • s 1 can represent a zoom parameter obtained through learning, and the zoom parameter can perform a zoom operation on different regions in the image, thereby achieving color enhancement of different regions in the image.
  • FIG. 14 is a schematic diagram of a processing flow of a hybrid enhancement unit proposed in an embodiment of the application (ie, a schematic diagram of color enhancement processing and brightness enhancement processing).
  • the processing procedure of the hybrid enhancement unit includes the following steps:
  • Step 1 Divide the multiple feature channels of the enhanced image features output by the Laplacian enhancement unit (that is, the enhanced image features output by the first neural network) into two branches, denoted as the first branch and the second branch .
  • Step 2 Perform a convolution operation on the first branch of the hybrid enhancement unit to obtain a confidence map, and then apply the confidence map to the input image to perform pixel-level scaling to obtain a scaling component.
  • the confidence map that is, the confidence image feature in the enhancement processing method shown in FIG. 10, is used to enhance the color of the input image.
  • the confidence map can represent a mapping relationship or a mapping function for performing color enhancement processing on an input image.
  • the zoom component is the color enhancement image feature in the enhancement processing method shown in FIG. 10.
  • the confidence map can correspond to the input image.
  • an element in the image feature of the confidence map can be used to indicate the zoom degree of the corresponding element in the image feature of the input image; by zooming different regions in the input image, Realize the color enhancement of the image to be input.
  • the confidence map may include several channels, which correspond to the channels of the input image.
  • applying the confidence map to the input image may refer to multiplying the confidence map and the input image, for example, multiplying the input image and the confidence map pixel by pixel.
  • Step 3 Perform a convolution operation through the second branch in the hybrid enhancement unit to obtain a generation component for illumination compensation.
  • the generating component is used to enhance the brightness of the input image; the generating component is the illumination compensation image feature in the enhancement processing method shown in FIG. 10.
  • Step 4 Perform fusion processing on the image features of the two branches to obtain an output image after image enhancement processing.
  • the N channels can be divided into two branches, where the M channel features are obtained by convolution Confidence map, the confidence map can be used to calculate the zoom component, M channels can correspond to the R, G, B three channels of the image; the other NM channels can be generated through convolution operation to generate components, and the generated components are used for illumination compensation. Contrast and brightness enhancement.
  • a convolution operation can be performed on a channel feature to obtain the illumination compensation image feature used for illumination enhancement processing.
  • the confidence image feature and the illumination compensation image feature of the input image can be obtained according to the enhanced image feature of the hybrid enhancement module and the input image
  • the hybrid enhancement module includes a first branch and a second branch.
  • the first branch is used to obtain the confidence image feature based on the enhanced image
  • the second branch is used to obtain the illumination compensation image feature based on the enhanced image feature
  • the confidence image feature can be used to represent the mapping relationship for color enhancement of the image to be processed (for example, mapping Function)
  • the illumination compensation image feature can be used to enhance the brightness of the input image
  • the output image is obtained according to the input image, the confidence image feature, and the illumination compensation image feature.
  • FIG. 15 is a schematic diagram of a processing flow of a hybrid enhancement unit provided by an embodiment of the present application.
  • the scale of the image to be processed and the confidence map can be the same, and different regions of the input image can be scaled to different degrees through the confidence map to achieve color enhancement; and then the overlay generation component is used to achieve illumination compensation and achieve contrast and brightness
  • the enhancement of the input image will finally get the output image after the feature enhancement, color enhancement and brightness enhancement of the input image.
  • the above confidence map may represent a mapping relationship or a mapping function for performing color enhancement processing on the image to be processed.
  • the confidence map can correspond to the image to be processed.
  • an element in the image feature of the confidence map can be used to indicate the zoom degree of the corresponding element in the image feature of the image to be processed; Scaling can realize the color enhancement of the image to be processed.
  • the confidence map used for color enhancement processing and the illumination compensation image feature used for brightness enhancement processing may be obtained through the enhanced image feature of the image to be processed, which is directly compared to the traditional zoom method.
  • the color and brightness of each pixel in the image to be processed are enhanced by a mapping function.
  • the confidence image feature and the illumination compensation image feature in the embodiments of the present application can not only perform color enhancement and brightness enhancement, but also realize the image to be processed The details are enhanced.
  • Table 1 is the quantitative performance evaluation result on the MIT-Adobe FiveK data set provided by the embodiment of the application.
  • Baseline represents the basic model architecture, that is, it can be a Unet model that does not include the above-mentioned hybrid enhancement unit (HEM) and the above-mentioned Laplacian enhancement unit (LEU); +LEU represents the image enhancement Unet that includes the above-mentioned Laplacian enhancement unit Model; +HEM represents the image enhancement Unet model including the above-mentioned hybrid enhancement unit; peak signal-to-noise ratio (PSNR) is usually used as a measurement method of signal reconstruction quality in image processing and other fields. The mean square error is defined.
  • HEM hybrid enhancement unit
  • LEU Laplacian enhancement unit
  • PSNR peak signal-to-noise ratio
  • Multi-scale structural similarity index can be used to measure the similarity of two images, and used to evaluate the quality of the output image processed by the algorithm.
  • the structural similarity index defines structural information as an attribute that reflects the structure of objects in the scene independent of brightness and contrast, and models distortion as a combination of three different factors: brightness, contrast, and structure. For example, use the mean as an estimate of brightness, standard deviation as an estimate of contrast, and covariance as a measure of structural similarity.
  • Time is used to represent the time for the model to perform image enhancement on the input image. Parameters can be used to describe the parameters included in the neural network and to evaluate the size of the model.
  • Table 2 is the quantitative performance evaluation results of different image enhancement models provided by the embodiments of the present application on the data set.
  • data set 1 can represent the MIT-Adobe FiveK data set
  • data set 2 can represent the DPED-iPhone data set
  • the tested models include: digital camera photo enhancer (Weakly Supervised Photo Enhancer for Digital camera, WESPE), context fusion network (context aggregation network, CAN), regional scale scaling global U-shaped network (range scaling global Unet, RSGUnet).
  • the PSNR and SSIM of the image enhancement model proposed in this application on the DPED-iPhone data set are not as good as the WESPE model, which is mainly caused by the non-pixel-level alignment of the DPED-iPhone data set.
  • the DPED-iPhone data set a pair of images obtained by using a SLR camera and a mobile phone in the same scene and shooting at the same angle at the same time. Because the sensors of the SLR camera and the mobile phone are different, the images captured by the SLR camera and the graphics captured by the camera are not sequentially Pixel alignment leads to relative deviations in HPEU during downsampling.
  • the image enhancement model (HPEU) proposed in the embodiment of this application has the highest PSNR and SSIM, and the visual effect is consistent with the true value ( Ground Truth) is closer.
  • Table 3 is the quantitative performance evaluation result of the MIT-Adobe FiveK data set by adding Guided Filter to the model provided by the embodiment of the application.
  • the tested models include: a trainable guided filter model (GDF), a high dynamic range network model (high dynamic range, HDR), and an image enhancement model (HPEU) provided in an embodiment of the present application with guided filtering added.
  • GDF trainable guided filter model
  • HDR high dynamic range network model
  • HPEU image enhancement model
  • Table 4 is the evaluation result of the image enhancement processing running time provided by the embodiment of the present application. It can be seen from the running time evaluation results shown in Table 4 that under the same processing conditions, the image enhancement model (HPEU) model proposed in the embodiment of the present application has the shortest running time, and the calculation efficiency of HPEU+Guided Filter is higher. After the Guided Filter is added to the HPEU model, the speed of HPEU+Guided Filter is faster than the GDF model and the HDR model, and the objective quantitative indicators are similar.
  • HPEU image enhancement model
  • Table 5 is the quantitative evaluation result based on the DPED-iphone data set after introducing the visual perception loss provided by the embodiment of the present application. Among them, it includes objective evaluation indicators PSNR, SSIM, and perceptual evaluation indicators (Perceptual Index).
  • Table 6 is the training of each network model based on the MIT-Adobe FiveK data set provided by the embodiments of this application, and the data set A (DPED-iphone), the data set B (DPED-Sony) and the data set C (DPED-Blackberry) are used as tests Data set, used to verify the generalization ability of the model.
  • FIG. 16 is a schematic diagram of a visual quality evaluation result provided by an embodiment of the present application. It should be noted that in FIG. 16, in order to be distinguished from the gray-scale image portion, the color image portion is indicated by hatching.
  • Fig. 16(a) represents the input image (for example, the image to be processed);
  • Fig. 16(b) represents the predicted output image obtained by using the basic model.
  • the basic model may not include the aforementioned hybrid enhancement unit (HEM) and the aforementioned The Unet model of the Laplacian enhancement unit (LEU);
  • Figure 16(c) shows the predicted output image obtained using the Unet model of the Laplacian enhancement unit;
  • Figure 16(d) shows the Unet model using the above-mentioned hybrid enhancement unit
  • Figure 16(e) represents the predicted output image obtained by using the model of the embodiment of the application (for example, the model shown in Figure 11 or Figure 12);
  • Figure 16(f) represents the input image Corresponding to the ground truth image (Ground Truth), the ground truth image can represent the sample enhanced image corresponding to the input image;
  • Figure 16 (e) represents the error graph 1, that is, the residual between the predicted output image output by the basic model and the true value image
  • Figure 16(j) represents the error Figure 4, that is, the predicted output image output by the model of the embodiment of the application is compared with The residual between the ground truth images.
  • FIG. 17 is a schematic flowchart of an image enhancement method provided by an embodiment of the present application.
  • the method 700 shown in FIG. 17 includes steps 710 to 740, and steps 710 to 740 are described in detail below.
  • Step 710 Detect the first operation used by the user to turn on the camera.
  • Step 720 In response to the first operation, display a shooting interface on the display screen, and display a shooting interface on the display screen.
  • the shooting interface includes a viewfinder frame, and the viewfinder frame includes a first image.
  • the user's shooting behavior may include a first operation of the user to turn on the camera; in response to the first operation, displaying a shooting interface on the display screen.
  • FIG. 18 shows a graphical user interface (GUI) of the mobile phone, and the GUI is the desktop 810 of the mobile phone.
  • GUI graphical user interface
  • the electronic device detects that the user clicks on the icon 820 of the camera application (application, APP) on the desktop 810, it can start the camera application and display another GUI as shown in (b) in FIG. 18, which can be called It is the shooting interface 830.
  • the shooting interface 830 may include a viewing frame 840. In the preview state, the preview image can be displayed in the viewfinder frame 840 in real time.
  • the color image portion is represented by oblique line filling.
  • a first image may be displayed in the view frame 840, and the first image is a color image.
  • the shooting interface may also include a control 850 for indicating the shooting mode, and other shooting controls.
  • the user's shooting behavior may include a first operation of the user to turn on the camera; in response to the first operation, displaying a shooting interface on the display screen.
  • the shooting interface may include a viewfinder frame. It is understandable that the size of the viewfinder frame may be different in the photo mode and the video mode.
  • the viewfinder frame may be the viewfinder frame in the photographing mode. In the video mode, the viewfinder frame can be the entire display screen.
  • the preview state that is, before the user turns on the camera and does not press the photo/video button, the preview image can be displayed in the viewfinder in real time.
  • the preview image may be a color image
  • the preview image may be an image displayed when the camera is set to an automatic photographing mode.
  • Step 730 It is detected that the second operation of the camera instructed by the user is detected.
  • the second operation instructing the first processing mode by the user may be detected.
  • the first processing mode may be a professional shooting mode (for example, an image enhancement shooting mode).
  • the shooting interface includes a shooting option 860.
  • the electronic device displays a shooting mode interface.
  • the mobile phone enters the professional shooting mode, for example, the mobile phone performs image enhancement shooting mode.
  • a second operation used by the user to instruct shooting may be detected, and the second operation is to shoot a long-distance object, or to shoot a tiny object, or when the shooting environment is poor. Used to indicate shooting operations.
  • the second operation for instructing shooting may be detected, that is, the operation shown in (a) and (b) of FIG. 19 may not be performed, and the operation shown in (a) and (b) of FIG. 19 may not be performed.
  • the second operation 870 of instructing shooting shown in (c) of FIG. 19 is performed.
  • the second operation used by the user to instruct the shooting behavior may include pressing the shooting button in the camera of the electronic device, or may include the user equipment instructing the electronic device to perform the shooting behavior through voice, or may also include other instructions from the user.
  • the device performs the shooting behavior.
  • Step 740 In response to the second operation, display a second image in the viewing frame, or save the second image in the electronic device, where the second image is the first image captured by the camera.
  • the enhanced image feature of an image is obtained by performing color enhancement processing and brightness enhancement processing on the first image
  • the enhanced image feature of the first image is obtained by performing feature enhancement processing on the first image through a neural network, so
  • the neural network includes N convolutional layers, and N is a positive integer.
  • the process of performing the image enhancement method on the first image may refer to the image enhancement method shown in FIG. 10, and the image enhancement model for executing the image enhancement method may adopt the model shown in FIG. 11.
  • the second image is displayed in the viewing frame in (d) of FIG. 19
  • the first image is displayed in the viewing frame in (c) of FIG. 19
  • the second image is displayed in the viewing frame of FIG. 19 (c).
  • the content of the first image is the same or substantially the same, but the quality of the second image is better than that of the first image.
  • the detail display of the second image is better than that of the first image; or, the brightness of the second image is better than that of the first image; or, the brightness of the second image is better than that of the first image.
  • the second image shown in (d) of FIG. 19 may not be displayed in the viewfinder, but the second image may be saved in the photo album of the electronic device.
  • the enhanced image feature of the first image is obtained by performing the feature enhancement processing on the first image according to a Laplacian enhancement algorithm.
  • the Laplacian enhancement algorithm is used to compare the i-th convolutional layer according to the residual characteristics of the i-th convolutional layer among the N convolutional layers.
  • the feature enhancement process is performed on the input image feature of the buildup layer to obtain the enhanced image feature of the i-th convolutional layer, wherein the residual feature represents the input image feature of the i-th convolutional layer and the first
  • the difference between the image features processed by the convolution operation in the i convolutional layers, the enhanced image feature of the i-th convolutional layer is the input image feature of the i+1th convolutional layer, and the input The image feature is obtained based on the first image, and i is a positive integer.
  • the enhanced image feature of the first image is an image feature output by the Nth convolutional layer among the N convolutional layers, and the enhanced image feature of the first image
  • the image features are obtained by the following equation:
  • L(F N ) represents the enhanced image feature of the Nth convolutional layer
  • F N represents the input image feature of the Nth convolutional layer
  • ⁇ represents the convolution of the Nth convolutional layer Core
  • s l represents the scaling parameter obtained through learning.
  • the output image is obtained according to the first image, the characteristic of the confidence image, and the characteristic of the illumination compensation image, and the characteristic of the confidence image and the characteristic of the illumination compensation image are obtained according to the characteristic of the illumination compensation image.
  • the enhanced image feature of the first image is obtained, the confidence image feature is used for color enhancement of the first image, and the illumination compensation image feature is used for brightness enhancement of the first image.
  • the output image is obtained by fusing the features of the color-enhanced image with the features of the illumination compensation image, and the feature of the color-enhanced image is based on the image of the first image
  • the feature is obtained by multiplying the feature of the confidence image
  • the feature of the confidence image is obtained by convolution operation on the enhanced image feature of the first image
  • the feature of the illumination compensation image is obtained by comparing the feature of the first image.
  • Enhanced image features are obtained by convolution operation.
  • FIG. 20 is a schematic block diagram of an image enhancement device provided by an embodiment of the present application. It should be understood that the image enhancement device 900 may execute the image enhancement method shown in FIG. 10.
  • the image enhancement device 900 includes: an acquisition unit 910 and a processing unit 920.
  • the acquisition unit 910 is configured to acquire an image to be processed;
  • the processing unit 920 is configured to perform feature enhancement processing on the image to be processed through a neural network to obtain enhanced image features of the image to be processed, and
  • the neural network includes N convolutional layers, where N is a positive integer; performing color enhancement processing and brightness enhancement processing on the image to be processed according to the enhanced image feature to obtain an output image.
  • the processing unit 920 is specifically configured to:
  • the feature enhancement processing is performed on the image to be processed by the Laplacian enhancement algorithm to obtain the enhanced image feature of the image to be processed.
  • the Laplacian enhancement algorithm is used to input the i-th convolutional layer according to the residual characteristics of the i-th convolutional layer in the N convolutional layers
  • the image feature is subjected to the feature enhancement processing to obtain the enhanced image feature of the i-th convolutional layer, wherein the residual feature represents the input image feature of the i-th convolutional layer and the i-th convolution
  • the difference between the image features processed by the convolution operation in the layer, the enhanced image feature of the i-th convolutional layer is the input image feature of the i+1th convolutional layer, the input image feature It is obtained from the image to be processed, and i is a positive integer less than or equal to N.
  • the enhanced image feature of the image to be processed is the image feature output by the Nth convolutional layer among the N convolutional layers, and the processing unit 920 is specifically configured to:
  • the enhanced image feature of the image to be processed is obtained by the equation:
  • L(F N ) represents the enhanced image feature of the Nth convolutional layer
  • F N represents the input image feature of the Nth convolutional layer
  • ⁇ represents the convolution of the Nth convolutional layer Core
  • s l represents the parameter obtained through learning.
  • the processing unit 920 is specifically configured to:
  • the confidence image feature and the illumination compensation image feature of the image to be processed are obtained, wherein the confidence image feature is used for color enhancement of the image to be processed, and the illumination compensation The image feature is used to enhance the brightness of the image to be processed;
  • the output image is obtained according to the image to be processed, the characteristic of the confidence image, and the characteristic of the illumination compensation image.
  • processing unit 920 is further configured to:
  • the color-enhanced image feature and the illumination compensation image feature are fused to obtain the output image.
  • the enhancement device 900 shown in FIG. 20 may also be used to perform the image enhancement methods shown in FIG. 17 to FIG. 19.
  • image enhancement device 900 is embodied in the form of a functional unit.
  • unit herein can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a "unit” can be a software program, a hardware circuit, or a combination of the two that realizes the above-mentioned functions.
  • the hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, and a processor for executing one or more software or firmware programs (such as a shared processor, a dedicated processor, or a group processor). Etc.) and memory, merged logic circuits and/or other suitable components that support the described functions.
  • the units of the examples described in the embodiments of the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • FIG. 21 is a schematic diagram of the hardware structure of an image enhancement device provided by an embodiment of the present application.
  • the image enhancement apparatus 1000 shown in FIG. 21 includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004.
  • the memory 1001, the processor 1002, and the communication interface 1003 implement communication connections between each other through the bus 1004.
  • the memory 1001 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 1001 may store a program.
  • the processor 1002 is configured to execute each step of the image enhancement method of the embodiment of the present application, for example, execute each of the steps shown in FIG. 10 to FIG. 15 Step, or execute each step shown in Figure 17 to Figure 19.
  • the image enhancement device shown in the embodiment of the present application may be a server, for example, it may be a server in the cloud, or may also be a chip configured in a server in the cloud; or, the image enhancement device shown in the embodiment of the present application
  • the device can be a smart terminal or a chip configured in the smart terminal.
  • the image enhancement method disclosed in the foregoing embodiments of the present application may be applied to the processor 1002 or implemented by the processor 1002.
  • the processor 1002 may be an integrated circuit chip with signal processing capabilities.
  • the steps of the above-mentioned image enhancement method can be completed by an integrated logic circuit of hardware in the processor 1002 or instructions in the form of software.
  • the processor 1002 may be a chip including the NPU shown in FIG. 8.
  • the aforementioned processor 1002 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor, a digital signal processor (DSP), or an application specific integrated circuit (application integrated circuit).
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory (RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory, or electrically erasable programmable memory, registers, etc. mature in the field Storage medium.
  • the storage medium is located in the memory 1001, and the processor 1002 reads the instructions in the memory 1001, and combines its hardware to complete the functions required by the units included in the image enhancement device shown in FIG. 20 in the implementation of this application, or execute the method of this application
  • the image enhancement method shown in FIG. 10 to FIG. 15 of the embodiment, or each step shown in FIG. 17 to FIG. 19 of the method embodiment of the present application is performed.
  • the communication interface 1003 uses a transceiver device such as but not limited to a transceiver to implement communication between the device 1000 and other devices or a communication network.
  • a transceiver device such as but not limited to a transceiver to implement communication between the device 1000 and other devices or a communication network.
  • the bus 1004 may include a path for transferring information between various components of the image enhancement device 1000 (for example, the memory 1001, the processor 1002, and the communication interface 1003).
  • image enhancement device 1000 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the image enhancement device 1000 may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned image enhancement device 1000 may further include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned image intensifying device 1000 may also only include the necessary devices for implementing the embodiments of the present application, and not necessarily all the devices shown in FIG. 21.
  • An embodiment of the present application also provides a chip, which includes a transceiver unit and a processing unit.
  • the transceiver unit may be an input/output circuit or a communication interface;
  • the processing unit is a processor, microprocessor, or integrated circuit integrated on the chip.
  • the chip can execute the image enhancement method in the above method embodiment.
  • the embodiment of the present application also provides a computer-readable storage medium on which an instruction is stored, and the image enhancement method in the foregoing method embodiment is executed when the instruction is executed.
  • the embodiments of the present application also provide a computer program product containing instructions that, when executed, execute the image enhancement method in the foregoing method embodiments.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • Part of the processor may also include non-volatile random access memory.
  • the processor can also store device type information.
  • the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor.
  • Part of the processor may also include non-volatile random access memory.
  • the processor can also store device type information.
  • the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional modules in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disks or optical disks and other media that can store program codes. .

Abstract

Un procédé et un appareil d'amélioration d'image dans le domaine de la vision artificielle dans le domaine de l'intelligence artificielle sont divulgués. Le procédé d'amélioration d'image comprend : l'acquisition d'une image à traiter ; la réalisation, au moyen d'un réseau de neurones artificiels, d'un traitement d'amélioration de caractéristiques sur l'image à traiter pour obtenir une caractéristique d'image améliorée de l'image à traiter, le réseau de neurones artificiels comprenant N couches de convolution, et N étant un nombre entier positif ; et la réalisation, en fonction de la caractéristique d'image améliorée, d'un traitement d'amélioration de couleur et d'un traitement d'amélioration de luminosité sur l'image à traiter afin d'obtenir une image de sortie. Au moyen de la solution technique de la présente invention, les performances, en termes des aspects des détails, de la couleur et de la luminosité, de l'image à traiter sont toutes améliorées, ce qui permet d'améliorer l'effet de traitement d'amélioration d'image.
PCT/CN2020/118721 2019-09-30 2020-09-29 Procédé et appareil d'amélioration d'image WO2021063341A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910943355.7 2019-09-30
CN201910943355.7A CN112581379A (zh) 2019-09-30 2019-09-30 图像增强方法以及装置

Publications (1)

Publication Number Publication Date
WO2021063341A1 true WO2021063341A1 (fr) 2021-04-08

Family

ID=75117268

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/118721 WO2021063341A1 (fr) 2019-09-30 2020-09-29 Procédé et appareil d'amélioration d'image

Country Status (2)

Country Link
CN (1) CN112581379A (fr)
WO (1) WO2021063341A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11468543B1 (en) 2021-08-27 2022-10-11 Hong Kong Applied Science and Technology Research Institute Company Limited Neural-network for raw low-light image enhancement
CN117612115A (zh) * 2024-01-24 2024-02-27 山东高速信息集团有限公司 一种基于高速公路的车辆识别方法
CN117612115B (zh) * 2024-01-24 2024-05-03 山东高速信息集团有限公司 一种基于高速公路的车辆识别方法

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113034407B (zh) * 2021-04-27 2022-07-05 深圳市慧鲤科技有限公司 图像处理方法及装置、电子设备和存储介质
CN113392804B (zh) * 2021-07-02 2022-08-16 昆明理工大学 一种基于多角度的交警目标数据集的场景化构建方法及系统
CN115601244B (zh) * 2021-07-07 2023-12-12 荣耀终端有限公司 图像处理方法、装置和电子设备
CN113852759B (zh) * 2021-09-24 2023-04-18 豪威科技(武汉)有限公司 图像增强方法及拍摄装置
CN116433800B (zh) * 2023-06-14 2023-10-20 中国科学技术大学 基于社交场景用户偏好与文本联合指导的图像生成方法
CN117575969A (zh) * 2023-10-31 2024-02-20 广州成至智能机器科技有限公司 一种红外图像画质增强方法、装置、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120155759A1 (en) * 2010-12-21 2012-06-21 Microsoft Corporation Establishing clusters of user preferences for image enhancement
CN109658345A (zh) * 2018-11-13 2019-04-19 建湖云飞数据科技有限公司 一种图像处理方法
CN109753978A (zh) * 2017-11-01 2019-05-14 腾讯科技（深圳）有限公司 图像分类方法、装置以及计算机可读存储介质
CN110163235A (zh) * 2018-10-11 2019-08-23 腾讯科技（深圳）有限公司 图像增强模型的训练、图像增强方法、装置和存储介质
CN110378848A (zh) * 2019-07-08 2019-10-25 中南大学 一种基于衍生图融合策略的图像去雾方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11030722B2 (en) * 2017-10-04 2021-06-08 Fotonation Limited System and method for estimating optimal parameters

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120155759A1 (en) * 2010-12-21 2012-06-21 Microsoft Corporation Establishing clusters of user preferences for image enhancement
CN109753978A (zh) * 2017-11-01 2019-05-14 腾讯科技（深圳）有限公司 图像分类方法、装置以及计算机可读存储介质
CN110163235A (zh) * 2018-10-11 2019-08-23 腾讯科技（深圳）有限公司 图像增强模型的训练、图像增强方法、装置和存储介质
CN109658345A (zh) * 2018-11-13 2019-04-19 建湖云飞数据科技有限公司 一种图像处理方法
CN110378848A (zh) * 2019-07-08 2019-10-25 中南大学 一种基于衍生图融合策略的图像去雾方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11468543B1 (en) 2021-08-27 2022-10-11 Hong Kong Applied Science and Technology Research Institute Company Limited Neural-network for raw low-light image enhancement
CN117612115A (zh) * 2024-01-24 2024-02-27 山东高速信息集团有限公司 一种基于高速公路的车辆识别方法
CN117612115B (zh) * 2024-01-24 2024-05-03 山东高速信息集团有限公司 一种基于高速公路的车辆识别方法

Also Published As

Publication number Publication date
CN112581379A (zh) 2021-03-30

Similar Documents

Publication Publication Date Title
WO2021063341A1 (fr) Procédé et appareil d'amélioration d'image
WO2021164731A1 (fr) Procédé d'amélioration d'image et appareil d'amélioration d'image
WO2021164234A1 (fr) Procédé de traitement d'image et dispositif de traitement d'image
US20210350168A1 (en) Image segmentation method and image processing apparatus
CN110532871B (zh) 图像处理的方法和装置
WO2021043273A1 (fr) Procédé et appareil d'amélioration d'image
WO2021043168A1 (fr) Procédé d'entraînement de réseau de ré-identification de personnes et procédé et appareil de ré-identification de personnes
WO2020192483A1 (fr) Procédé et dispositif d'affichage d'image
US20230214976A1 (en) Image fusion method and apparatus and training method and apparatus for image fusion model
WO2020177607A1 (fr) Procédé et appareil de débruitage d'image
CN112446380A (zh) 图像处理方法和装置
WO2021018106A1 (fr) Procédé de détection de piétons, appareil, support de stockage lisible par ordinateur, et puce
CN111667399A (zh) 风格迁移模型的训练方法、视频风格迁移的方法以及装置
US20230177641A1 (en) Neural network training method, image processing method, and apparatus
CN113066017B (zh) 一种图像增强方法、模型训练方法及设备
CN113076685A (zh) 图像重建模型的训练方法、图像重建方法及其装置
EP4006777A1 (fr) Procédé et dispositif de classification d'image
WO2021103731A1 (fr) Procédé de segmentation sémantique et procédé et appareil d'apprentissage de modèle
CN112529904A (zh) 图像语义分割方法、装置、计算机可读存储介质和芯片
WO2024002211A1 (fr) Procédé de traitement d'image et appareil associé
CN115861380B (zh) 雾天低照度场景下端到端无人机视觉目标跟踪方法及装置
CN114627034A (zh) 一种图像增强方法、图像增强模型的训练方法及相关设备
CN115512251A (zh) 基于双分支渐进式特征增强的无人机低照度目标跟踪方法
CN115131256A (zh) 图像处理模型、图像处理模型的训练方法及装置
WO2021042774A1 (fr) Procédé de récupération d'image, procédé d'entraînement de réseau de récupération d'image, dispositif, et support de stockage

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20872333

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20872333

Country of ref document: EP

Kind code of ref document: A1