CN115908088A

CN115908088A - Image processing method, model training method and related device

Info

Publication number: CN115908088A
Application number: CN202111168995.9A
Authority: CN
Inventors: 张恒
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-09-30
Filing date: 2021-09-30
Publication date: 2023-04-04

Abstract

The application discloses an image processing method, a model training method and a related device, wherein the method comprises the following steps: acquiring a first image characteristic of first original image data; acquiring a first parameter set; acquiring P groups of first image quality evaluation data, wherein P is a positive integer; inputting the first image feature, the first parameter set and the P groups of first image quality evaluation data into a first neural network model for operation to obtain a second parameter set: and carrying out image processing on the first original image data through the second parameter set by using an image processor to obtain a target image. By adopting the method and the device, the image processing effect can be improved, and the image processing effect can meet the visual aesthetic requirements of users.

Description

Image processing method, model training method and related device

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image processing method, a model training method, and a related apparatus.

Background

In the prior art, commercial imaging systems rely on hardware Image Signal Processing (ISP) procedures consisting of a series of processing blocks that reconstruct a color image from RAW sensor measurements based on its hyper-parameters. Since hardware ISP hyper-parameters have complex interactions with the output images, there is complex interaction with downstream applications that take these images. Conventionally, the hyper-parameters of the ISP are manually adjusted by an imaging expert alone, and there is no end-to-end target, so that the image processing effect is relatively single, and it is difficult to meet the visual aesthetic requirements of users.

Disclosure of Invention

The embodiment of the application provides an image processing method, a model training method and a related device, which can improve the image processing effect, and the image processing effect can meet the visual aesthetic requirements of users.

In a first aspect, an embodiment of the present application provides an image processing method, where the method includes:

acquiring a first image characteristic of first original image data;

acquiring a first parameter set;

acquiring P groups of first image quality evaluation data, wherein P is a positive integer;

inputting the first image feature, the first parameter set and the P groups of first image quality evaluation data into a first neural network model for operation to obtain a second parameter set:

and performing image processing on the first original image data through the second parameter set by using an image processor to obtain a target image.

In a second aspect, an embodiment of the present application provides a model training method, where the method includes:

acquiring a second image feature of second original image data for training;

acquiring a third parameter set for training;

obtaining Q groups of second image quality evaluation data for training, wherein Q is a positive integer;

inputting the second image characteristics, the third parameter set and the Q groups of second image quality evaluation data into a first neural network for operation to obtain a fourth parameter set;

processing the second original image data through the fourth parameter set by using an image processor to obtain a reference image;

performing image quality evaluation on the reference image to obtain Q groups of third image quality evaluation data;

and adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data until a training termination condition is reached.

In a third aspect, an embodiment of the present application provides an image processing apparatus, including: a first acquisition unit, a second acquisition unit, a third acquisition unit, a recommendation unit and an image processing unit, wherein,

the first acquisition unit is used for acquiring first image characteristics of first original image data;

the second acquisition unit is used for acquiring a first parameter set;

the third acquiring unit is configured to acquire P groups of first image quality evaluation data, where P is a positive integer;

the recommending unit is used for inputting the first image characteristic, the first parameter set and the P groups of first image quality evaluation data into a first neural network model for operation to obtain a second parameter set;

and the image processing unit is used for carrying out image processing on the first original image data through the second parameter set by using an image processor to obtain a target image.

In a fourth aspect, an embodiment of the present application provides a model training apparatus, including: an acquisition unit, a recommendation unit, a processing unit, an evaluation unit and an adjustment unit, wherein,

the acquisition unit is used for acquiring second image characteristics of second original image data used for training; acquiring a third parameter set for training; obtaining Q groups of second image quality evaluation data for training, wherein Q is a positive integer;

the recommending unit is used for inputting the second image characteristics, the third parameter set and the Q groups of second image quality evaluation data into a first preset neural network for operation to obtain a fourth parameter set;

the processing unit is used for processing the second original image data through the fourth parameter set by using an image processor to obtain a reference image;

the evaluation unit is used for evaluating the image quality of the reference image to obtain Q groups of third image quality evaluation data;

and the adjusting unit is used for adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data until a training termination condition is reached.

In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory for storing one or more programs and configured to be executed by the processor, the program including instructions for performing the steps in the method according to any one of the first or second aspects.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, where the computer program enables a computer to perform some or all of the steps described in the first aspect or the second aspect of the embodiment of the present application.

In a seventh aspect, this application embodiment provides a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, where the computer program is operable to cause a computer to perform some or all of the steps as described in the first aspect or the second aspect of this application embodiment. The computer program product may be a software installation package.

The embodiment of the application has the following beneficial effects:

it can be seen that, in the image processing method, the model training method, and the related apparatus described in the embodiments of the present application, the first image feature of the first original image data is obtained, the first parameter set is obtained, the P groups of first image quality evaluation data are obtained, where P is a positive integer, the first image feature, the first parameter set, and the P groups of first image quality evaluation data are input to the first neural network model for operation, the second parameter set is obtained, and the image processor performs image processing on the first original image data through the second parameter set to obtain the target image.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;

fig. 2 is a schematic diagram of a software structure of an electronic device according to an embodiment of the present application;

fig. 3A is a schematic flowchart of an image processing method according to an embodiment of the present application;

fig. 3B is a schematic flowchart of another image processing method provided in the embodiment of the present application;

FIG. 3C is a structural schematic diagram of a second neural network model provided by an embodiment of the present application;

fig. 3D is a schematic structural diagram of a first neural network model provided in an embodiment of the present application;

FIG. 3E is a schematic flow chart of a distillation algorithm provided by an embodiment of the present application;

fig. 3F is a schematic flowchart of ISP image processing provided by the embodiment of the present application;

FIG. 4 is a schematic flowchart of another image processing method according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

fig. 6 is a block diagram of functional units of an image processing apparatus according to an embodiment of the present application;

fig. 7 is a block diagram illustrating functional units of a model training apparatus according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

In order to better understand the scheme of the embodiments of the present application, the following first introduces the related terms and concepts that may be involved in the embodiments of the present application.

In the embodiment of the present application, the electronic device may include various devices having a communication function, for example, a smart phone, a vehicle-mounted device, a wearable device, a charging apparatus (such as a power bank), a smart watch, smart glasses, a wireless bluetooth headset, a computing device or other processing devices connected to a wireless modem, and various forms of User Equipment (UE), a Mobile Station (MS), a virtual reality/augmented reality device, a terminal device (terminal device), and the like, where the electronic device may also be a base Station or a server.

In a first section, the software and hardware operating environment of the technical solution disclosed in the present application is described as follows.

As shown, fig. 1 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a compass 190, a motor 191, a pointer 192, a camera 193, a display screen 194, a Subscriber Identification Module (SIM) card interface 195, and the like.

It is to be understood that the illustrated structure of the embodiment of the present application does not specifically limit the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as: the processor 110 may include an application processor AP, a modem processor, a graphics processor GPU, an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural network processor NPU, among others. Wherein the different processing units may be separate components or may be integrated in one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. The controller can generate an operation control signal according to the instruction operation code and the time sequence signal to complete the control of instruction fetching and instruction execution. In other embodiments, a memory may also be provided in processor 110 for storing instructions and data. Illustratively, the memory in the processor 110 may be a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from memory. This avoids repeated accesses and reduces the latency of the processor 110, thereby increasing the efficiency with which the electronic device 100 processes data or executes instructions. The processor may also include an image processor, which may be an image Pre-processor (Pre-ISP), which may be understood as a simplified ISP, which may also perform some image processing operations.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit audio source (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose-output (GPIO) interface, a SIM card interface, and/or a USB interface. The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device. The USB interface 130 may also be used to connect to a headset to play audio through the headset.

It should be understood that the connection relationship between the modules illustrated in the embodiment of the present application is only an exemplary illustration, and does not limit the structure of the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners or a combination of multiple interface connection manners in the above embodiments.

The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be used to monitor parameters such as battery capacity, battery cycle count, battery state of health (leakage, impedance), etc. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be disposed in the same device.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution including wireless communication of 2G/3G/4G/5G/6G, etc. applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The mobile communication module 150 may receive the electromagnetic wave from the antenna 1, filter, amplify, etc. the received electromagnetic wave, and transmit the electromagnetic wave to the modem processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modem processor, and convert the signal into electromagnetic wave through the antenna 1 to radiate the electromagnetic wave. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the same device as at least some of the modules of the processor 110.

The wireless communication module 160 may provide a solution for wireless communication applied to the electronic device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (blue tooth, BT), global Navigation Satellite System (GNSS), frequency Modulation (FM), near Field Communication (NFC), infrared (IR), and the like. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and convert the signal into electromagnetic waves through the antenna 2 to radiate the electromagnetic waves.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. The display panel may be an LCD, an OLED, an active-matrix organic light emitting diode (AMOLED), a Flexible Light Emitting Diode (FLED), a mini light emitting diode (mini-led), a Micro led, a Micro-oeled, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the electronic device 100 may include 1 or more display screens 194.

The electronic device 100 may implement a photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV and other formats. In some embodiments, electronic device 100 may include 1 or more cameras 193.

The digital signal processor is used for processing digital signals, and can process digital image signals and other digital signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to perform fourier transform or the like on the frequency bin energy.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 may play or record video in a variety of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor that processes input information quickly by using a biological neural network structure, for example, by using a transfer mode between neurons of a human brain, and can also learn by itself continuously. Applications such as intelligent recognition of the electronic device 100 can be realized through the NPU, for example: image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the memory capability of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.

Internal memory 121 may be used to store one or more computer programs, which include instructions. The processor 110 may execute the above-mentioned instructions stored in the internal memory 121, so as to enable the electronic device 100 to execute the method for displaying page elements provided in some embodiments of the present application, and various applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Wherein, the storage program area can store an operating system; the storage program area may also store one or more applications (e.g., gallery, contacts, etc.), and the like. The storage data area may store data (e.g., photos, contacts, etc.) created during use of the electronic device 100, and the like. Further, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage units, flash memory units, universal Flash Storage (UFS), and the like. In some embodiments, the processor 110 may cause the electronic device 100 to execute the method for displaying page elements provided in the embodiments of the present application and other applications and data processing by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor 110. The electronic device 100 may implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. Such as music playing, recording, etc.

The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

The pressure sensor 180A is used for sensing a pressure signal, and converting the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The pressure sensor 180A can be of a wide variety, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor may be a sensor comprising at least two parallel plates having an electrically conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the strength of the pressure from the change in capacitance. When a touch operation is applied to the display screen 194, the electronic apparatus 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic apparatus 100 may also calculate the touched position from the detection signal of the pressure sensor 180A. In some embodiments, the touch operations that are applied to the same touch position but different touch operation intensities may correspond to different operation instructions. For example: and when the touch operation with the touch operation intensity smaller than the first pressure threshold value acts on the short message application icon, executing an instruction for viewing the short message. And when the touch operation with the touch operation intensity larger than or equal to the first pressure threshold value acts on the short message application icon, executing an instruction of newly building the short message.

The gyro sensor 180B may be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of electronic device 100 about three axes (i.e., X, Y and the Z axis) may be determined by gyroscope sensor 180B. The gyro sensor 180B may be used for photographing anti-shake. For example, when the shutter is pressed, the gyro sensor 180B detects a shake angle of the electronic device 100, calculates a distance to be compensated for by the lens module according to the shake angle, and allows the lens to counteract the shake of the electronic device 100 through a reverse movement, thereby achieving anti-shake. The gyroscope sensor 180B may also be used for navigation, somatosensory gaming scenes.

The acceleration sensor 180E may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. The method can also be used for recognizing the posture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

The ambient light sensor 180L is used to sense the ambient light level. Electronic device 100 may adaptively adjust the brightness of display screen 194 based on the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust the white balance when taking a picture. The ambient light sensor 180L may also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

The fingerprint sensor 180H is used to collect a fingerprint. The electronic device 100 can utilize the collected fingerprint characteristics to unlock the fingerprint, access the application lock, photograph the fingerprint, answer an incoming call with the fingerprint, and so on.

The temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 implements a temperature processing strategy using the temperature detected by temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 performs a reduction in performance of a processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In other embodiments, the electronic device 100 heats the battery 142 when the temperature is below another threshold to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than a further threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown due to low temperature.

The touch sensor 180K is also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type. Visual output associated with the touch operation may be provided through the display screen 194. In other embodiments, the touch sensor 180K may be disposed on a surface of the electronic device 100, different from the position of the display screen 194.

By way of example, fig. 2 shows a block diagram of a software structure of the electronic device 100. The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, an application layer, an application framework layer, an Android runtime (Android runtime) and system library, and a kernel layer from top to bottom. The application layer may include a series of application packages.

As shown in fig. 2, the application layer may include applications such as camera, gallery, calendar, phone call, map, navigation, WLAN, bluetooth, music, video, short message, etc.

The application framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions.

As shown in FIG. 2, the application framework layers may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and the like.

The window manager is used for managing window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and answered, browsing history and bookmarks, phone books, etc.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 100. Such as management of call status (including on, off, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

The notification manager enables the application to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a brief dwell, and does not require user interaction. Such as a notification manager used to inform download completion, message alerts, etc. The notification manager may also be a notification that appears in the form of a chart or scrollbar text in a status bar at the top of the system, such as a notification of a running application in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, sounding a prompt tone, vibrating the electronic device, flashing an indicator light, etc.

The Android Runtime comprises a core library and a virtual machine. The Android Runtime is responsible for scheduling and managing an Android system.

The core library comprises two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. And executing java files of the application program layer and the application program framework layer into a binary file by the virtual machine. The virtual machine is used for performing the functions of object life cycle management, stack management, thread management, safety and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface managers (surface managers), media libraries (media libraries), three-dimensional graphics processing libraries (e.g., openGL ES), 2D graphics engines (e.g., SGL), and the like.

The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

Based on the electronic device described in fig. 1 or fig. 2, the following functions can be implemented by the electronic device:

acquiring a first image characteristic of first original image data;

acquiring a first parameter set;

inputting the first image characteristic, the first parameter set and the P groups of first image quality evaluation data into a first neural network model for operation to obtain a second parameter set;

and carrying out image processing on the first original image data through the second parameter set by using an image processor to obtain a target image.

It can be seen that, in the electronic device described in this embodiment of the present application, a first image feature of first original image data is obtained, a first parameter set is obtained, P groups of first image quality evaluation data are obtained, where P is a positive integer, the first image feature, the first parameter set, and the P groups of first image quality evaluation data are input to a first neural network model for operation, a second parameter set is obtained, an image processor performs image processing on the first original image data through the second parameter set, and a target image is obtained.

In the second section, the image processing method, the model training method and the device disclosed in the embodiments of the present application are described as follows.

Referring to fig. 3A, fig. 3A is a schematic flowchart of an image processing method provided in an embodiment of the present application, and is applied to an electronic device, where as shown in the diagram, the image processing method includes:

301. a first image feature of the first raw image data is acquired.

In this embodiment of the application, the first image feature may include at least one of the following: feature points, feature vectors, feature lines, and the like, which are not limited herein. The first raw image data may be understood as image data that has been acquired by an image sensor and that has not been processed by an image processor.

In specific implementation, the electronic device may capture images through a camera to obtain first original image data, and then perform feature extraction on the first original image data to obtain a first image feature.

Optionally, in step 301, the obtaining of the first image feature of the first original image data may include the following steps:

and inputting the first original image data into a second neural network model to obtain the first image feature.

Wherein the second neural network model may include at least one of: convolutional neural network models, recurrent neural network models, fully-connected neural network models, and the like, without limitation. The second neural network model may be used to implement feature extraction, and specifically, the first raw image data may be input to the second neural network model to obtain the first image feature.

302. A first set of parameters is obtained.

Wherein, the first parameter set may include at least one hyper-parameter, and the hyper-parameter may be any one of the following: a hyper-parameter of a color of an image, a hyper-parameter of an edge sharpness of an image, a hyper-parameter of a noise of an image, etc., which are not limited herein.

In a specific implementation, the first parameter set may be implemented by an ISP initial parameter, or may also be implemented by a parameter combination corresponding to more than 80% of the effect obtained by fast debugging by an effect parameter adjusting engineer.

Optionally, in step 302, the obtaining the first parameter set may include the following steps:

21. acquiring target shooting parameters;

22. determining a target parameter identification set corresponding to the target shooting parameters;

23. and acquiring the first parameter set according to the target parameter identification set.

In this embodiment of the present application, the target shooting parameter may include at least one of the following: sensitivity, exposure duration, ambient light brightness, ambient temperature, ambient humidity, intensity before magnetic field disturbance, etc., without limitation. The target parameter identification set may comprise at least one parameter identification, the parameter identification being indicative of a parameter type. The electronic equipment can pre-store the mapping relation between the shooting parameters and the parameter identification set, further determine a target parameter identification set corresponding to the target shooting parameters based on the mapping relation, and acquire corresponding parameters based on the target parameter identification set to obtain a first parameter set.

303. And acquiring P groups of first image quality evaluation data, wherein P is a positive integer.

In a specific implementation, the first parameter set may correspond to the P group of first image quality evaluation data, and the first parameter set may not correspond to the P group of first image quality evaluation data. In the case where the first parameter set and the P groups of first image quality evaluation data may correspond, specifically, for example, each hyper-parameter in the first parameter set may correspond to at least one group of first image quality evaluation data. Each set of the first image quality evaluation data is an evaluation result evaluated by using at least one image quality evaluation index, and the image quality evaluation index may include at least one of: signal-to-noise ratio, sharpness, edge preservation, average gradient, entropy, etc., and is not limited herein. Of course, the set of image quality evaluation data may be an evaluation result obtained by evaluating with a single image quality evaluation index or may be an evaluation result obtained by evaluating with a combination of image quality evaluation indexes. The image quality evaluation index may include a subjective image quality evaluation index, or the image quality evaluation index may include an objective image quality evaluation index, or the image quality evaluation index may include a subjective image quality evaluation index and an objective image quality evaluation index. The P groups of first image quality evaluation data may be empirical data or may be image quality evaluation data obtained after image quality evaluation is performed last time or last N times, where N is a positive integer.

In the embodiment of the present application, for image quality evaluation, in an actual test, 20 algorithms and corresponding tools for evaluating image quality may be adopted, for example, dimensions such as color, noise, sharpness, and the like may be evaluated.

304. And inputting the first image characteristic, the first parameter set and the P groups of first image quality evaluation data into a first neural network model for operation to obtain a second parameter set.

Wherein the first neural network model may include at least one of: convolutional neural network models, recurrent neural network models, fully-connected neural network models, and the like, without limitation. In specific implementation, the electronic device may input the first image feature, the first parameter set, and the P-group image quality evaluation data to the first neural network model for operation to obtain a second parameter set, where the second parameter set takes subjective or objective evaluation factors into account, so that the processing effect depth of these parameters is suitable for the visual aesthetic requirement of the user in the image processing process.

305. And carrying out image processing on the first original image data through the second parameter set by using an image processor to obtain a target image.

In the specific implementation, the electronic device can perform image processing on the first original image data based on the second parameter set, and then can obtain a target image, and due to the effect of the image imaging process, the parameters obtained under the image evaluation and the subjective visual quality evaluation are comprehensively considered for image processing, so that a high-quality image which better meets the image quality evaluation requirement can be obtained, and the visual experience for the image quality evaluation is better met.

Optionally, before step 301, the following steps may be further included:

a1, acquiring a second image feature of second original image data for training;

a2, acquiring a third parameter set for training;

a3, Q groups of second image quality evaluation data used for training are obtained, wherein Q is a positive integer;

a4, inputting the second image characteristics, the third parameter set and the Q groups of second image quality evaluation data into a first neural network model for operation to obtain a fourth parameter set;

a5, carrying out image processing on the second original image data through the fourth parameter set by using the image processor to obtain a reference image;

a6, carrying out image quality evaluation on the reference image to obtain Q groups of third image quality evaluation data;

and A7, adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data until a training termination condition is reached.

Wherein, the above steps A1 to A5 can refer to the above corresponding steps, which are not described herein again. Q is a positive integer greater than or equal to P, and the image quality evaluation indexes corresponding to the Q sets of second image quality evaluation data include the image quality evaluation indexes of the P sets of second image quality evaluation data. The training termination condition can be set by the user or the system defaults, and the training termination condition can include any one of the following conditions: the preset training round, the convergence of the loss function and the set threshold of the loss function are reached.

Optionally, in the step A7, adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data may include the following steps:

a71, constructing a loss function according to the Q groups of third image quality evaluation data;

and A72, iteratively training the first neural network model by using the loss function until the training termination condition is reached.

Wherein the content of the first and second substances,

in a specific implementation, a loss function may be constructed according to Q groups of third image quality evaluation data and Q groups of second image quality evaluation data, specifically as follows:

where δ =1,y and y' represent the generated quality index and the input index combination, respectively.

Optionally, the image processor is an image processor of physical hardware or a simulator corresponding to the image processor.

The image processor may be an image processor of physical hardware, that is, the image processor processes the original image data by using the second parameter set, and may output the processed image. The image processor can also be a simulator corresponding to the image processor, the simulator of the hardware can use the parameter combination same as that of the hardware, but the simulator can take effect without restarting, and the final image effect is equivalent to the image effect generated by the hardware, so the data quality is not influenced.

In a specific implementation, taking a mobile phone as an example, a corresponding data set can be constructed by manipulating ISP parameter combinations. The mobile phone camera process can be simply summarized as mobile phone preprocessing output original image data (RAW image), ISP hardware receives the RAW image and combines a group of self-owned parameter combinations to generate a final effect image, namely an output image (target image), wherein the format of the output image can be png format or jpeg format.

Specifically, as shown in fig. 3B, first, a RAW image may be captured, then, the parameters of the ISP are synchronously adjusted, after the parameters take effect, a second parameter set is obtained, and the original image data and the second parameter set are input to the ISP hardware to obtain an effect image; the effect maps can then be submitted to an image evaluation tool for image quality evaluation, which can evaluate image quality based on 20 dimensions of noise, color, sharpness, etc. Thus, the data pairs that can be obtained are: (ISP parameter combinations, images, image evaluation dimensions x 20).

Furthermore, more than 1000 groups of data pairs can be obtained by changing parameter combinations, and considering that the new parameters are realized on the mobile phone and need to be restarted, the efficiency is low, and the data pairs can be generated by synchronously using simulators of ISP hardware. The simulator of the ISP hardware uses the same parameter combination as the hardware, but can take effect without restarting, and the final image effect is equivalent to the image effect generated by the hardware, so the data quality is not influenced.

In a specific implementation, the first neural network model may also be referred to as a parameter recommendation model, and the parameter recommendation model is implemented through two stages, namely a first stage (training stage) and a deep learning-based model is constructed. Considering that hardware (ISP hardware) is in a ring, training and obtaining a model with better effect on verification data; as shown in fig. 3B, both ISP hardware and image profiling tools may participate in model training. And in the second stage (application stage), only the trained model is used for parameter recommendation, namely, a demand-based image evaluation (IQA) index expected to be obtained, an initial parameter combination and a shot RAW image are input, and the model can automatically recommend corresponding ISP hardware parameters.

And the recommendation algorithm uses the used data sets to construct corresponding data sets by manipulating the ISP parameter combinations. The mobile phone camera process can be simply summarized as mobile phone preprocessing output RAW image, ISP hardware receives RAW image, and combines a group of parameters to generate final effect image, generally in png or jpeg format. Firstly, shooting a RAW image, then synchronously adjusting parameters of an ISP (Internet service provider), and inputting the image into ISP hardware after the parameters take effect to obtain an effect image; the effect map is then submitted to an image evaluation tool that can evaluate image quality based on 20 dimensions of noise, color, sharpness, etc. Thus, the data pairs that may be obtained may be: (ISP parameter combinations, images, image evaluation dimensions x 20).

Further, more than 1000 such data pairs may be obtained by transformation parameter combination. Considering that the new parameters are realized on the mobile phone, which needs to be restarted and is low in efficiency, the simulator of the ISP hardware is synchronously used for generating the data pairs. The simulator of the ISP hardware uses the same parameter combination as the hardware, but can take effect without restarting, and the final image effect is equivalent to the image effect generated by the hardware, so the data quality is not influenced.

A conventional convolutional neural network can be used as a model framework of the scheme for the first neural network model, but in actual operation, a dense neural network (MLP) can also be used as a model for construction. For the input and output of the first neural network model, as shown in fig. 3B, it is composed of three parts: a first set of parameters, i.e. an initial parameter combination; a first image feature, i.e., an image feature obtained based on the RAW map; p sets of image quality evaluation data. The three portions of data may be superimposed into a single input into the network. The initial parameter combination can be realized by the initial parameters of the ISP, and the parameter combination corresponding to more than 80% of the effect obtained by the rapid debugging of an effect parameter adjusting engineer can also be used; the first image features can be obtained through a second neural network model based on the RAW graph, and the second neural network model can be obtained by using the most advanced deep learning framework at present, such as EfficientNet-B0 (shown in fig. 3C). Wherein, fig. 3C is a schematic diagram of the EfficientNet-B0 network. Conv3 × 3 represents a convolution kernel of size 3*3, and MBConv1 and MBConv6 represent 1 and 6 superimposed mobile inverted block convolution layers, respectively.

In general, the features of the last or next to last layer of the skeleton portion of the network may be used. The P groups of image quality evaluation data can be obtained by evaluating the image quality of the effect map by using a pre-prepared image quality evaluation index. The dimension of the initial parameter combination is 1*N, the dimension of the image feature is 1*N, the dimension of the image quality evaluation index is 1 × 20, superposition is performed along the first dimension, a broadcast (broadcast) strategy is used for data with different lengths in the second dimension to obtain 3*N-dimensional data, and finally the data can be deformed into 3 × sqrt (N). The output of the first neural network model may use a combination of ISP parameters corresponding to the image evaluation dimension, which may be derived from the data pairs.

A LeNet-5 network can be used for the first neural network model as a backbone network with the last two layers of dimensions changed to 120 and N to learn useful features and ultimately output ISP parameter combinations. As shown in fig. 3D, fig. 3D is a LeNet-5 network architecture. Wherein, the constraint represents a convolution kernel, the feature maps represent a feature map obtained after transformation, the sub-sampling refers to maximum pooling operation, and the full connection represents a full connection layer. In the embodiment of the present application, the last two layers may be replaced with F6:120 and output: n, and converting the last output layer into sigmoid layer output.

In the training phase, an automatic supervision algorithm scheme based on deep learning and hardware-in-the-loop can be constructed. In the embodiment of the application, the parameter recommendation algorithm can be realized by a deep learning framework, and the training of the framework can be divided into two stages, namely initial training and model compression, so that the final output model is limited in calculation force requirement and is more suitable for deployment; the calculation of the loss function needs to be realized by hardware effect generation and quality evaluation, the hardware effect generation is realized by ISP hardware or a corresponding ISP simulator, after a picture is generated, each image quality index is obtained by an image evaluation tool, and finally the loss function is calculated by distance calculation on the input index combination and the generated quality index.

For initial training of the model, the prepared data pairs can be input into the network for training according to the data transformation method described in (1) and (2). And the convolutional network is trained by using a PyTorch framework, so that the model is convenient and rapid to iterate and deploy. The trained batch size was 4, and the data was subjected to amplification processing with the upper and lower sides and the left and right sides reversed, with the probability of amplification set to 0.5. The optimizer for training can be selected to be SGD with momentum, the learning rate is set to 1e-3, and the learning strategy is cosine _ connecting _ policy. For example, 100 epochs can be trained, taking no more than 30 minutes.

The loss function is formulated as follows:

/>

Work on model compression. The LeNet-5 based deep learning model is not large (less than 10 MB) per se, is light and convenient to deploy in a server, and does not require GPU to be used by model reasoning. On a platform with limited calculation capacity, such as a mobile phone, model compression, such as knowledge distillation, can be selected to further compress the size of the model, namely, the first neural network model can be compressed through knowledge distillation. For example, a 3-layer MLP distillation parameter recommendation model, structure and distillation scheme can be used as shown in FIG. 3E. As shown in fig. 3E, which is the basic flow scheme of the knowledge distillation. The teacher model (teacher model) is a parameter recommendation model obtained through training, the student model (student model) is a model needing distillation, the softmax layer represents softmax parameters of the output layer, and the soft labels (soft labels) represent outputs when maximization processing is not performed. Loss functions include distillation loss (distillation loss) and student loss (student loss); the hard label (hard label) represents the actual parameter that is known.

In addition, in the generation of the hardware effect, the new ISP parameters which can be output can be sent to the ISP hardware or ISP corresponding simulator in combination with the prepared RAW map to generate a new effect map. The schematic diagram of the ISP hardware is shown in fig. 3F, and fig. 3F is the basic flow of the ISP. The RAW image output by the BPC (band pixel correction) can be retained, and after the parameters of the following series of modules are adjusted and validated, the RAW image is sent to a demosaic module to obtain the final output.

In a particular application, for example, the user need only provide: the existing parameter combination, the original image data and the expected image index can be inferred by the parameter recommendation model, and the corresponding optimized parameter combination can be automatically generated. The optimized parameter combination can be used as new input to be provided for ISP hardware or a corresponding ISP simulator to generate a new image, and finally the new image is provided for an image evaluation tool or an effect call participant to judge whether the requirements are met.

In addition, due to model compression, the debugging time of one group of parameters can be shortened from 1 week to 1 day, and of course, multiple groups of parameters can be provided for engineers to select to continue fine tuning, develop own ISP parameter adjusting algorithm and popularize to other hardware bodies required by parameter adjustment. In the embodiment of the application, the model can be optimized by using an automatic supervision method, the IQA index is used as input and supervision information, the thinking of a user is better met, and the hardware is used at one end of the loss function calculation in a ring, so that the algorithm iteration is more efficiently completed.

Certainly, in practical application, the initial parameter combination can be cancelled at the input side, so that the model training and deployment are simpler and more convenient; the input side can increase the parameter field that needs focus attention according to the user's demand, can restrain the parameter of output, promotes the optimization effect. In addition, a plurality of models can be used, for example, the models comprise a swarm algorithm and an iterative optimization algorithm, and the models are assembled, so that the parameter recommendation effect is further improved.

It can be seen that, in the image processing method described in the embodiment of the present application, the first image feature of the first original image data is obtained, the first parameter set is obtained, P groups of first image quality evaluation data are obtained, where P is a positive integer, the first image feature, the first parameter set, and the P groups of first image quality evaluation data are input to the first neural network model for operation, the second parameter set is obtained, and the image processor performs image processing on the first original image data through the second parameter set to obtain the target image.

Referring to fig. 4, fig. 4 is a schematic flowchart of an image processing method applied to an electronic device according to an embodiment of the present application, where as shown in the figure, the image processing method includes:

401. second image features of second raw image data for training are obtained.

In this embodiment, the second image feature may include at least one of the following: feature points, feature vectors, feature lines, and the like, which are not limited herein. The second raw image data may be understood as image data which has been acquired by the image sensor and which has not been processed by the image processor.

In specific implementation, the electronic device may capture images through a camera to obtain second original image data, and then perform feature extraction on the second original image data to obtain second image features.

Optionally, in step 401, the obtaining of the second image feature of the second original image data may include the following steps:

and inputting the second original image data into a second neural network model to obtain the second image feature.

Wherein the second neural network model may include at least one of: convolutional neural network models, recurrent neural network models, fully-connected neural network models, and the like, without limitation. The second neural network model may be used to implement feature extraction, and specifically, the second raw image data may be input to the second neural network model to obtain the first image feature.

402. A third set of parameters is obtained for training.

Wherein the third parameter set may include at least one hyper-parameter, and the hyper-parameter may be any one of: a hyper-parameter of a color of an image, a hyper-parameter of an edge sharpness of an image, a hyper-parameter of a noise of an image, etc., and is not limited herein.

In a specific implementation, the third parameter set may be implemented by an ISP initial parameter, or may be implemented by a parameter combination corresponding to more than 80% of the effect obtained by rapid debugging by an effect parameter adjusting engineer.

Optionally, in the step 402, the obtaining the third parameter set may include the following steps:

21. acquiring target shooting parameters;

23. and acquiring the third parameter set according to the target parameter identification set.

In this embodiment of the present application, the target shooting parameter may include at least one of the following: sensitivity, exposure duration, ambient light brightness, ambient temperature, ambient humidity, intensity before magnetic field disturbance, etc., without limitation. The target parameter identification set may comprise at least one parameter identification, the parameter identification being indicative of a parameter type. The electronic equipment can pre-store the mapping relation between the shooting parameters and the parameter identification sets, further determine the target parameter identification sets corresponding to the target shooting parameters based on the mapping relation, and then obtain corresponding parameters based on the target parameter identification sets to obtain third parameter sets.

403. And acquiring Q groups of second image quality evaluation data for training, wherein Q is a positive integer.

In a specific implementation, the third parameter set may correspond to the Q groups of image quality evaluation data, or the third parameter set may not correspond to the P groups of image quality evaluation data. When the third parameter set and the Q sets of image quality evaluation data may correspond, specifically, for example, each hyper-parameter in the third parameter set may correspond to at least one set of image quality evaluation data. Each set of image quality evaluation data is an evaluation result evaluated by at least one image quality evaluation index, and the image quality evaluation index may include at least one of the following: signal-to-noise ratio, sharpness, edge preservation, average gradient, entropy, etc., and is not limited herein.

404. And inputting the second image characteristic, the third parameter set and the Q groups of second image quality evaluation data into a first neural network for operation to obtain a fourth parameter set.

Wherein the first neural network model may include at least one of: convolutional neural network models, recurrent neural network models, fully-connected neural network models, and the like, without limitation. In specific implementation, the electronic device may input the second image feature, the third parameter set, and Q groups of second image quality evaluation data to the first neural network model for operation to obtain a fourth parameter set, where the fourth parameter set takes subjective or objective evaluation factors into account, so that in an image processing process, a processing effect depth of the parameters is suitable for a visual aesthetic requirement of a user.

405. And processing the second original image data through the fourth parameter set by using an image processor to obtain a reference image.

In the specific implementation, the electronic device can perform image processing on the second original image data based on the fourth parameter set, and then can obtain a target image, and due to the effect of the image imaging process, the parameters obtained under the image evaluation and the subjective visual quality evaluation are comprehensively considered for image processing, so that a high-quality image which better meets the image quality evaluation requirement can be obtained, and the visual experience for the image quality evaluation is better met.

406. And carrying out image quality evaluation on the reference image to obtain Q groups of third image quality evaluation data.

Wherein the Q groups of third image quality evaluation data correspond one-to-one to the Q groups of second image quality evaluation data. Each set of the third image quality evaluation data may correspond to a single image quality evaluation index or a combined image quality evaluation index. The combined image quality evaluation index can be obtained by combining at least two image quality evaluation indexes, and in the image evaluation process, each image quality evaluation index can be evaluated respectively to obtain at least two evaluation results, and the two evaluation results are subjected to weighting operation to obtain final image quality evaluation data.

407. And adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data until a training termination condition is reached.

In a specific implementation, a loss function may be constructed according to differences between the Q sets of third image quality evaluation data and the Q sets of second image quality evaluation data, and model parameters of the first neural network model may be adjusted based on the loss function until a training termination condition is reached. The training termination condition can be set by the user or defaulted by the system, and the training termination condition can include any one of the following conditions: and the preset training round is reached, the loss function is converged, and the loss function reaches the set threshold.

The image processor may be an image processor of physical hardware, that is, the image processor processes the second original image data by using the fourth parameter set, and may output a processed image, that is, a reference image. The image processor can also be a simulator corresponding to the image processor, the simulator of the hardware can use the parameter combination same as that of the hardware, but the simulator can take effect without restarting, and the final image effect is equivalent to the image effect generated by the hardware, so the data quality is not influenced.

Optionally, in step 407, adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data may include the following steps:

71. constructing a loss function according to the Q groups of third image quality evaluation data;

72. iteratively training the first neural network model using the loss function until the training termination condition is reached.

It can be seen that, in the model training method described in the embodiment of the present application, the second image feature of the second original image data is obtained, the third parameter set used for training is obtained, Q sets of second image quality evaluation data are obtained, Q is a positive integer, the second image feature, the third parameter set, and Q sets of second image quality evaluation data are input to the first neural network model for operation, a fourth parameter set is obtained, the second original image data is processed by the image processor through the fourth parameter set, a reference image is obtained, image quality evaluation is performed on the reference image, Q sets of third image quality evaluation data are obtained, the model parameters of the first neural network model are adjusted according to the Q sets of third image quality evaluation data until a training termination condition is reached, since image evaluation and subjective visual quality evaluation are considered during parameter recommendation model training, and parameter acquisition is realized through an evaluation result, thus, the optimized parameters are consistent with the visual aesthetic sense of the user in final effect evaluation, and further, the image processing effect can be improved, and the image processing effect can meet the visual sense requirement of the aesthetic sense of the user.

Consistent with the foregoing embodiment, please refer to fig. 5, where fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application, and as shown in the drawing, the electronic device includes a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and in an embodiment of the present application, the programs include instructions for performing the following steps:

acquiring a first image characteristic of first original image data;

acquiring a first parameter set;

Optionally, in the aspect of acquiring the first image feature of the first raw image data, the program includes instructions for performing the following steps:

and inputting the first original image data into a second neural network model to obtain the first image characteristic.

Optionally, the program further includes instructions for performing the following steps:

acquiring a second image feature of second original image data for training;

acquiring a third parameter set for training;

q groups of second image quality evaluation data used for training are obtained, wherein Q is a positive integer;

inputting the second image characteristics, the third parameter set and the Q groups of second image quality evaluation data into a first neural network model for operation to obtain a fourth parameter set;

performing image processing on the second original image data through the fourth parameter set by using the image processor to obtain a reference image;

Optionally, in the aspect of adjusting the model parameters of the first neural network model according to the Q sets of third image quality evaluation data, the program includes instructions for performing the following steps:

constructing a loss function according to the Q groups of third image quality evaluation data;

iteratively training the first neural network model using the loss function until the training termination condition is reached.

In addition, the program described above is also for executing a model training method including instructions for performing the steps of:

acquiring a second image feature of second original image data for training;

acquiring a third parameter set for training;

Optionally, in the aspect of acquiring the second image feature of the second raw image data for training, the program includes instructions for performing the following steps:

The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the electronic device comprises corresponding hardware structures and/or software modules for performing the respective functions in order to realize the above-mentioned functions. Those of skill in the art will readily appreciate that the present application is capable of hardware or a combination of hardware and computer software implementing the various illustrative elements and algorithm steps described in connection with the embodiments provided herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the electronic device may be divided into the functional units according to the method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

Fig. 6 is a block diagram showing functional units of an image processing apparatus 600 according to an embodiment of the present application. The image processing apparatus 600 includes: a first acquisition unit 601, a second acquisition unit 602, a third acquisition unit 603, a recommendation unit 604, and an image processing unit 605, wherein,

the first obtaining unit 601 is configured to obtain a first image feature of first original image data;

the second obtaining unit 602 is configured to obtain a first parameter set;

the third obtaining unit 603 is configured to obtain P groups of first image quality evaluation data, where P is a positive integer;

the recommending unit 604 is configured to input the first image feature, the first parameter set, and the P groups of first image quality evaluation data to a first neural network model for operation to obtain a second parameter set;

the image processing unit 605 is configured to perform image processing on the first original image data through the second parameter set by using an image processor to obtain a target image.

Optionally, in terms of acquiring the first image feature of the first original image data, the first acquiring unit 601 is specifically configured to:

Optionally, the image processing apparatus 600 is further specifically configured to:

acquiring a second image feature of second original image data for training;

acquiring a third parameter set for training;

Optionally, in terms of adjusting the model parameters of the first neural network model according to the Q sets of third image quality evaluation data, the image processing apparatus 600 is specifically configured to:

It should be noted that the electronic device described in the embodiments of the present application is presented in the form of a functional unit. The term "unit" as used herein should be understood in its broadest possible sense, and objects used to implement the functionality described in each "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The first obtaining unit 601, the second obtaining unit 602, and the third obtaining unit 603 may be memories or processors, and the recommending unit 604 and the image processing unit 605 may be processors, based on which the functions or steps of any of the above methods can be implemented.

Fig. 7 is a block diagram showing functional units of a model training apparatus 700 according to an embodiment of the present application. The model training apparatus 700 includes: an acquisition unit 701, a recommendation unit 702, a processing unit 703, an evaluation unit 704 and an adjustment unit 705, wherein,

the acquiring unit 701 is configured to acquire a second image feature of second original image data for training; acquiring a third parameter set for training; obtaining Q groups of second image quality evaluation data, wherein Q is a positive integer;

the recommending unit 702 is configured to input the second image feature, the third parameter set, and the Q groups of second image quality evaluation data to a first preset neural network for operation, so as to obtain a fourth parameter set;

the processing unit 703 is configured to process the second original image data through the fourth parameter set by using an image processor, so as to obtain a reference image;

the evaluation unit 704 is configured to perform image quality evaluation on the reference image to obtain Q groups of third image quality evaluation data;

the adjusting unit 705 is configured to adjust the model parameters of the first neural network model according to the Q sets of third image quality evaluation data until a training termination condition is reached.

Optionally, in terms of obtaining the second image feature of the second original image data for training, the obtaining unit 701 is specifically configured to:

Optionally, in terms of adjusting the model parameters of the first neural network model according to the Q groups of third image quality evaluation data, the adjusting unit 705 is specifically configured to:

It should be noted that the electronic device described in the embodiments of the present application is presented in the form of a functional unit. The term "unit" as used herein is to be understood in its broadest possible sense, and objects used to implement the functions described by the respective "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or chipset) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The obtaining unit 701 may be a memory or a processor, and the recommending unit 702, the processing unit 703, the evaluating unit 704, and the adjusting unit 705 may be processors, based on which the functions or steps of any of the above methods can be implemented.

The present embodiment also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute the embodiments of the present application to implement any one of the methods in the embodiments.

The present embodiment also provides a computer program product, which when run on a computer causes the computer to execute the relevant steps described above to implement any of the methods in the above embodiments.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip can execute any one of the methods in the above method embodiments.

The electronic device, the computer storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, and therefore, the beneficial effects that can be achieved by the electronic device, the computer storage medium, the computer program product, or the chip may refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the description of the foregoing embodiments, those skilled in the art will understand that, for convenience and simplicity of description, only the division of the functional modules is used for illustration, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device may be divided into different functional modules, so as to complete all or part of the functions described above.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, a module or a unit may be divided into only one logic function, and may be implemented in other ways, for example, a plurality of units or components may be combined or integrated into another apparatus, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed to a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application, or portions of the technical solutions that substantially contribute to the prior art, or all or portions of the technical solutions may be embodied in the form of a software product, where the software product is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that the method comprises:

acquiring a first image characteristic of first original image data;

acquiring a first parameter set;

2. The method of claim 1, wherein the image processor is a physical hardware image processor or a simulator corresponding to the image processor.

3. The method of claim 1 or 2, wherein obtaining the first image feature of the first raw image data comprises:

4. The method according to claim 1 or 2, characterized in that the method further comprises:

acquiring a second image feature of second original image data for training;

acquiring a third parameter set for training;

5. The method of claim 4, wherein adjusting the model parameters of the first neural network model according to the Q sets of third image quality assessment data comprises:

and iteratively training the first neural network model by using the loss function until the training termination condition is reached.

6. A method of model training, the method comprising:

acquiring a second image feature of second original image data for training;

acquiring a third parameter set for training;

7. The method of claim 6, wherein the image processor is a physical hardware image processor or a simulator corresponding to the image processor.

8. The method of claim 6 or 7, wherein the obtaining second image features of second raw image data for training comprises:

9. The method according to claim 6 or 7, wherein the adjusting the model parameters of the first neural network model according to the Q sets of third image quality evaluation data comprises:

10. An image processing apparatus, characterized in that the apparatus comprises: a first acquisition unit, a second acquisition unit, a third acquisition unit, a recommendation unit and an image processing unit, wherein,

the second obtaining unit is used for obtaining the first parameter set;

and the image processing unit is used for processing the first original image data by using the image processor through the second parameter set to obtain a target image.

11. A model training apparatus, the apparatus comprising: an acquisition unit, a recommendation unit, a processing unit, an evaluation unit and an adjustment unit, wherein,

the acquisition unit is used for acquiring second image characteristics of second original image data used for training; acquiring a third parameter set for training; acquiring Q groups of second image quality evaluation data for training, wherein Q is a positive integer;

12. An electronic device, comprising a processor, a memory for storing one or more programs and configured for execution by the processor, the programs comprising instructions for performing the steps of the method of any of claims 1-9.

13. A computer-readable storage medium, characterized in that a computer program for electronic data exchange is stored, wherein the computer program causes a computer to perform the method according to any one of claims 1-9.