WO2023001110A1

WO2023001110A1 - Neural network training method and apparatus, and electronic device

Info

Publication number: WO2023001110A1
Application number: PCT/CN2022/106274
Authority: WO
Inventors: 刘行
Original assignee: 维沃移动通信（杭州）有限公司
Priority date: 2021-07-23
Filing date: 2022-07-18
Publication date: 2023-01-26
Also published as: CN113706402A

Abstract

A neural network training method and apparatus, and an electronic device. The method comprises: determining N degradation degrees according to M environment parameters of M obtained first images (101), one degradation degree corresponding to at least one environment parameter, one degradation degree corresponding to at least one first image, and both M and N being positive integers; performing degradation processing on the first image corresponding to each degradation degree on the basis of each degradation degree so as to obtain a second image corresponding to each degradation degree (102), each second image corresponding to one first image; generating a sample set on the basis of the first image and the second image corresponding to each degradation degree to obtain N sample sets (103); respectively training Q neural networks on the basis of the N sample sets (104), one sample set corresponding to at least one neural network, and Q being a positive integer.

Description

Neural network training method, device and electronic equipment

Cross References to Related Applications

This application claims priority to Chinese Patent Application No. 202110838140.6 filed in China on July 23, 2021, the entire contents of which are hereby incorporated by reference.

technical field

The application belongs to the field of image processing and deep learning, and specifically relates to a neural network training method, device and electronic equipment.

Background technique

With the continuous development of artificial intelligence technology, electronic devices are becoming more and more popular, and users have higher and higher requirements on the shooting quality of electronic devices (such as smart terminals). At present, the solution to improve the shooting quality of electronic equipment is mainly realized by learning the deep network of face quality enhancement through convolutional neural network.

In related technologies, a deep network with a single input resolution (such as 256, 512) and a single image degradation is trained through a training set to perform image enhancement processing on captured images. Using the above image processing method, in some special scenes, such as low-illumination scenes, the amount of light entering is small when shooting, which is greatly affected by noise. Compared with scenes with normal brightness, the quality of images captured in low-light scenes is degraded. More severe (ie, lower image quality). For another example, when shooting group portraits, the degrees of degradation of faces with inconsistent sizes in the lens are also different.

Therefore, if a deep learning network trained with a single input resolution and single data degradation in related technologies is still used to process the captured images, it will cause the trained deep learning network to be unable to perform targeted enhancement processing on images with different degrees of degradation. As a result, it is difficult to obtain a dominant face quality enhancement effect, which in turn leads to a poor image processing effect.

Contents of the invention

The purpose of the embodiment of the present application is to provide a neural network training method, device and electronic equipment, which can solve the technical problem of poor image processing effect of the deep learning network in the related art.

In order to solve the above-mentioned technical problems, the application is implemented as follows:

In the first aspect, the embodiment of the present application provides a neural network training method, the method includes: determining N degradation degrees according to the acquired M environmental parameters of the M first images, and one degradation degree corresponds to at least one environmental parameter, A degree of degradation corresponds to at least one first image, and M and N are both positive integers; based on each degree of degradation, the first image corresponding to each degree of degradation is degraded to obtain a second image corresponding to each degree of degradation , each second image corresponds to a first image; based on the first image and the second image corresponding to each degree of degradation, a sample set is generated respectively, and N sample sets are obtained; based on N sample sets, Q The neural network is trained, and a sample set corresponds to at least one neural network, and Q is a positive integer.

In the second aspect, the embodiment of the present application provides a neural network training device, which includes: the device includes: a determination module, a processing module, a generation module and a training module, wherein: the determination module is used to The M environmental parameters of the first image determine N degradation degrees, one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to at least one first image, M and N are both positive integers; the above processing module is used to For each degradation degree determined by the determination module, perform degradation processing on the first image corresponding to each degradation degree to obtain a second image corresponding to each degradation degree, and each second image corresponds to a first image respectively; the above-mentioned generation module , used to generate a sample set based on the first image and the second image corresponding to each degree of degradation obtained by the processing module to obtain N sample sets; the above training module is used to obtain N sample sets based on the generation module, Q neural networks are trained respectively, a sample set corresponds to at least one neural network, and Q is a positive integer.

In a third aspect, an embodiment of the present application provides an electronic device, the electronic device includes a processor, a memory, and a program or instruction stored in the memory and operable on the processor, and the program or instruction is The processor implements the steps of the method described in the first aspect when executed.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or an instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented .

In the fifth aspect, the embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions, so as to implement the first aspect the method described.

In a sixth aspect, an embodiment of the present application provides a computer program product, the program product is stored in a non-volatile storage medium, and the program product is executed by at least one processor to implement the method described in the first aspect.

In the embodiment of the present application, the neural network training device determines N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to at least one first image , M and N are both positive integers; and based on each degree of degradation, degrade the first image corresponding to each degree of degradation to obtain the second image corresponding to each degree of degradation, and each second image corresponds to a The first image, then, based on the first image and the second image corresponding to each degree of degradation above, generate a sample set respectively to obtain N sample sets, and finally, based on the above N sample sets, respectively perform Q neural networks For training, a sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the neural network training device can degrade the first image under different environmental parameters to a corresponding degree of degradation (that is, there is a difference), so as to obtain the corresponding degraded image after the first image is degraded differently (that is, the second image). image), so as to construct a training sample set that conforms to the shooting quality under various environmental parameters (that is, different degrees of degradation), and then obtain a neural network suitable for targeted processing of captured images under different environmental parameters. Image enhancement processing effect.

Description of drawings

Fig. 1 is the flowchart of a kind of neural network training method provided by the embodiment of the present application;

Fig. 2 is a schematic flow chart of processing by a multi-intensity degradation module provided in an embodiment of the present application;

Fig. 3 is a schematic diagram of constructing a low-definition-high-definition training set provided by the embodiment of the present application;

Fig. 4 is a schematic diagram of ISO and face size information processing provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of a multi-complexity GAN network flow provided by an embodiment of the present application;

FIG. 6 is a flow chart of an image processing method provided by an embodiment of the present application;

Fig. 7 is a schematic structural diagram of a neural network training device provided by an embodiment of the present application;

FIG. 8 is a schematic structural diagram of an image processing device provided in an embodiment of the present application;

FIG. 9 is one of the schematic diagrams of the hardware structure of an electronic device provided in the embodiment of the present application;

FIG. 10 is a second schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific sequence or sequence. It should be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application can be practiced in sequences other than those illustrated or described herein, and that references to "first," "second," etc. distinguish Objects are generally of one type, and the number of objects is not limited. For example, there may be one or more first objects. In addition, "and/or" in the specification and claims means at least one of the connected objects, and the character "/" generally means that the related objects are an "or" relationship.

The neural network training method provided by the embodiment of the present application will be described in detail below through specific embodiments and application scenarios with reference to the accompanying drawings.

An embodiment of the present application provides a neural network training method, which can be applied to electronic devices. FIG. 1 shows a flowchart of the neural network training method provided in the embodiment of the present application. As shown in Figure 1, the neural network training method provided by the embodiment of the present application may include the following steps 101 to 104:

Step 101: Determine N degradation degrees according to the acquired M environmental parameters of the M first images.

Wherein, one degree of degradation corresponds to at least one environmental parameter, one degree of degradation corresponds to at least one first image, and both M and N are positive integers.

Optionally, in the embodiment of the present application, the above-mentioned neural network training method can be a faceEnhanceGAN network (faceEnhanceGAN) training method, since the faceEnhanceGAN network (faceEnhanceGAN) training method is a supervised AI training method, it needs Paired image pairs (ie, low-resolution images-high-resolution images) constitute the training dataset. However, due to factors such as inability to align pixels, it is impossible to directly capture a one-to-one corresponding low-definition-high-definition image pair. Therefore, low-definition pictures can be obtained by performing various degradation processes (such as blurring, adding noise, etc.) on the acquired high-definition pictures (such as high-definition pictures taken by SLR) to obtain low-definition-high-definition image pairs, namely LQ -HQ dataset.

In this embodiment of the present application, the above-mentioned first image may be a high-definition picture. Exemplarily, the above-mentioned first image may be a high-definition picture captured by a camera of a high-quality imaging device (such as a single-lens reflex camera), or the above-mentioned first image may be a high-definition picture obtained by performing image enhancement on a low-quality image.

It should be noted that the clarity and quality of a picture can be measured by the degree of blur and noise, and the above-mentioned high-definition picture refers to a picture with less blur and less noise.

Exemplarily, when the neural network training device acquires the above-mentioned M first images, each first image and the environmental parameters corresponding to the first images can be stored in the database, and the neural network training device can be used when necessary The M first images can be called from the database.

Optionally, in the embodiment of the present application, the foregoing environmental parameters may include at least one of the following: ISO value of sensitivity and brightness.

It should be noted that the sensitivity refers to the sensitivity to light expressed by numbers. The higher the ISO value, the stronger the sensitivity to light, and vice versa.

Optionally, in the embodiment of the present application, the M environmental parameters of the M first images may be obtained by inputting the M first images to the ISO and face size information processing module for processing.

Exemplarily, the neural network training apparatus may determine the degree of degradation of the first image according to the magnitude relationship between the environmental parameters of the first image and the first threshold. Further, the above-mentioned first threshold is an ISO threshold, which may specifically be the size of the ISO. For example, when the ISO value of image 1 is greater than the first threshold, the image 1 corresponds to a first degree of degradation (ie, a higher degree of degradation).

Furthermore, the multi-stage ISO threshold (ISO_threshold) can be reasonably set by collecting the user's photo data and analyzing the darkness of the scene.

It should be noted that the aforementioned environmental parameters (namely ISO) may be referred to as prior information, and the aforementioned ISO and face size information processing module is a preprocessing module for obtaining the prior information.

Optionally, in this embodiment of the present application, the foregoing N degradation degrees may be one or more of the preset L degradation degrees. Exemplarily, the above L degradation degrees may include: a first degradation degree and a second degradation degree. Further, the first degree of degradation may be a high degree of degradation, and the second degree of degradation may be a low degree of degradation. It should be noted that, the above degradation degree may be flexibly determined according to actual conditions, for example, three or more degradation degrees may be set, which is not limited in this embodiment of the present application.

Optionally, in the embodiment of the present application, the above N degradation degrees correspond to N degradation modules respectively, each degradation module is used to perform corresponding degradation processing on the first image, and the degradation algorithm of each degradation module is different, namely , each degradation module corresponds to a different degradation effect.

Exemplarily, the above N degradation modules may include: a high ISO segment degradation module and a low ISO segment degradation module. Further, the high ISO segment degradation module may correspond to the above-mentioned first degradation degree, and the low ISO segment degradation module may correspond to the above-mentioned second degradation degree.

It should be noted that the above N degradation modules may also be refined into degradation modules of more ISO segments according to actual requirements, which is not limited in this embodiment of the present application.

It should be noted that the high ISO segment degradation module is used to process the first image with a large ISO value, and its degradation degree is relatively high, and the low ISO segment degradation module is used to process the first image with a large ISO value, and its degradation degree is low .

Exemplarily, the neural network training device may determine the degree of degradation corresponding to the environment parameter according to the environment parameter of each first image in the M first images, so as to determine the degree of degradation of each first image. For example, taking an ISO value lower than 50 as a low sensitivity and corresponding to the first degree of degradation (for example, a low degree of degradation) as an example, if the ISO value of image 1 is 45, then the degree of degradation corresponding to image 1 is the first degree of degradation , that is, the image 1 corresponds to a low degree of degradation.

Step 102: Based on each degree of degradation, perform degradation processing on the first image corresponding to each degree of degradation to obtain a second image corresponding to each degree of degradation.

Wherein, each second image corresponds to a first image.

In the embodiment of the present application, the neural network training device may use a degradation module corresponding to each degradation degree to perform quality degradation processing on the first image corresponding to each degradation degree to obtain a low-quality image or a low-resolution image corresponding to the first image (i.e., the second image above).

Optionally, in the embodiment of the present application, the neural network training device can use the multi-intensity data degradation module to determine corresponding data degradation modules of different intensities according to the degree of degradation corresponding to the first image, and perform corresponding quality degradation on the first image deal with.

Exemplarily, the neural network training device can determine the ISO flag bit information through the above-mentioned ISO and face size information processing module, and then determine the corresponding degradation module based on the ISO flag bit information through the above-mentioned multi-intensity data degradation module. Further, the above ISO flag is used to indicate the level of ISO, for example, the ISO flag is a high flag (ie highISO_flag), or the ISO flag is a low flag (ie lowISO_flag).

Further, when the ISO flag bit information is determined by the above-mentioned ISO and face size information processing module, the size relationship between the ISO value of the first image and the above-mentioned ISO threshold (ie, ISO_threshold) can be judged, if the ISO value is greater than ISO_threshold, then Set the high ISO flag bit (or high ISO sign bit) to 1, and if the ISO value is less than ISO_threshold, set the low ISO flag bit to 1.

Further, when the corresponding degradation module is determined based on the ISO flag bit information through the above multi-intensity data degradation module, the ISO flag bit information in the ISO and face size information processing module can be identified, and then based on the identified flag bit information, Determine the corresponding degenerate modules.

Example 1, after inputting multiple high-definition images to the ISO and face size information processing module and the multi-intensity data degradation module, if the high ISO flag highISO_flag is recognized to be equal to 1 from the ISO and face size information processing module, then It indicates that the high ISO segment is entered, and at this time, the high ISO segment degradation module is entered, through which the image (that is, the image to be degraded) can be added with multiple types of noise with greater intensity during the degradation process, and multiple types of noise with higher intensity can be added. Large model operations, such as triangular models, Gaussian models, linear models, motion blur and other fuzzy functions, all increase the fuzziness. In addition, the input high-definition image is randomly darkened to simulate the brightness of the image under dark light.

Example 2, after inputting multiple high-definition images to the ISO and face size information processing module and the multi-intensity data degradation module, if the low ISO flag bit lowISO_flag is recognized as 1 from the ISO and face size information processing module, then It means that the image is entered into the low ISO segment. At this time, it enters the low ISO segment degradation module. Through this degradation module, the image can be added with less intense noise, and less types of blur functions can be added for lesser degree of blurring.

Combining the processes of Example 1 and Example 2 above, FIG. 2 is a schematic flow chart of processing using a multi-intensity degradation module. As shown in Figure 2, if the low ISO flag bit lowISO_flag corresponding to the high-definition image is equal to 1, enter the low ISO degradation module to process the high-definition image to obtain the corresponding low-definition image 1; if the high-ISO flag bit highISO_flag corresponding to the high-definition image If it is equal to 1, enter the high-ISO degradation module to process the high-definition image to obtain the corresponding low-definition image 2.

Exemplarily, the above-mentioned degradation processing includes but is not limited to at least one of the following: noise addition processing, blur processing (defocus blur, motion blur, Gaussian blur, linear blur), reduce/improve brightness (3D lighting), increase/decrease Shadows (random shadows), area-aware degradation.

Further, when using the degradation module to process the first image, various blurring models can be added to blur the first image through various blurring functions, for example, defocus model, Gaussian model, linear model, Fuzzy functions such as motion models.

Exemplarily, the above degradation degree may include at least one of the following: degradation intensity and degradation mode. Further, the above degradation intensity refers to the degree of processing, for example, the size of added noise, the size of the degree of blur; the above degradation mode refers to the processing method, for example, adding noise to image 1 and blurring image 2, that is There are two different processing modes.

It should be noted that, in the actual shooting scene, the degree of image quality degradation obtained by shooting in different ISO scenes is usually different. The image quality degradation degree of high ISO scene and low ISO scene is higher, and the quality of the obtained image is worse. Specifically, through multiple experiments and tests, it is found that the worse image quality in high ISO scenes is mainly manifested in: 1) the image is more blurred and the focus is more inaccurate, 2) the noise in the image is larger and the noise is uneven. Therefore, in the embodiment of the present application, the multi-intensity degradation module is designed to perform differential image quality degradation on the first images of different ISOs (ie, high ISO and low ISO) according to the degradation intensity and degradation mode, so as to construct a better data set Simulate the image quality of electronic equipment from different ISO ranges. Fig. 3 shows a schematic diagram of low-definition-high-definition training set construction. As shown in Fig. 3, the image training device performs one or more of the above-mentioned degradation processes on several high-definition images to obtain several quality degraded images (i.e., low clear image), so as to obtain the high-definition-low-definition training set.

For example, take M first images as 100 high-definition images as an example. Assuming that the ISO values of 40 images in the 100 high-definition images are greater than the ISO threshold value, corresponding to the first degree of degradation, and the ISO values of the 60 images other than the 40 images are less than the ISO threshold value, corresponding to the second degree of degradation, then the The 40 images are input to the high ISO segment degradation module corresponding to the first degradation degree for processing, and the 60 images are input to the low ISO segment degradation module corresponding to the second degradation degree for processing, so as to obtain low clear image.

Step 103: Based on the first image and the second image corresponding to each degree of degradation, respectively generate a sample set to obtain N sample sets.

In the embodiment of the present application, one degree of degradation corresponds to one sample set, and N degrees of degradation correspond to N sample sets, that is, the above N sample sets respectively correspond to N degrees of degradation, and the N sample sets constitute the training data set.

Optionally, in this embodiment of the present application, a sample set includes at least one first image and at least one second image, that is, each sample set includes at least one low-definition-high-definition image pair, that is, LQ-HQ data set.

Optionally, in this embodiment of the present application, one degree of degradation corresponds to one training data set. Exemplarily, the neural network training device may store the first image and the second image corresponding to the first image as training sample images in a corresponding training data set.

It should be noted that in the actual shooting scene, the larger the ISO value is, the darker the scene is, and the quality of the picture taken in the darker scene is more severely degraded, such as more blurred, more noise, etc.; the ISO value The smaller the value, the brighter the shooting scene, and the quality degradation of the picture taken in a bright scene is obviously weaker than that in a high ISO scene. Based on this, when training the neural network to enhance the image captured by the electronic device, from the perspective of the algorithm, if the image signal processing (i.e., ISP) process of the electronic device and the image quality during the shooting process can be simulated more closely In the degradation mode, a high-quality training set can be constructed, and a neural network with better face enhancement effect can be trained.

Step 104: Based on the above N sample sets, train Q neural networks respectively.

Wherein, one sample set corresponds to at least one neural network, and Q is a positive integer.

In the embodiment of the present application, the aforementioned Q neural networks may include a generative adversarial network (GAN network).

Exemplarily, the aforementioned Q neural networks may be neural networks among a plurality of preset neural networks of different complexity.

Optionally, in the embodiment of the present application, the neural network training device may train Q neural networks respectively based on the aforementioned N sample sets in the multi-complexity GAN network module. Exemplarily, the above multi-complexity GAN network module includes P neural networks, and the neural network training device can determine Q neural networks from the P neural networks for training. Further, the neural network training device determines Q neural networks from the P neural networks for training according to the degree of degradation corresponding to the N sample sets, where P is greater than Q, and P is a positive integer.

Exemplarily, the neural network training device may respectively input the above N sample sets into the Q neural network to train the neural network, and obtain Q neural networks after training. Further, the above Q neural networks may be the same, different or partly the same.

It should be noted that the same neural network refers to a neural network with the same complexity and input resolution.

It should be noted that, since the degradation degrees of the above N sample sets are different, after the neural network is trained based on each sample set in the N sample sets, the parameters (ie weights) of the trained neural network are different. , that is, neural networks trained with different sample sets have different effects on image enhancement.

In the neural network training method provided in the embodiment of the present application, the neural network training device determines N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to At least one first image, M and N are both positive integers; and each degradation degree is used to perform degradation processing on the first image corresponding to each degradation degree to obtain a second image corresponding to each degradation degree, and each The second images respectively correspond to a first image, and then, based on the above-mentioned first image and second image corresponding to each degree of degradation, respectively generate a sample set to obtain N sample sets, and finally, based on the above-mentioned N sample sets, respectively Q neural networks are trained, a sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the neural network training device can degrade the first image under different environmental parameters to a corresponding degree of degradation (that is, there is a difference), so as to obtain the corresponding degraded image after the first image is degraded differently (that is, the second image). image), so as to construct a training sample set that conforms to the shooting quality under various environmental parameters (that is, different degrees of degradation), and then obtain a neural network suitable for targeted processing of captured images under different environmental parameters. Image enhancement processing effect.

Optionally, in the embodiment of the present application, the process of the above step 101 may include the following steps 101a and 101b:

Step 101a: From the X parameter ranges, determine N target parameter ranges corresponding to the above M environmental parameters.

Wherein, each parameter range corresponds to a degree of degradation, and a target parameter range corresponds to at least one environmental parameter.

Step 101b: Determine the degradation degree corresponding to the i-th target parameter range among the N target parameter ranges as the i-th degradation degree.

Among them, the above i-th target parameter range is: according to the order of the corresponding degradation degree from small to large, the i-th target parameter range after sorting the above-mentioned N target parameter ranges, i is a positive integer; the above-mentioned i-th target The environment parameter corresponding to the parameter range is smaller than the environment parameter corresponding to the i+1th target parameter range.

Optionally, the above-mentioned X parameter ranges may be preset parameter ranges, and the parameter ranges refer to the range of ISO values. For example, the X parameter ranges may include three parameter ranges with an ISO value less than 50, an ISO value greater than or equal to 50, less than 200, and an ISO value greater than or equal to 200.

It should be noted that the greater the environmental parameter corresponding to the image, the darker the shooting scene corresponding to the image is, and the greater the degradation degree of the captured image is. Therefore, the image needs to be degraded to a greater extent to obtain a training data set that can better simulate the real shooting scene. That is, the greater the environmental parameter (ISO value) in the parameter range of the environmental parameter of the first image is, the greater the ISO value of the first image is, and the corresponding degradation degree of the first image is higher.

Optionally, in this embodiment of the present application, each of the M first images includes at least one human face element.

Optionally, before the above step 104, the neural network training method provided in the embodiment of the present application further includes the following steps A1 and B1:

Step A1: Obtain R area size parameters corresponding to the human face elements in the above M first images.

Wherein, each area size parameter is respectively: a size parameter of the area where the face element in the first image is located, R is a positive integer, and R is greater than or equal to M.

Step B1: Determine the aforementioned Q neural networks according to the aforementioned M environmental parameters and the aforementioned R area size parameters.

Wherein, a neural network corresponds to: at least one environment parameter and at least one area size parameter.

Optionally, one first image may include one or more human face elements. For example, if the image 2 is a group photo of three people, the image 2 includes three face elements, and the image 2 corresponds to three area size parameters.

Optionally, the aforementioned region size parameters may include at least one of the following: length, width, and area.

Exemplarily, the neural network training device may obtain the region size parameters corresponding to at least one face element in each of the above M first images to obtain R region size parameters, wherein each first The image may correspond to one or more of the above R area size parameters.

Optionally, the neural network training device may respectively determine corresponding environmental parameter levels and area size levels according to the aforementioned M environmental parameters and the aforementioned R area size parameters. Further, the aforementioned environmental parameter level may include a low ISO scene and a high ISO scene, and the aforementioned area size level may include a large area and a small area. Exemplarily, the position information of the face in the first image can be acquired through a face detection algorithm. Further, the face element in the first image can be frame-selected by using a rectangular frame, and the information of the rectangular frame is expanded by a certain ratio to obtain the width (ie, face_width) and height (ie, face_width) of the face area corresponding to the face element. , face_height), so that the area (face_area) of the size of the face can be calculated as face_area=face_width*face_height.

Exemplarily, the above-mentioned ISO and face size information processing module can be used to determine the face area flag information. Further, the face area flag is used to characterize the size of the area of the face element, for example, the face area flag is a large area flag (that is, bigFace_flag), or the face area flag is a small area flag ( That is, smallFace_flag).

Further, when determining the face area flag bit information by the above-mentioned ISO and face size information processing module, the size relationship between the area of the face element of the first image and the area threshold (that is, area_threshold) can be judged, if face_area is greater than area_threshold, set the large area flag (or large area sign) to 1, and if face_area is smaller than area_threshold, set the small area flag to 1. Further, the above-mentioned area threshold is a multi-stage area threshold set in advance, and the specific area threshold can be determined by collecting user photo data.

Further, the neural network training device can identify the flag information of the face area in the ISO and face size information processing module, and then determine the level of the size of the face area based on the identified flag information. For example, if it is recognized that bigFace_flag is 1, it is determined that the face element is a face element with a large area.

Exemplarily, the neural network training apparatus may determine the ISO flag bit by comparing the environmental parameter (ie, the ISO value) of the first image with the ISO threshold. For example, if the ISO value is greater than ISO_threshold, set the high ISO flag bit (or high ISO sign bit) to 1, and if the ISO value is smaller than ISO_threshold, set the low ISO flag bit to 1.

Further, the neural network training device can identify the ISO flag information in the ISO and face size information processing module, and then determine the ISO level based on the identified flag information. For example, if it is recognized that highISO_flag is equal to 1, it is determined that the first image corresponds to a low ISO value, that is, a low ISO scene. Figure 4 is a schematic diagram of ISO and face size information processing.

Exemplarily, the neural network training device may determine the neural network corresponding to each sample set according to the identified flag information of the face area corresponding to each sample set and the ISO flag information.

Further, the neural network device can identify the N sample sets one by one. In the case where the flag bit information of the face area of the first image in a certain sample set is recognized to indicate a large-area face element (that is, bigFace_flag is equal to 1), The neural network training device inputs the sample set into the neural network of the first complexity (ie high complexity), and trains the neural network of the first complexity to obtain the trained neural network; In the case where the flag bit information of the face area of the first image in this set indicates a small-area face element (that is, smallFace_flag is equal to 1), the sample set is input into a low-complexity neural network for training.

Further optionally, in the embodiment of the present application, among the above Q neural networks, the target parameter corresponding to the neural network with high complexity is greater than the target parameter corresponding to the neural network with small complexity;

Wherein, the above target parameters include: environment parameters and area size parameters.

Optionally, the aforementioned neural network with high complexity refers to a neural network with relatively high time complexity and/or space complexity. Further, the complexity of the neural network can be reflected in the depth and breadth of the neural network.

Exemplarily, for the first image with a large environmental parameter (ie, the ISO value), that is, the first image in a high ISO scene, due to the implementation of more complex (ie, higher degree of degradation) degradation processing, this A set of high-definition-low-definition data sets will be obtained with corresponding more severely degraded images. Therefore, a deeper, larger and more complex neural network can be used. For low-ISO scenes, due to the implementation of a higher Simple (that is, less degraded) degradation processing, at this time, corresponding images with relatively light degradation will be obtained to form a set of high-definition-low-definition data sets. Therefore, a shallower and smaller neural network can be used. That is, it is possible to select a high-complexity neural network for the training set with a high degree of degradation for training, and select a neural network with a low complexity for training for a data set with a low degree of degradation. In this way, while ensuring a better processing effect, It can reduce the computational complexity and speed up the processing time of the algorithm, thereby improving the training efficiency.

Exemplarily, FIG. 5 is a schematic diagram of a multi-complexity GAN network process. As shown in FIG. 5, assuming that the ISO value of image 1 corresponds to a low ISO scene, it is input into a simple GAN network for training to obtain the low ISO value of the scene. The image to be processed has a trained neural network with better processing effect; assuming that the ISO value of image 1 corresponds to a high ISO scene, it is input into a complex GAN network for training to obtain an image to be processed that corresponds to a high ISO scene. A trained neural network with better processing results.

Further optionally, in the embodiment of the present application, each sample set corresponds to at least two neural networks.

Optionally, the process of the above step B1 may include the following step C1:

Step C1: Determine the first neural network according to the first environment parameter and the first area size parameter;

Wherein, the above-mentioned first environmental parameter is: among the above-mentioned N sample sets, among the environmental parameters of the first image in any one of the sample sets; the above-mentioned first area size parameter is: the area size corresponding to the first image in any of the above-mentioned sample sets in the parameter.

The above-mentioned first neural network is: one of the neural networks corresponding to any of the above-mentioned sample sets.

Optionally, the first image in the aforementioned one sample set corresponds to a degree of degradation, and the degree of degradation may correspond to a neural network.

It should be noted that each sample set is a high-definition-low-definition image pair, and the low-definition image in each sample set is a degraded image of the high-definition image.

Exemplarily, when the area size parameters (that is, the face area) of the face elements in the first image in a sample set are not the same, for example, there are large-area face elements and small-area face elements in the sample set. When the first image of an element is selected, a neural network with a larger input resolution and higher complexity can be selected for the first image including a large-area face element, and the first image selection for a face element including a small area Feed a neural network with a smaller resolution and lower complexity for training. In this way, the training efficiency can be improved while achieving a better training effect.

An embodiment of the present application provides an image processing method, which can be applied to an electronic device, and the electronic device can include Q neural networks trained by the above neural network training method, where Q is a positive integer. FIG. 6 shows a flow chart of the image processing method provided by the embodiment of the present application. As shown in Figure 6, the neural network training method provided by the embodiment of the present application may include the following steps 201 and 202:

Step 201: In the case of displaying the preview image collected by the camera of the above-mentioned electronic device, according to the target environment parameters of the above-mentioned preview image, and the target area size parameters corresponding to the face elements in the above-mentioned preview image, from the Q neural networks after training , determine the target neural network.

In the embodiment of the present application, the above-mentioned target environment parameters include at least one of the following: ISO value and brightness value; the above-mentioned target area size parameters include at least one of the following: width, height, and area.

Exemplarily, when a preview image is displayed, the current ISO value of the camera may be obtained, and the current ISO value of the camera may be determined as the ISO value of the preview image, and the face of the face element in the preview image may be obtained area information.

Further, the image processing device may determine whether the current shooting scene is a high ISO scene or a low ISO scene based on the obtained ISO value, and determine whether the face element to be processed is a large area or a small area based on the obtained face area information. area.

Optionally, in the embodiment of the present application, the above-mentioned target neural network is a trained adversarial neural network that matches the above-mentioned target environment parameters and target area size parameters. The above-mentioned target neural network is used to perform image enhancement processing on the images captured by the camera.

Step 202: Input the third image captured by the above-mentioned camera into the above-mentioned target neural network for image processing to obtain a processed image.

Wherein, the above-mentioned third image is: an image captured by the camera within a predetermined time period, and the above-mentioned predetermined time period is a time period between the time when the camera collects the preview image and the time when the camera stops collecting the preview image.

Optionally, in this embodiment of the present application, the above-mentioned third image may be an image including human face elements.

Optionally, in the embodiment of the present application, the above image processing may be image enhancement processing, for example, performing enhancement processing on the human face area in the image by means of denoising, removing blur, increasing brightness, increasing contrast, and the like.

In the image processing method provided in the embodiment of the present application, the image processing device can display the preview image captured by the camera of the electronic device, according to the target environment parameters of the preview image, and the size of the target area corresponding to the face elements in the preview image parameters, and determine the target neural network from the Q neural networks after training. Through this method, the image processing device can intelligently select a neural network that conforms to the current environmental parameters and face size for image enhancement processing based on the environmental parameters when the image is captured and the size of the human face in the captured image, thereby greatly Improves the effect and performance of face enhancement on captured images.

Optionally, in the embodiment of the present application, in the above-mentioned step 201, according to the target environment parameters of the above-mentioned preview image, and the target area size parameters corresponding to the face elements in the above-mentioned preview image, from the Q neural networks after training, determine Before the target neural network is generated, the image processing method provided in the embodiment of the present application also includes the following step D1:

Step D1: In the case that the above-mentioned preview image contains human face elements, obtain target environment parameters and target area size parameters corresponding to the human face elements.

Exemplarily, it is possible to detect whether the preview image contains human face elements by acquiring contours.

Optionally, in the embodiment of the present application, in the above-mentioned step 201, according to the target environment parameters of the above-mentioned preview image, and the target area size parameters corresponding to the face elements in the above-mentioned preview image, from the Q neural networks after training, determine The process of obtaining the target neural network may include the following steps 201a:

Step 201a: Based on the pre-stored Q correspondences, determine the above-mentioned target neural network corresponding to the above-mentioned target environment parameters and the above-mentioned target area size parameters.

Wherein, each corresponding relationship is respectively: a corresponding relationship between at least one environment parameter, at least one region size parameter and a trained neural network.

Alternatively, each of the above correspondences is respectively: a correspondence between an environment parameter level, an area size level and a trained neural network.

Exemplarily, the above corresponding relationship may be established when a trained neural network is obtained after the training of a neural network is completed.

For example, the trained neural network A corresponds to a low-ISO scene, and the trained neural network B corresponds to a high-ISO scene. Assuming that the above-mentioned target environment parameters correspond to a low-ISO scene, the above-mentioned target neural network can be neural network A.

For another example, the trained neural network C corresponds to a large-area face element, and the trained neural network D corresponds to a small-area face element. Assuming that the above-mentioned target environment parameters correspond to a large-area face element, the above-mentioned target neural network can be neural network network C.

It should be noted that, the neural network training method provided in the embodiment of the present application may be executed by a neural network training device, or a control module in the neural network training device for executing the neural network training method. In the embodiment of the present application, the neural network training device provided by the embodiment of the present application is described by taking the neural network training method executed by the neural network training device as an example.

The embodiment of the present application provides a neural network training device 600, as shown in Figure 7, the device includes: a determination module 601, a processing module 602, a generation module 603 and a training module 604, wherein:

The above determination module 601 is configured to determine N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to at least one first image, M, N are positive integers; the processing module 602 is configured to perform degradation processing on the first image corresponding to each degradation degree based on each degradation degree determined by the determination module 601 to obtain the second image corresponding to each degradation degree. image, each second image corresponds to a first image; the generation module 603 is used to generate a sample set based on the first image and the second image corresponding to each degree of degradation obtained by the processing module 602, and obtain N sample sets; the training module 604 is used to train Q neural networks based on the above N sample sets obtained by the generating module 603, one sample set corresponds to at least one neural network, and Q is a positive integer.

Optionally, in the embodiment of the present application, the determination module 601 is specifically configured to determine N target parameter ranges corresponding to the above M environmental parameters from the X parameter ranges, and each parameter range corresponds to a degradation degree , one target parameter range corresponds to at least one environmental parameter; the determination module 601 is specifically configured to determine the degradation degree corresponding to the i-th target parameter range in the above-mentioned N target parameter ranges as the i-th degradation degree; wherein, the above-mentioned The i-th target parameter range is: the i-th target parameter range after sorting the above N target parameter ranges according to the order of the corresponding degradation degree from small to large, where i is a positive integer; the above-mentioned i-th target parameter range corresponds to The environmental parameter of is smaller than the environmental parameter corresponding to the i+1th target parameter range.

Optionally, in this embodiment of the present application, each of the M first images includes at least one human face element;

The above device also includes: an acquisition module 605;

The acquisition module 605 is configured to acquire R area size parameters corresponding to the face elements in the M first images, and each area size parameter is: the area where the face elements in the first image are located. Size parameter, R is a positive integer, and R is greater than or equal to M; the above determination module 601 is also used to determine the above Q neural networks according to the M environmental parameters obtained by the above acquisition module 605 and the above R area size parameters; wherein, one The neural network corresponds to: at least one environment parameter and at least one region size parameter.

Optionally, in the embodiment of the present application, among the above-mentioned Q neural networks, the target parameters corresponding to the neural networks with high complexity are greater than the target parameters corresponding to the neural networks with small complexity; wherein, the above-mentioned target parameters include: environmental parameters and area size parameters.

Optionally, in the embodiment of the present application, each sample set corresponds to at least one neural network;

The above determination module 601 is specifically configured to determine the first neural network according to the first environment parameter and the first area size parameter; wherein, the above first environment parameter is: the first image in any one of the N sample sets mentioned above Among the environmental parameters; the above-mentioned first area size parameter is: among the area size parameters corresponding to the first image in any of the above-mentioned sample sets;

In the neural network training device provided in the embodiment of the present application, the neural network training device determines N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to At least one first image, M and N are both positive integers; and each degradation degree is used to perform degradation processing on the first image corresponding to each degradation degree to obtain a second image corresponding to each degradation degree, and each The second images respectively correspond to a first image, and then, based on the above-mentioned first image and second image corresponding to each degree of degradation, respectively generate a sample set to obtain N sample sets, and finally, based on the above-mentioned N sample sets, respectively Q neural networks are trained, a sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the neural network training device can degrade the first image under different environmental parameters to a corresponding degree of degradation (that is, there is a difference), so as to obtain the corresponding degraded image after the first image is degraded differently (that is, the second image). image), so as to construct a training sample set that conforms to the shooting quality under various environmental parameters (that is, different degrees of degradation), and then obtain a neural network suitable for targeted processing of captured images under different environmental parameters. Image enhancement processing effect.

It should be noted that, the image processing method provided in the embodiment of the present application may be executed by an image processing device, or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing device executed by the image processing device is taken as an example to describe the image processing device provided in the embodiment of the present application.

The embodiment of the present application provides an image processing device 700. As shown in FIG. 8, the device includes Q neural networks trained by the above neural network training method, and Q is a positive integer; the device includes: a determination module 701 and a processing module 702, of which:

The above-mentioned determination module 701 is used for displaying the preview image collected by the camera of the electronic device, according to the target environment parameters of the preview image, and the target area size parameters corresponding to the face elements in the preview image, from the Q neurons after training In the network, determine the target neural network;

The processing module 702 is configured to input the third image captured by the camera into the target neural network determined by the determination module 701 for image processing to obtain a processed image. The third image is: an image captured by the camera within a predetermined period of time. The predetermined time period is: the time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.

Optionally, in the embodiment of the present application, the device 700 further includes an acquisition module 703, the above-mentioned acquisition module 703 is used to acquire the target environment parameters corresponding to the face element when the above-mentioned preview image contains a face element Target area size parameter.

Optionally, in the embodiment of the present application, the above-mentioned determination module 701 is specifically configured to determine the target neural network corresponding to the above-mentioned target environment parameters and the above-mentioned target area size parameters based on the pre-stored Q correspondences; wherein, each corresponding The relationships are respectively: a corresponding relationship between at least one environment parameter, at least one area size parameter and a trained neural network.

In the image processing device provided in the embodiment of the present application, the image processing device can display the preview image captured by the camera of the electronic device, according to the target environment parameters of the preview image, and the size of the target area corresponding to the human face element in the preview image parameters, and determine the target neural network from the Q neural networks after training. Through this method, the image processing device can intelligently select a neural network that conforms to the current environmental parameters and face size for image enhancement processing based on the environmental parameters when the image is captured and the size of the human face in the captured image, thereby greatly Improves the effect and performance of face enhancement on captured images.

The neural network training device and the image processing device in the embodiments of the present application may be devices, or components, integrated circuits, or chips in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a handheld computer, a vehicle electronic device, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant). assistant, PDA), etc., non-mobile electronic devices can be servers, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (television, TV), teller machine or self-service machine, etc., this application Examples are not specifically limited.

The neural network training device and the image processing device in the embodiment of the present application may be devices with an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, which are not specifically limited in this embodiment of the present application.

The neural network training device provided by the embodiment of the present application can realize various processes realized by the method embodiments in FIG. 1 to FIG. 5 , and details are not repeated here to avoid repetition. The image processing apparatus provided in the embodiment of the present application can implement various processes implemented by the method embodiments in FIG. 6 and FIG. 7 , and details are not repeated here to avoid repetition.

Optionally, as shown in FIG. 9 , the embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and programs or instructions stored in the memory 802 and operable on the processor 801. When the programs or instructions are executed by the processor 801, the various processes of the above-mentioned neural network training method or the above-mentioned image processing method embodiments can be achieved, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

FIG. 10 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 100 includes but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110, etc. part.

Those skilled in the art can understand that the electronic device 100 can also include a power supply (such as a battery) for supplying power to various components, and the power supply can be logically connected to the processor 110 through the power management system, so that the management of charging, discharging, and function can be realized through the power management system. Consumption management and other functions. The structure of the electronic device shown in FIG. 10 does not constitute a limitation to the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange different components, and details will not be repeated here. .

Wherein, the above-mentioned processor 110 is configured to determine N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M , N are both positive integers; the processor 110 is configured to perform degradation processing on the first image corresponding to each degradation degree based on each degradation degree, to obtain the second image corresponding to each degradation degree above, each The second images respectively correspond to a first image; the processor 110 is configured to generate a sample set based on the first image and the second image corresponding to each degree of degradation, and obtain N sample sets; the processor 110, It is used to train Q neural networks respectively based on the above N sample sets, one sample set corresponds to at least one neural network, and Q is a positive integer.

Optionally, in the embodiment of the present application, the above-mentioned processor 110 is specifically configured to determine N target parameter ranges corresponding to the above-mentioned M environmental parameters from the X parameter ranges, and each parameter range corresponds to a degradation degree , one target parameter range corresponds to at least one environmental parameter; the above-mentioned processor 110 is specifically configured to determine the degradation degree corresponding to the i-th target parameter range in the N target parameter ranges as the i-th degradation degree; wherein, the above-mentioned The range of the i target parameter is: according to the order of the corresponding degradation degree from small to large, the ith target parameter range after sorting the above N target parameter ranges, i is a positive integer; the above i-th target parameter range corresponds to The environment parameter is smaller than the environment parameter corresponding to the i+1th target parameter range.

Optionally, in this embodiment of the present application, each of the M first images includes at least one human face element; the processor 110 is configured to obtain the human face element in the M first images Corresponding R area size parameters, each area size parameter is: a size parameter of the area where the face element in the first image is located, R is a positive integer, and R is greater than or equal to M; the processor 110 also It is used to determine the aforementioned Q neural networks according to the aforementioned M environmental parameters and the aforementioned R area size parameters; wherein, one neural network corresponds to: at least one environmental parameter and at least one area size parameter.

The above-mentioned processor 110 is specifically configured to determine the first neural network according to the first environment parameter and the first area size parameter; wherein, the above-mentioned first environment parameter is: the first image in any one of the above-mentioned N sample sets Among the environmental parameters; the above-mentioned first area size parameter is: among the area size parameters corresponding to the first image in any of the above-mentioned sample sets;

In the electronic device provided in the embodiment of the present application, the electronic device determines N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, and one degradation degree corresponds to at least one of the first images. An image, where M and N are both positive integers; and using each degree of degradation, degrade the first image corresponding to each degree of degradation to obtain a second image corresponding to each degree of degradation, and each second image is respectively Corresponding to a first image, then, based on the above-mentioned first image and second image corresponding to each degree of degradation, generate a sample set respectively, and obtain N sample sets, and finally, based on the above-mentioned N sample sets, Q nerves The network is trained, a sample set corresponds to at least one neural network, and Q is a positive integer. Through this method, the electronic device can degrade the first image under different environmental parameters corresponding to the degree of degradation (that is, there is a difference), so as to obtain the corresponding degraded image (that is, the second image) after the first image is degraded differently. , so as to construct a training sample set that conforms to the shooting quality under various environmental parameters (that is, different degrees of degradation), and then obtain a neural network suitable for targeted processing of captured images under different environmental parameters, and improve the accuracy of the image. Enhanced processing.

Alternatively, the above-mentioned processor 110 is configured to, in the case of displaying a preview image collected by a camera of an electronic device, according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image, from the trained Q In the first neural network, the target neural network is determined; the third image taken by the camera is input to the target neural network determined by the determination module 701 for image processing, and the processed image is obtained. The above-mentioned third image is: the camera shoots within a predetermined time period The predetermined time period is the time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.

Optionally, in the embodiment of the present application, the processor 110 is configured to acquire target environment parameters and target area size parameters corresponding to the face elements when the preview image contains face elements.

Optionally, in the embodiment of the present application, the above-mentioned processor 110 is specifically configured to determine the target neural network corresponding to the target environment parameter and the target area size parameter based on the pre-stored Q correspondences; wherein, each correspondence is respectively is: a corresponding relationship between at least one environment parameter, at least one region size parameter and a trained neural network.

In the electronic device provided in the embodiment of the present application, the electronic device may display the preview image captured by the camera of the electronic device, according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image, A target neural network is determined from the Q neural networks after training. Through this method, the electronic device can intelligently select a neural network that conforms to the current environmental parameters and face size for image enhancement processing based on the environmental parameters and the size of the human face in the captured image, thereby greatly improving Effects and performance of face enhancements on captured images.

It should be understood that, in the embodiment of the present application, the input unit 104 may include a graphics processing unit (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processing unit 1041 is used by the image capturing device ( Such as the image data of the still picture or video obtained by the camera) for processing. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 107 includes a touch panel 1071 and other input devices 1072 . The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts, a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here. Memory 109 may be used to store software programs as well as various data, including but not limited to application programs and operating systems. The processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, and the modem processor mainly processes wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 110 .

The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, each process of the above-mentioned neural network training method embodiment is realized, or the above-mentioned image Each process in the embodiment of the processing method can achieve the same technical effect, so in order to avoid repetition, details are not repeated here.

Wherein, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer readable storage medium, such as computer read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.

The embodiment of the present application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the above-mentioned embodiment of the neural network training method Each process, or each process of the above image processing method embodiment, and can achieve the same technical effect, in order to avoid repetition, it is not repeated here.

It should be understood that the chips mentioned in the embodiments of the present application may also be called system-on-chip, system-on-chip, system-on-a-chip, or system-on-a-chip.

The embodiment of the present application provides a computer program product, the program product is stored in a non-volatile storage medium, and the program product is executed by at least one processor to realize the various processes of the above-mentioned neural network training method embodiment, or the above-mentioned image Each process of the embodiment of the method is processed, and the same technical effect can be achieved.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. Functions are performed, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product in essence or the part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to enable a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present application.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

Claims

A neural network training method, said method comprising:

According to the M environmental parameters of the acquired M first images, N degradation degrees are determined, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, and M and N are both positive integers;

Based on each degree of degradation, performing degradation processing on the first image corresponding to each degree of degradation to obtain a second image corresponding to each degree of degradation, and each second image corresponds to a first image;

Generate a sample set based on the first image and the second image corresponding to each degree of degradation to obtain N sample sets;

Based on the N sample sets, Q neural networks are respectively trained, one sample set corresponds to at least one neural network, and Q is a positive integer.
The method according to claim 1, wherein said determining N degradation degrees according to the M environmental parameters of the acquired M first images comprises:

From the X parameter ranges, determine N target parameter ranges corresponding to the M environmental parameters, each parameter range corresponds to a degree of degradation, and one target parameter range corresponds to at least one environmental parameter;

The degradation degree corresponding to the i-th target parameter range in the N target parameter ranges is determined as the i-th degradation degree.
The method according to claim 1, wherein each of the M first images includes at least one face element;

Before said training of Q neural networks based on said N sample sets, said method also includes:

Obtain R area size parameters corresponding to the face elements in the M first images, each area size parameter is: a size parameter of the area where the face elements in the first image are located, and R is a positive integer , R is greater than or equal to M;

determining the Q neural networks according to the M environmental parameters and the R area size parameters;

Wherein, a neural network corresponds to: at least one environment parameter and at least one area size parameter.
The method according to claim 3, wherein, among the Q neural networks, the target parameter corresponding to the neural network with high complexity is greater than the target parameter corresponding to the neural network with small complexity;

Wherein, the target parameters include: environment parameters and area size parameters.
The method according to claim 3, wherein each sample set corresponds to at least two neural networks;

The determining the Q neural networks according to the M environmental parameters and the R area size parameters includes:

determining a first neural network according to the first environment parameter and the first area size parameter;

Wherein, the first environmental parameter is: among the environmental parameters of the first image in any one of the N sample sets; the first area size parameter is: the first image in any one of the sample sets In the corresponding area size parameter;

The first neural network is: one of the neural networks corresponding to any one sample set.
An image processing method applied to an electronic device, the electronic device comprising Q neural networks trained by the neural network training method according to any one of claims 1 to 5, where Q is a positive integer; the method includes :

In the case of displaying the preview image collected by the camera of the electronic device, according to the target environment parameters of the preview image and the target area size parameters corresponding to the face elements in the preview image, Q neural networks after training , determine the target neural network;

Inputting the third image captured by the camera into the target neural network for image processing to obtain a processed image, the third image is: an image captured by the camera within a predetermined time period, and the predetermined time period is : the time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.
The method according to claim 6, wherein, according to the target environment parameters of the preview image, and the target area size parameters corresponding to the human face elements in the preview image, it is determined from Q neural networks after training Before producing the target neural network, the method also includes:

In the case that the preview image contains a human face element, the target environment parameter and the target area size parameter corresponding to the human face element are acquired.
The method according to claim 6, wherein, according to the target environment parameters of the preview image, and the target area size parameters corresponding to the human face elements in the preview image, it is determined from Q neural networks after training The target neural network, including:

Based on the pre-stored Q correspondences, determine the target neural network corresponding to the target environment parameter and the target area size parameter;

Wherein, each corresponding relationship is respectively: a corresponding relationship between at least one environment parameter, at least one region size parameter and a trained neural network.
A neural network training device, said device comprising: a determination module, a processing module, a generation module and a training module, wherein:

The determination module is configured to determine N degradation degrees according to the acquired M environmental parameters of the M first images, one degradation degree corresponds to at least one environmental parameter, one degradation degree corresponds to at least one first image, M, N are positive integers;

The processing module is configured to perform degradation processing on the first image corresponding to each degradation degree based on each degradation degree determined by the determination module, to obtain a second image corresponding to each degradation degree, and each The second images respectively correspond to a first image;

The generation module is configured to generate a sample set based on the first image and the second image corresponding to each degree of degradation obtained by the processing module to obtain N sample sets;

The training module is configured to respectively train Q neural networks based on the N sample sets obtained by the generating module, one sample set corresponds to at least one neural network, and Q is a positive integer.
An image processing device, wherein the device comprises Q neural networks trained by the neural network training method according to any one of claims 1 to 5, and Q is a positive integer; the device comprises: a determination module and processing module, where:

The determination module is configured to, in the case of displaying a preview image collected by the camera of the electronic device, according to the target environment parameter of the preview image and the target area size parameter corresponding to the human face element in the preview image, from Determine the target neural network among the Q neural networks after training;

The processing module is configured to input the third image captured by the camera into the target neural network determined by the determination module for image processing to obtain a processed image, and the third image is: The predetermined time period is a time period between the moment when the camera captures the preview image and the moment when the camera stops capturing the preview image.
An electronic device, comprising a processor, a memory, and a program or instruction stored on the memory and operable on the processor, when the program or instruction is executed by the processor, claims 1-5 are realized The step of the neural network training method described in any one, or the step of the image processing method as described in any one of claims 6-8.
A readable storage medium, storing programs or instructions on the readable storage medium, and implementing the steps of the neural network training method according to any one of claims 1-5 when the programs or instructions are executed by the processor, or The steps of the image processing method according to any one of claims 6-8.
A neural network training device, characterized in that the device is configured to execute the neural network training method according to any one of claims 1 to 5.
An image processing device, characterized in that the device is configured to execute the image processing method according to any one of claims 6 to 8.
A computer program product, characterized in that the program product is executed by at least one processor to implement the neural network training method according to any one of claims 1 to 5, or according to any one of claims 6-8 The image processing method described above.
A chip, characterized in that the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the process described in any one of claims 1 to 5. The neural network training method described above, or the image processing method as described in any one of claims 6-8.