WO2022222652A1

WO2022222652A1 - Image processing method and apparatus, storage medium, device, and model training method

Info

Publication number: WO2022222652A1
Application number: PCT/CN2022/081257
Authority: WO
Inventors: 孙亚锋
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-04-23
Filing date: 2022-03-16
Publication date: 2022-10-27
Also published as: CN115238884A

Abstract

Embodiments of the present application disclose an image processing method and apparatus, a storage medium, a device, and a model training method. The image processing method comprises: obtaining RAW image data by means of an image sensor; and processing the RAW image data by means of an AI processor, wherein the AI processor comprises a preset neural network model, and the preset neural network model is obtained by training according to preset RAW image data and first label information corresponding to the preset RAW image data.

Description

Image processing method, device, storage medium, device, and model training method

This application claims the priority of the Chinese patent application filed on April 23, 2021 with the application number 202110443987.4 and the application name "image processing method, device, storage medium, equipment and model training method", the entire content of which is Incorporated herein by reference.

technical field

In the inference process of neural network-based AI (Artificial Intelligence) chips, such as in the process of target recognition, special software and hardware modules are required to convert the images collected by the image sensor into the original RAW format (the image sensor will capture the The light source signal is converted into the original data of the digital signal) into RGB (Red, Green, Blue, red, green, blue color mode) format, and then input into the neural network model for processing. In the whole processing process, due to the existence of RAW format and The conversion step of RGB format data leads to low processing efficiency.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image processing method, apparatus, storage medium, device, and model training method, which can improve image processing efficiency.

In a first aspect, the embodiments of the present application provide an image processing method, including:

Obtain RAW image data through the image sensor;

The RAW image data is processed by an AI processor, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and the first RAW image data corresponding to the preset RAW image data. A label information is trained.

In a second aspect, an embodiment of the present application also provides a model training method, including:

Obtain the preset RAW image data as the input data of the neural network model;

Acquiring first label information corresponding to the preset RAW image data as output data of the neural network model;

The preset neural network model is trained according to the input data and the output data to determine model parameters.

In a third aspect, an embodiment of the present application further provides an image processing apparatus, including:

The image acquisition module is used to acquire RAW image data through the image sensor;

The image processing module is used to process the RAW image data through an AI processor, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and preset The first label information corresponding to the RAW image data is obtained by training.

In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program runs on a computer, the computer is made to execute the method provided by any embodiment of the present application. an image processing method; or, when the computer program runs on a computer, causing the computer to execute the neural network model training method provided by any embodiment of the present application.

In a fifth aspect, the embodiments of the present application further provide an electronic device,

including an image sensor, and an AI processor connected to the image sensor;

the image sensor for acquiring RAW image data;

The AI processor is used to process the RAW image data, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and the preset RAW image. The first label information corresponding to the data is obtained by training.

In a sixth aspect, an embodiment of the present application further provides an electronic device, including a processor and a memory, the memory has a computer program, and the processor is used to execute the computer program provided by any embodiment of the present application by invoking the computer program. image processing method;

Alternatively, the processor is configured to execute the neural network model training method provided by any embodiment of the present application by invoking the computer program.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic flowchart of a first type of image processing method provided by an embodiment of the present application.

FIG. 2 is a schematic diagram of a first structure of an electronic device provided by an embodiment of the present application.

FIG. 3 is a schematic diagram of a second structure of an electronic device according to an embodiment of the present application.

FIG. 4 is a schematic diagram of an operation manner of a convolution kernel in a preset neural network model in an embodiment of the present application.

FIG. 5 is a schematic diagram of a RAW image data in an embodiment of the present application.

FIG. 6 is a schematic diagram of pixel separation in an image processing method provided by an embodiment of the present application. .

FIG. 7 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application.

FIG. 8 is a schematic diagram of a third structure of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art under the premise of not paying creative work belong to the protection scope of the present application.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The embodiment of the present application provides an image processing method, and the execution body of the image processing method may be the image processing apparatus provided by the embodiment of the present application, or an electronic device integrated with the image processing apparatus, wherein the image processing apparatus may adopt hardware or implemented in software. The electronic device may be a smartphone, a tablet computer, a palmtop computer, a notebook computer, or a desktop computer, a vehicle terminal, and other devices.

The application provides an image processing method, including:

Obtain RAW image data through the image sensor;

In some embodiments, the RAW image data is processed by an AI processor, including:

converting the RAW image data from single-channel RAW image data to multi-channel RAW image data; and

The multi-channel RAW image data is processed by an AI processor.

In some embodiments, the preset neural network model is an image classification model, an object detection model, an image segmentation model or an instance segmentation model.

In some embodiments, before acquiring the RAW image data through the image sensor, the method further includes:

Acquiring the preset RAW image data as the input data of the preset neural network model;

Acquiring the first label information corresponding to the preset RAW image data as output data of the preset neural network model; and

The neural network model is trained based on the input data and the output data to determine model parameters.

The present application also provides a neural network model training method, including:

Obtain the preset RAW image data as the input data of the neural network model;

In some embodiments, acquiring preset RAW image data as input data for the preset neural network model includes:

obtaining the first preset RGB image data;

Converting the first preset RGB image data into corresponding preset RAW image data according to a preset fully convolutional neural network, and using the preset RAW image data as input data of the preset neural network model;

Obtain the first label information corresponding to the preset RAW image data as the output data of the preset neural network model, including:

acquiring second label information corresponding to the first preset RGB image data;

The second label information of the first preset RGB image data is mapped to the preset RAW image data to obtain first label information, which is used as output data of the preset neural network model.

In some embodiments, acquiring the first label information corresponding to the preset RAW image data as the output data of the preset neural network model includes:

Acquiring preset RGB image data corresponding to the preset RAW image data, and second label information of the preset RGB image data;

The second label information of the preset RGB image data is mapped to the preset RAW image data to obtain the first label information, which is used as output data of model training.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of a first image processing method provided by an embodiment of the present application. The specific process of the image processing method provided by the embodiment of the present application may be as follows:

101. Acquire RAW image data through an image sensor.

The embodiments of the present application can be applied to applications in the field of computer vision, including but not limited to classification, localization, semantic segmentation, and instance segmentation. Please refer to FIG. 2 , which is a schematic diagram of a first structure of an electronic device according to an embodiment of the present application. The electronic device is provided with an image sensor and an AI processor. The AI processor is a processor capable of performing neural network operations. For example, the operations of the neural network model can be performed by any processor capable of executing neural networks, such as GPU (Graphics Processing Unit, graphics processor), DSP (Digital Signal Process, digital signal processor), NPU (Neural-network Processing Unit, embedded AI processor), CPU (central processing unit, central processing unit), etc. These processors can be considered as AI processors.

Please refer to FIG. 3 , which is a schematic diagram of a second structure of an electronic device provided by an embodiment of the present application. FIG. 3 depicts a specific implementation of the electronic device provided by the embodiment of the present application, wherein the electronic device includes a data acquisition module, a data receiving module, and an AI processor, and the data acquisition module includes one or more image sensors (in the figure Multiple image sensors are shown), the RAW image data collected by the image sensor is transmitted to the data receiving module through the transmitting-end media access controller and the transmitting-end physical layer device, and the data receiving module includes the receiving-end physical layer device and the receiving-end media connected in turn. The access controller, the first buffer unit and the data interface unit, and the data receiving module transmits the received RAW image data to the AI processor for processing. The AI processor includes an AI computing unit, a first cache unit, and a storage unit, and the storage unit may be a DDR. The first buffer unit is used for buffering the received RAW image data, the storage unit is used for storing the preset neural network model, and the AI computing unit is used for processing the RAW image data through the preset neural network model.

The AI processor stores a pre-trained preset neural network model, and the preset neural network model may be an image classification model, a target detection model, an image segmentation model or an instance segmentation model. The preset neural network model includes a convolution layer. Please refer to FIG. 4 . FIG. 4 is a schematic diagram of an operation manner of a convolution kernel in the preset neural network model in an embodiment of the present application. The 6×6 input feature map is subjected to the convolution operation of a convolution kernel with a size of 2×2 and a stride of 2 to obtain a 3×3 output feature map.

The electronic device shoots the scene to be detected through the image sensor to generate RAW image data. Among them, the RAW image (that is, the Bayer image with the suffix .raw format) data is CMOS (Complementary Metal Oxide Semiconductor, complementary metal oxide semiconductor) or CCD (Charge Coupled Device, charge coupled device image sensor) The captured light source signal is converted into the original data of the digital signal. The original RAW image data has a higher gray level and preserves the complete data information. Among them, the three colors of RGB in the RAW image data can be arranged in various ways, such as RGGB, GRBG, GBRG, and BGGR. Alternatively, if the filter in the CCD is a complementary color filter, a RAW image in which the colors of the four channels of CMYB (cyan, magenta, yellow, and black) are arranged can be obtained. Please refer to FIG. 5 , which is a schematic diagram of RAW image data according to an embodiment of the present application.

After the RAW image to be detected is acquired, the RAW image to be detected is transmitted to the AI processor, and the RAW image to be detected is used as input data of the preset neural network model for image processing.

102. Process the RAW image data by an AI processor, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and corresponds to the preset RAW image data The first label information of the training is obtained.

After the RAW image to be detected is obtained, the RAW image to be detected is input into a preset neural network model for calculation. For example, if the preset neural network model is a classification model, the detected object category can be output. For example, the electronic device is a vehicle identification device, and the vehicle identification device can be an in-vehicle device. The image in front of the vehicle is captured in real time by an image sensor to obtain a Raw image to be detected, and the raw image to be detected is transmitted to the AI processor for image processing. Determine if there is a vehicle ahead.

The preset neural network model is obtained by performing model training according to the preset RAW image data and the first label information corresponding to the preset RAW image data. For example, acquiring preset RAW image data as input data of a preset neural network model; acquiring first label information corresponding to the preset RAW image data as output data of the preset neural network model; according to the input The data and the output data train the preset neural network model to determine model parameters.

In the embodiment of the present application, in the application stage of the preset neural network model, the format of the input data used is the RAW format. The default neural network model needs to keep the format and size of the input data the same during the training phase and the application phase. Based on this, the preset neural network model also needs to use RAW images as input data for model training during the training phase.

In conventional datasets, the format of image data used for model training is generally RGB format, which is a visual image format, and users can directly add label information to samples as needed. In addition, even if images in RAW format can be collected as input data for model training, they cannot be directly labeled due to the invisibility of RAW format. Because for computer equipment, RAW data records the most original image information output by the image sensor and cannot be directly used for display, so users cannot directly mark RAW data.

Based on the above reasons, the present application adopts different data processing methods for different data formats in the data set to obtain input data and output data for model training.

In some embodiments, acquiring preset RAW image data as input data for the preset neural network model includes: acquiring first preset RGB image data; The first preset RGB image data is converted into corresponding preset RAW image data, and the preset RAW image data is used as input data of the preset neural network model.

Obtaining the first label information corresponding to the preset RAW image data as the output data of the preset neural network model includes: obtaining second label information corresponding to the first preset RGB image data; The second label information of a preset RGB image data is mapped to the preset RAW image data, and the first label information is obtained, which is used as the output data of the preset neural network model.

In this embodiment, a data set is obtained, the data set includes several preset RGB image data, and the preset RGB image data is converted according to the pre-trained full preset neural network model to obtain each preset RGB image data The corresponding preset RAW image data.

A pixel in a RAW image has only one color information of the three colors of RGB, while a pixel in the RGB image data contains three color information of RGB. In this embodiment of the present application, a fully convolutional network is pre-built. The preset fully convolutional neural network includes a fully convolutional part and a deconvolutional part. After the input data is subjected to the convolution operation of the fully convolutional part, depth features are obtained, and then After the operation of the deconvolution part, the output image of the same size as the original input image can be obtained. Obtain several RAW images and the corresponding RGB image data processed by ISP to form several image pairs, use the RGB image data as input data and the RAW image as output data to train the fully convolutional neural network, and determine the network parameters , the network parameters learned by the trained network represent the conversion relationship between the RGB image data and the RAW image, and the preset RGB image data is input into the network for calculation, and the corresponding preset RAW image data of the same size can be obtained as a model input data for training.

The preset RGB image data can be directly displayed on the computer, and the user can mark the preset RGB image data correspondingly according to the type of the preset neural network model. For example, the preset neural network model is the target detection model, and the label information can be the bounding box of the target subject in the sample image. The user manually marks the rectangular bounding box of the image to be recognized on the image, and the corresponding position coordinates and Size information, as label information. For another example, if the convolutional neural network is an image classification model, the label information is the category information of the target subject in the sample image.

Obtain the second label information corresponding to the preset RGB image data. The preset RGB image data and the preset RAW image data have the same image size. For the label information containing position information, such as a bounding box, the preset RGB image data can be The second label information on the data is directly mapped to the preset RAW image data according to its position to obtain the corresponding first label information. In this way, the indirect marking of the preset RAW image data can be completed. The preset neural network model is trained by preset RAW image data and its first label information to determine model parameters.

Or, in other embodiments, acquiring the first label information corresponding to the preset RAW image data as the output data of the preset neural network model includes: acquiring the preset corresponding to the preset RAW image data RGB image data, and second label information of the preset RGB image data; mapping the second label information of the preset RGB image data to the preset RAW image data to obtain the first label information, which is used as model training output data.

In this embodiment, the data set used for model training includes the preset RAW image data, but the sample RAAW image cannot be marked directly, so the preset RAW image data is converted into the corresponding preset RGB image data and then marked to obtain the first RAW image data. The second label information is mapped to the preset RAW image data to obtain the first label information, which is used as the output data of model training. In this way, the indirect marking of the preset RAW image data can be completed. The preset neural network model is trained by preset RAW image data and its first label information to determine model parameters.

During specific implementation, the present application is not limited by the execution order of the described steps, and certain steps may also be performed in other sequences or simultaneously under the condition of no conflict.

It can be seen from the above that, in the image processing method provided by the embodiment of the present application, when performing image processing, the RAW image data is obtained through the image sensor; the RAW image data is processed through the AI processor, because the preset neural network model is pre- Assuming that the RAW image data and the corresponding first label information are obtained by training, in the image processing stage, the original RAW image output by the image sensor can be used for direct detection without ISP processing, which improves the efficiency of image processing.

In some embodiments, processing the RAW image data by the AI processor includes: converting the RAW image data from single-channel RAW image data to multi-channel RAW image data; and processing the multi-channel RAW image data by the AI processor Channel RAW image data for processing.

The original RAW image to be detected is a single-channel image. In order to improve the operation speed, before inputting it into the preset neural network model for operation, it can be processed by pixel separation to obtain multi-channel RAW image data. The feature map on the channel corresponds to only one color.

The specific pixel separation manner is determined according to the arrangement manner of the pixels in the RAW image to be detected. Please refer to FIG. 6, which is a schematic diagram of pixel separation in an image processing method provided by an embodiment of the present application. In this embodiment, the pixels in the RAW image to be detected are arranged in the order of RGGB. After pixel separation, a channel map of four channels is obtained, and the four channels are arranged in the order of R, G, G, and B. . In the summary of other embodiments, the pixel arrangement order in the RAW image to be detected may be GRBG and GBRG, then the pixel arrangement is adjusted to RGGB by deleting the first row of pixels or the first column of pixels, and then the pixel is adjusted to RGGB. Separation processing, so that the channels after separation are arranged in the order of R, G, G, B. Assuming that the size of the RAW image to be detected is W*H, after pixel separation processing, the size of the feature maps on the four channels is W/2*H/2.

It can be understood that if the RAW image to be detected needs to be converted in the model training stage in the image stage, the input data needs to be subjected to the same pixel separation processing in the model training stage, and the channel order should be consistent.

An embodiment of the present application further provides a model training method, including: acquiring preset RAW image data as input data of a neural network model; acquiring first label information corresponding to the preset RAW image data as the neural network model output data; train the preset neural network model according to the input data and the output data to determine model parameters.

In some embodiments, acquiring preset RAW image data as input data for the preset neural network model includes: acquiring first preset RGB image data; The first preset RGB image data is converted into corresponding preset RAW image data, and the preset RAW image data is used as input data of the preset neural network model. Obtaining the first label information corresponding to the preset RAW image data as the output data of the preset neural network model includes: obtaining second label information corresponding to the first preset RGB image data; The second label information of a preset RGB image data is mapped to the preset RAW image data, and the first label information is obtained, which is used as the output data of the preset neural network model.

In some embodiments, acquiring first label information corresponding to the preset RAW image data as output data of the preset neural network model includes: acquiring preset RGB image data corresponding to the preset RAW image data , and the second label information of the preset RGB image data; map the second label information of the preset RGB image data to the preset RAW image data to obtain the first label information as the output data of model training .

The model training methods provided in the above embodiments can be applied to a server. After the server performs model training, it updates model parameters, and sends the updated model parameters to the electronic device, so that the electronic device updates its stored presets. Neural network model.

In the model training method provided by this embodiment, the RAW image is used as the input data for model training, and the RGB image data corresponding to the RAW image is marked, and then the label information is mapped, so as to obtain the preset RAW image data with the label information, Carry out model training. Since the preset neural network model is obtained by training the preset RAW image data and the corresponding label information, in the image processing stage, the original RAW image output by the image sensor can be used for direct detection without ISP processing. Improve image processing efficiency.

The present application also provides an image processing device, comprising:

In some embodiments, the image acquisition module for converting the RAW image data from single-channel RAW image data to multi-channel RAW image data; and

The multi-channel RAW image data is processed by an AI processor.

In some embodiments, the apparatus further includes:

a model training module for acquiring the preset RAW image data as input data for the preset neural network model;

The embodiment of the present application also provides a neural network model training device, including:

In some embodiments, the model training module is also used to:

obtaining the first preset RGB image data;

In some embodiments, the model training module is also used to:

In an embodiment, an image processing apparatus is also provided. Please refer to FIG. 7 , which is a schematic structural diagram of an image processing apparatus 300 according to an embodiment of the present application. The image processing apparatus 300 is applied to electronic equipment, and the image processing apparatus 300 includes an image acquisition module 301 and an image processing module 302, as follows:

An image acquisition module 301, configured to acquire RAW image data through an image sensor;

The image processing module 302 is configured to process the RAW image data through an AI processor, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and the preset RAW image data. It is assumed that the first label information corresponding to the RAW image data is obtained by training.

In some embodiments, the image processing module 302 is further configured to convert the RAW image data from single-channel RAW image data to multi-channel RAW image data; and process the multi-channel RAW image data through an AI processor.

In some embodiments, the image processing apparatus 300 further includes:

In some embodiments, the model training module is further configured to obtain the first preset RGB image data;

According to the pre-trained fully convolutional neural network, the first preset RGB image data is converted into corresponding preset RAW image data, and the preset RAW image data is used as the input data of the preset neural network model ;

and obtaining second label information corresponding to the first preset RGB image data; mapping the second label information of the first preset RGB image data to the preset RAW image data to obtain the first label information, as the output data of the preset neural network model.

In some embodiments, the model training module is further configured to acquire preset RGB image data corresponding to the preset RAW image data, and second label information of the preset RGB image data;

It should be noted that the image processing apparatus provided in the embodiments of the present application and the image processing methods in the above embodiments belong to the same concept, and any method provided in the image processing method embodiments can be implemented by the image processing apparatus. For details of the process, please refer to the embodiment of the image processing method, which will not be repeated here.

It can be seen from the above that the image processing device proposed in the embodiments of the present application acquires RAW image data through an image sensor when performing image processing, and processes the RAW image data through an AI processor, because the preset neural network model Assuming that the RAW image data and the corresponding first label information are obtained by training, in the image processing stage, the original RAW image output by the image sensor can be used for direct detection without ISP processing, which improves the efficiency of image processing.

An embodiment of the present application further provides an electronic device, and the electronic device may be a terminal, and the terminal may be a smart phone, a tablet computer, a notebook computer, a touch screen, a game console, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and other terminal equipment. As shown in FIG. 8 , FIG. 8 is a third schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 400 includes a processor 401 having one or more processing cores, a memory 402 having one or more computer-readable storage media, and a computer program stored on the memory 402 and executable on the processor. The processor 401 is electrically connected to the memory 402 . Those skilled in the art can understand that the structure of the electronic device shown in the figures does not constitute a limitation on the electronic device, and may include more or less components than those shown in the figures, or combine some components, or arrange different components.

The processor 401 is the control center of the electronic device 400, uses various interfaces and lines to connect various parts of the entire electronic device 400, runs or loads the software programs and/or modules stored in the memory 402, and calls the software programs and/or modules stored in the memory 402. to perform various functions of the electronic device 400 and process data, so as to monitor the electronic device 400 as a whole.

In this embodiment of the present application, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 executes the instructions stored in the memory. 402 application in order to achieve various functions:

Obtain RAW image data through the image sensor;

For the specific implementation of the above operations, reference may be made to the foregoing embodiments, and details are not described herein again.

Optionally, as shown in FIG. 5 , the electronic device 400 further includes: a touch display screen 403 , a radio frequency circuit 404 , an audio circuit 405 , an input unit 406 and a power supply 407 . The processor 401 is electrically connected to the touch display screen 403 , the radio frequency circuit 404 , the audio circuit 405 , the input unit 406 and the power supply 407 , respectively. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 5 does not constitute a limitation on the electronic device, and may include more or less components than the one shown, or combine some components, or arrange different components.

The touch screen 403 can be used to display a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface. The touch display 403 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light emitting diode (OLED, Organic Light-Emitting Diode), and the like. The touch panel can be used to collect the user's touch operations on or near it (such as the user's operations on or near the touch panel using a finger, stylus, etc., any suitable object or accessory), and generate corresponding operations instruction, and the operation instruction executes the corresponding program. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the touch controller. To the processor 401, and can receive the command sent by the processor 401 and execute it. The touch panel can cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor 401 to determine the type of the touch event, and then the processor 401 provides the display panel according to the type of the touch event. Corresponding visual output. In this embodiment of the present application, the touch panel and the display panel may be integrated into the touch display screen 403 to implement input and output functions. However, in some embodiments, the touch panel and the touch panel may be used as two independent components to implement input and output functions. That is, the touch display screen 403 can also be used as a part of the input unit 406 to realize the input function.

The radio frequency circuit 404 can be used to send and receive radio frequency signals, so as to establish wireless communication with the network device or other electronic devices through wireless communication, and to send and receive signals with the network device or other electronic devices.

The audio circuit 405 may be used to provide an audio interface between the user and the electronic device through speakers and microphones. The audio circuit 405 can convert the received audio data into an electrical signal, and transmit it to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is converted after being received by the audio circuit 405. As audio data, the audio data is output to the processor 401 for processing, and then sent to, for example, another electronic device via the radio frequency circuit 404, or the audio data is output to the memory 402 for further processing. The audio circuit 405 may also include an earphone jack to provide for communication of peripheral headphones with the electronic device.

The input unit 406 can be used to receive input numbers, character information or user characteristic information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control .

Power supply 407 is used to power various components of electronic device 400 . Optionally, the power supply 407 may be logically connected to the processor 401 through a power management system, so that functions such as charging, discharging, and power consumption management are implemented through the power management system. The power source 407 may also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

Although not shown in FIG. 5 , the electronic device 400 may further include an image sensor, a sensor, a Wi-Fi module, a Bluetooth module, and the like, which will not be repeated here.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

As can be seen from the above, the electronic device provided in this embodiment acquires RAW image data through an image sensor when performing image processing; and processes the RAW image data through an AI processor, because the preset neural network model is obtained by preset RAW image data. The image data and the corresponding first label information are obtained by training. In the image processing stage, the original RAW image output by the image sensor can be used for direct detection without ISP processing, which improves the efficiency of image processing.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructions, or by instructions that control relevant hardware, and the instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor.

To this end, the embodiments of the present application provide a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer programs can be loaded by a processor to execute any of the image processing methods provided by the embodiments of the present application. A step of. For example, the computer program may perform the following steps:

Obtain RAW image data through the image sensor;

Or, in another embodiment, the computer program can be loaded by the processor to execute the steps in any of the neural network model training methods provided in the embodiments of the present application. For example, the computer program may perform the following steps:

Acquiring preset RAW image data as input data of the neural network model; acquiring first label information corresponding to the preset RAW image data as output data of the neural network model; according to the input data and the output data The preset neural network model is trained to determine model parameters.

Wherein, the storage medium may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

Since the computer program stored in the storage medium can execute the steps in any image processing method provided by the embodiments of the present application, it is possible to realize the realization of any image processing method provided by the embodiments of the present application. For the beneficial effects, please refer to the previous embodiments for details, which will not be repeated here.

The image processing method, device, storage medium, device, and model training method provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described with specific examples. The above embodiments The description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, there will be changes in the specific embodiments and application scope. In summary, the above , the contents of this specification should not be construed as limiting the application.

Claims

An image processing method, comprising:

Obtain RAW image data through the image sensor;

The RAW image data is processed by an AI processor, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and the first RAW image data corresponding to the preset RAW image data. A label information is trained.
The method of claim 1, wherein processing the RAW image data by an AI processor comprises:

converting the RAW image data from single-channel RAW image data to multi-channel RAW image data; and

The multi-channel RAW image data is processed by an AI processor.
The method of claim 1, wherein the preset neural network model is an image classification model, a target detection model, an image segmentation model or an instance segmentation model.
The method according to any one of claims 1 to 3, wherein before acquiring the RAW image data through the image sensor, the method further comprises:

Acquiring the preset RAW image data as the input data of the preset neural network model;

Acquiring the first label information corresponding to the preset RAW image data as output data of the preset neural network model; and

The neural network model is trained based on the input data and the output data to determine model parameters.
A neural network model training method, comprising:

Obtain the preset RAW image data as the input data of the neural network model;

Acquiring first label information corresponding to the preset RAW image data as output data of the neural network model;

The preset neural network model is trained according to the input data and the output data to determine model parameters.
The method of claim 5, wherein acquiring preset RAW image data as the input data of the preset neural network model comprises:

obtaining the first preset RGB image data;

Converting the first preset RGB image data into corresponding preset RAW image data according to a preset fully convolutional neural network, and using the preset RAW image data as input data of the preset neural network model;

Obtain the first label information corresponding to the preset RAW image data as the output data of the preset neural network model, including:

acquiring second label information corresponding to the first preset RGB image data;

The second label information of the first preset RGB image data is mapped to the preset RAW image data to obtain first label information, which is used as output data of the preset neural network model.
The method according to claim 5, wherein obtaining the first label information corresponding to the preset RAW image data as the output data of the preset neural network model, comprising:

Acquiring preset RGB image data corresponding to the preset RAW image data, and second label information of the preset RGB image data;

The second label information of the preset RGB image data is mapped to the preset RAW image data to obtain the first label information, which is used as output data of model training.
An image processing device, comprising:

The image acquisition module is used to acquire RAW image data through the image sensor;

The image processing module is used to process the RAW image data through an AI processor, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and preset The first label information corresponding to the RAW image data is obtained by training.
The apparatus of claim 8, wherein the image acquisition module is configured to convert the RAW image data from single-channel RAW image data to multi-channel RAW image data; and

The multi-channel RAW image data is processed by an AI processor.
The apparatus of claim 8, wherein the preset neural network model is an image classification model, a target detection model, an image segmentation model or an instance segmentation model.
The apparatus of claim 8, wherein the apparatus further comprises:

a model training module for acquiring the preset RAW image data as input data for the preset neural network model;

Acquiring the first label information corresponding to the preset RAW image data as output data of the preset neural network model; and

The neural network model is trained based on the input data and the output data to determine model parameters.
A neural network model training device, comprising:

a model training module for acquiring the preset RAW image data as input data for the preset neural network model;

Obtaining the first label information corresponding to the preset RAW image data as the output data of the preset neural network model; And

The neural network model is trained based on the input data and the output data to determine model parameters.
The apparatus of claim 12, wherein the model training module is further used for:

obtaining the first preset RGB image data;

Converting the first preset RGB image data into corresponding preset RAW image data according to a preset fully convolutional neural network, and using the preset RAW image data as input data of the preset neural network model;

acquiring second label information corresponding to the first preset RGB image data;

The second label information of the first preset RGB image data is mapped to the preset RAW image data to obtain first label information, which is used as output data of the preset neural network model.
The apparatus of claim 12, wherein the model training module is further used for:

Acquiring preset RGB image data corresponding to the preset RAW image data, and second label information of the preset RGB image data;

The second label information of the preset RGB image data is mapped to the preset RAW image data to obtain the first label information, which is used as output data of model training.
A computer-readable storage medium on which a computer program is stored, wherein, when the computer program runs on a computer, the computer is caused to execute the image processing method according to any one of claims 1 to 4;

Alternatively, when the computer program runs on a computer, the computer is caused to execute the neural network model training method according to any one of claims 5 to 7.
An electronic device, including an image sensor, and an AI processor connected to the image sensor;

the image sensor for acquiring RAW image data;

The AI processor is used to process the RAW image data, wherein the AI processor includes a preset neural network model, and the preset neural network model is based on the preset RAW image data and the preset RAW image. The first label information corresponding to the data is obtained by training.
The electronic device of claim 16, wherein the image sensor is further configured to:

converting the RAW image data from single-channel RAW image data to multi-channel RAW image data; and

The multi-channel RAW image data is processed by an AI processor.
The electronic device according to claim 16, wherein the preset neural network model is an image classification model, a target detection model, an image segmentation model or an instance segmentation model.
The electronic device of claim 16, wherein the AI processor is further configured to:

Acquiring the preset RAW image data as the input data of the preset neural network model;

Acquiring the first label information corresponding to the preset RAW image data as output data of the preset neural network model; and

The neural network model is trained based on the input data and the output data to determine model parameters.
An electronic device, comprising a processor and a memory, wherein the memory stores a computer program, the processor is used to execute the image processing method according to any one of claims 1 to 4 by calling the computer program ;

The processor is configured to execute the neural network model training method according to any one of claims 5 to 7 by invoking the computer program.