CN115238884A

CN115238884A - Image processing method, image processing apparatus, storage medium, device, and model training method

Info

Publication number: CN115238884A
Application number: CN202110443987.4A
Authority: CN
Inventors: 孙亚锋
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-04-23
Filing date: 2021-04-23
Publication date: 2022-10-25
Also published as: WO2022222652A1

Abstract

The embodiment of the application discloses an image processing method, an image processing device, a storage medium, equipment and a model training method, wherein the image processing method comprises the following steps: acquiring RAW image data through an image sensor; and processing the RAW image data through an AI processor, wherein the AI processor comprises a preset neural network model, and the preset neural network model is obtained by training according to the preset RAW image data and first label information corresponding to the preset RAW image data. Because the preset neural network model is obtained by training the preset RAW image data and the corresponding label information, the original RAW image data output by the image sensor can be directly used for detection in the image processing stage, and the image processing efficiency is improved without ISP (internet service provider) processing.

Description

Image processing method, image processing apparatus, storage medium, device, and model training method

Technical Field

The present application relates to the field of electronic device technologies, and in particular, to an image processing method, an image processing apparatus, a storage medium, a device, and a model training method.

Background

In an inference process, for example, in a target recognition process, an AI (Artificial Intelligence) chip based on a neural network needs to convert an image acquired by an image sensor from an original RAW format (original data obtained by converting a captured light source signal into a digital signal by an image sensor) into an RGB (Red, green, blue, red, green, blue color mode) format through a special software and hardware module, and then input the RGB (Red, green, blue color mode) format into a neural network model for processing.

Disclosure of Invention

The embodiment of the application provides an image processing method, an image processing device, a storage medium, equipment and a model training method, and can improve the image processing efficiency.

In a first aspect, an embodiment of the present application provides an image processing method, including:

acquiring RAW image data through an image sensor;

and processing the RAW image data through an AI processor, wherein the AI processor comprises a preset neural network model, and the preset neural network model is obtained by training according to the preset RAW image data and first label information corresponding to the preset RAW image data.

In a second aspect, an embodiment of the present application further provides a model training method, including:

acquiring preset RAW image data as input data of a neural network model;

acquiring first label information corresponding to the preset RAW image data as output data of the neural network model;

and training the preset neural network model according to the input data and the output data to determine model parameters.

In a third aspect, an embodiment of the present application further provides an image processing apparatus, including:

the image acquisition module is used for acquiring RAW image data through the image sensor;

and the image processing module is used for processing the RAW image data through an AI (artificial intelligence) processor, wherein the AI processor comprises a preset neural network model, and the preset neural network model is obtained by training according to the preset RAW image data and first label information corresponding to the preset RAW image data.

In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program runs on a computer, the computer is caused to execute an image processing method as provided in any embodiment of the present application; alternatively, the computer program, when run on a computer, causes the computer to perform a neural network model training method as provided in any embodiment of the present application.

In a fifth aspect, embodiments of the present application further provide an electronic device,

the device comprises an image sensor and an AI processor connected with the image sensor;

the image sensor is used for acquiring RAW image data;

the AI processor is configured to process the RAW image data, and the AI processor includes a preset neural network model, where the preset neural network model is obtained by training according to preset RAW image data and first label information corresponding to the preset RAW image data.

In a sixth aspect, an embodiment of the present application further provides an electronic device, including a processor and a memory, where the memory has a computer program, and the processor is configured to execute the image processing method provided in any embodiment of the present application by calling the computer program;

alternatively, the processor is configured to execute the neural network model training method according to any embodiment of the present application by calling the computer program.

According to the technical scheme provided by the embodiment of the application, when image processing is carried out, RAW image data are obtained through an image sensor; the RAW image data are processed through the AI processor, and the preset neural network model is obtained through training of the preset RAW image data and the corresponding first label information, so that in an image processing stage, an original RAW image output by the image sensor can be directly detected without being processed by an ISP (internet service provider), and the image processing efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic flowchart of a first image processing method according to an embodiment of the present application.

Fig. 2 is a first structural schematic diagram of an electronic device according to an embodiment of the present application.

Fig. 3 is a schematic structural diagram of a second electronic device according to an embodiment of the present application.

Fig. 4 is a schematic diagram of an operation manner of a convolution kernel in a preset neural network model in the embodiment of the present application.

Fig. 5 is a schematic diagram of RAW image data in an embodiment of the present application.

Fig. 6 is a schematic diagram of pixel separation in an image processing method according to an embodiment of the present disclosure. .

Fig. 7 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application.

Fig. 8 is a third schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without inventive step, are within the scope of the present application.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

The embodiment of the present application provides an image processing method, and an execution subject of the image processing method may be the image processing apparatus provided in the embodiment of the present application, or an electronic device integrated with the image processing apparatus, where the image processing apparatus may be implemented in a hardware or software manner. The electronic device can be a smart phone, a tablet computer, a palm computer, a notebook computer, or a desktop computer, a vehicle-mounted terminal and other devices.

Referring to fig. 1, fig. 1 is a first flowchart illustrating an image processing method according to an embodiment of the present disclosure. The specific flow of the image processing method provided by the embodiment of the application can be as follows:

101. RAW image data is acquired by an image sensor.

The embodiment of the application can be applied to the application in the field of computer vision, including but not limited to classification, positioning, semantic segmentation and instance segmentation. Referring to fig. 2, fig. 2 is a schematic view of a first structure of an electronic device according to an embodiment of the present disclosure. The electronic device is provided with an image sensor and an AI processor. The AI processor is a processor capable of performing an operation of the Neural network, for example, the operation of the Neural network model may be performed by any processor capable of executing the Neural network, such as a GPU (Graphics Processing Unit), a DSP (Digital Signal processor), an NPU (embedded AI processor), a CPU (central Processing Unit), and the like, which may be considered as the AI processor.

Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to a second embodiment of the present disclosure. Fig. 3 depicts a specific implementation manner of an electronic device according to an embodiment of the present application, where the electronic device includes a data acquisition module, a data receiving module, and an AI processor, the data acquisition module includes one or more image sensors (a plurality of image sensors are shown in the figure), RAW image data acquired by the image sensors is transmitted to the data receiving module through a sending-end media access controller and a sending-end physical layer device, the data receiving module includes a receiving-end physical layer device, a receiving-end media access controller, a first buffer unit, and a data interface unit that are sequentially connected, and the data receiving module transmits the received RAW image data to the AI processor for processing. The AI processor includes an AI arithmetic unit, a first buffer unit and a memory unit, wherein the memory unit can be DDR. The first cache unit is used for caching received RAW image data, the storage unit is used for storing a preset neural network model, and the AI operation unit is used for processing the RAW image data through the preset neural network model.

The AI processor stores a pre-trained preset neural network model, which can be an image classification model, a target detection model, an image segmentation model or an instance segmentation model. Referring to fig. 4, fig. 4 is a schematic diagram illustrating an operation manner of a convolution kernel in the preset neural network model according to the embodiment of the present disclosure. The 6 × 6 input feature map is subjected to convolution operation with a convolution kernel of 2 × 2 in size and 2 steps, to obtain a 3 × 3 output feature map.

The electronic equipment shoots a scene to be detected through the image sensor to generate RAW image data. RAW image data (i.e., bayer image with the suffix of RAW format) is RAW data obtained by converting a captured light source signal into a digital signal by a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (charge coupled Device) image sensor, and the RAW image data has a higher gray scale level and stores complete data information. The three RGB colors in the RAW image data may be arranged in various ways, such as RGGB, GRBG, GBRG, and BGGR. Alternatively, if the filter in the CCD is a complementary color filter, a RAW image in which colors of four channels of CMYB (cyan, magenta, yellow, black) are arranged can be obtained. Referring to fig. 5, fig. 5 is a schematic diagram of RAW image data according to an embodiment of the present application.

And after the RAW image to be detected is acquired, transmitting the RAW image to be detected to an AI (Artificial intelligence) processor, and performing image processing by taking the RAW image to be detected as input data of a preset neural network model.

102. And processing the RAW image data through an AI processor, wherein the AI processor comprises a preset neural network model, and the preset neural network model is obtained by training according to the preset RAW image data and first label information corresponding to the preset RAW image data.

And after obtaining the RAW image to be detected, inputting the RAW image to be detected into a preset neural network model for calculation. For example, if the neural network model is preset as the classification model, the detected object class may be output. For example, the electronic device is a vehicle identification device, which may be a vehicle-mounted device, and the image sensor captures a picture in front of the vehicle in real time to obtain a Raw image to be detected, and transmits the Raw image to be detected to the AI processor for image processing to determine whether there is a vehicle in front.

The preset neural network model is obtained by performing model training according to preset RAW image data and first label information corresponding to the preset RAW image data. For example, acquiring preset RAW image data as input data of a preset neural network model; acquiring first label information corresponding to the preset RAW image data as output data of the preset neural network model; and training the preset neural network model according to the input data and the output data to determine model parameters.

In the embodiment of the present application, in the application stage of the preset neural network model, the format of the input data used is the RAW format. The preset neural network model needs to keep the format and the size of input data the same in a training stage and an application stage. Based on this, the preset neural network model also needs to be trained by using RAW images as input data in the training phase.

In a conventional data set, the format of image data used for model training is generally RGB, which is a visual image format, and a user can add label information directly on a sample as required. Furthermore, even if images in RAW format can be acquired as input data for model training, they cannot be directly labeled because of the invisibility of RAW format. Since RAW data records the most primitive image information output by an image sensor for a computer device and cannot be directly used for display, a user cannot directly mark the RAW data.

For the reasons, the input data and the output data of the model training are obtained by adopting different data processing modes aiming at different data formats in the data set.

In some embodiments, acquiring preset RAW image data as input data of the preset neural network model includes: acquiring first preset RGB image data; and converting the first preset RGB image data into corresponding preset RAW image data according to a pre-trained full-convolution neural network, and taking the preset RAW image data as input data of the preset neural network model.

Acquiring first tag information corresponding to the preset RAW image data, wherein the first tag information is used as output data of the preset neural network model and comprises the following steps: acquiring second label information corresponding to the first preset RGB image data; and mapping second label information of the first preset RGB image data to the preset RAW image data to obtain first label information serving as output data of the preset neural network model.

In this embodiment, a data set is obtained, where the data set includes a plurality of preset RGB image data, and the preset RGB image data is converted according to a pre-trained full-preset neural network model to obtain preset RAW image data corresponding to each preset RGB image data.

Only one of RGB three colors is information at a pixel point in the RAW image, and one pixel point in the RGB image data contains RGB three color information. The embodiment of the application establishes a full convolution network in advance, the preset full convolution neural network comprises a full convolution part and a deconvolution part, the input data of the full convolution part is subjected to convolution operation to obtain depth characteristics, and the output image with the same size as the original input image can be obtained through operation of the deconvolution part. The method comprises the steps of obtaining a plurality of RAW images and corresponding RGB image data obtained through ISP processing to form a plurality of image pairs, training the full convolution neural network by taking the RGB image data as input data and the RAW images as output data to determine network parameters, representing the conversion relation between the RGB image data and the RAW images by the network parameters obtained through training, inputting preset RGB image data into the network for calculation, and obtaining corresponding preset RAW image data with the same size as input data of model training.

The preset RGB image data can be directly displayed on a computer, and a user can correspondingly mark the preset RGB image data according to the type of the preset neural network model. For example, the preset neural network model is a target detection model, the label information may be a bounding box of a target subject in the sample image, the user manually marks a rectangular bounding box of the image to be recognized on the image, and corresponding position coordinates and size information are generated according to the bounding box as the label information. For another example, if the convolutional neural network is an image classification model, the label information is the class information of the target subject in the sample image.

And for the label information containing the position information, such as a bounding box, the second label information on the preset RGB image data can be directly mapped onto the preset RAW image data according to the position of the second label information, so as to obtain the corresponding first label information. In this way, the indirect marking of the preset RAW image data can be completed. And training a preset neural network model through the preset RAW image data and the first label information thereof to determine model parameters.

Or, in another embodiment, acquiring first tag information corresponding to the preset RAW image data as output data of the preset neural network model includes: acquiring preset RGB image data corresponding to the preset RAW image data and second label information of the preset RGB image data; and mapping second label information of the preset RGB image data to the preset RAW image data to obtain first label information serving as output data of model training.

In this embodiment, the data set used for model training includes the preset RAW image data, but the sample RAW image cannot be directly labeled, so that the preset RAW image data is converted into the corresponding preset RGB image data and then labeled to obtain the second label information, and then the second label information is mapped to the preset RAW image data to obtain the first label information as the output data of the model training. In this way, the indirect marking of the preset RAW image data can be completed. And training a preset neural network model through preset RAW image data and first label information thereof to determine model parameters.

In particular implementation, the present application is not limited by the execution sequence of the described steps, and some steps may be performed in other sequences or simultaneously without conflict.

As can be seen from the above, in the image processing method provided in the embodiment of the present application, when performing image processing, RAW image data is acquired by an image sensor; the RAW image data are processed through the AI processor, and the preset neural network model is obtained through training of the preset RAW image data and the corresponding first label information, so that in an image processing stage, an original RAW image output by the image sensor can be directly detected without being processed by an ISP (internet service provider), and the image processing efficiency is improved.

In some embodiments, processing the RAW image data by an AI processor includes: converting the RAW image data from single-channel RAW image data into multi-channel RAW image data; and processing the multi-channel RAW image data by an AI processor.

The original RAW image to be detected is a single-channel image, and in order to improve the operation speed, before the RAW image to be detected is input into a preset neural network model for operation, pixel separation processing can be carried out on the RAW image to obtain multi-channel RAW image data, wherein a feature map on each channel only corresponds to one color.

The specific pixel separation mode is determined according to the arrangement mode of the pixels in the RAW image to be detected. Referring to fig. 6, fig. 6 is a schematic diagram illustrating pixel separation in an image processing method according to an embodiment of the present disclosure. In this embodiment, pixels in a RAW image to be detected are arranged in the order of RGGB, a channel map of four channels is obtained after pixel separation, and the four channels are arranged in the order of R, G, and B. In other embodiments, summarizing, if the pixel arrangement order in the RAW image to be detected may be GRBG and GBRG, the pixel arrangement mode is adjusted to RGGB by deleting the first row of pixel points or the first column of pixel points, and then pixel separation processing is performed, so that the separated channels are arranged according to the order of R, G, and B. Assuming that the size of the RAW image to be detected is W × H, the sizes of the feature maps on the four channels obtained after the pixel separation processing are W/2 × H/2.

It can be understood that if the RAW image to be detected needs to be converted in the model training stage in the image stage, the same pixel separation processing needs to be performed on the input data in the model training stage, and the channel sequence is kept consistent.

The embodiment of the present application further provides a model training method, including: acquiring preset RAW image data as input data of a neural network model; acquiring first label information corresponding to the preset RAW image data as output data of the neural network model; and training the preset neural network model according to the input data and the output data to determine model parameters.

In some embodiments, acquiring preset RAW image data as input data of the preset neural network model includes: acquiring first preset RGB image data; and converting the first preset RGB image data into corresponding preset RAW image data according to a pre-trained full convolution neural network, and taking the preset RAW image data as input data of the preset neural network model. Acquiring first label information corresponding to the preset RAW image data as output data of the preset neural network model, wherein the first label information comprises: acquiring second label information corresponding to the first preset RGB image data; and mapping second label information of the first preset RGB image data to the preset RAW image data to obtain first label information serving as output data of the preset neural network model.

In some embodiments, obtaining first label information corresponding to the preset RAW image data as output data of the preset neural network model includes: acquiring preset RGB image data corresponding to the preset RAW image data and second label information of the preset RGB image data; and mapping the second label information of the preset RGB image data to the preset RAW image data to obtain first label information serving as output data of model training.

The model training method provided by each embodiment can be applied to a server, and after the server performs model training, the server performs model parameter updating and sends the updated model parameters to the electronic device, so that the electronic device updates the stored preset neural network model.

In the model training method provided in this embodiment, the RAW image is used as input data for model training, the RGB image data corresponding to the RAW image is labeled, and then mapping of the label information is performed, so as to obtain preset RAW image data with label information, and perform model training.

An image processing apparatus is also provided in an embodiment. Referring to fig. 7, fig. 7 is a schematic structural diagram of an image processing apparatus 300 according to an embodiment of the present disclosure. The image processing apparatus 300 is applied to an electronic device, and the image processing apparatus 300 includes an image capturing module 301 and an image processing module 302, as follows:

an image acquisition module 301, configured to acquire RAW image data through an image sensor;

the image processing module 302 is configured to process the RAW image data through an AI processor, where the AI processor includes a preset neural network model, and the preset neural network model is obtained by training according to preset RAW image data and first label information corresponding to the preset RAW image data.

In some embodiments, the image processing module 302 is further configured to convert the RAW image data from single-channel RAW image data to multi-channel RAW image data; and processing the multi-channel RAW image data by an AI processor.

In some embodiments, the preset neural network model is an image classification model, an object detection model, an image segmentation model or an instance segmentation model.

In some embodiments, the image processing apparatus 300 further comprises:

the model training module is used for acquiring the preset RAW image data as input data of the preset neural network model;

acquiring the first label information corresponding to the preset RAW image data as output data of the preset neural network model; and

training the neural network model based on the input data and the output data to determine model parameters.

In some embodiments, the model training module is further configured to obtain first preset RGB image data;

converting the first preset RGB image data into corresponding preset RAW image data according to a pre-trained full-convolution neural network, and taking the preset RAW image data as input data of a preset neural network model;

acquiring second label information corresponding to the first preset RGB image data; and mapping second label information of the first preset RGB image data to the preset RAW image data to obtain first label information serving as output data of the preset neural network model.

In some embodiments, the model training module is further configured to obtain preset RGB image data corresponding to the preset RAW image data and second label information of the preset RGB image data;

and mapping the second label information of the preset RGB image data to the preset RAW image data to obtain first label information serving as output data of model training.

It should be noted that the image processing apparatus provided in the embodiment of the present application and the image processing method in the foregoing embodiment belong to the same concept, and any method provided in the embodiment of the image processing method can be implemented by the image processing apparatus, and the specific implementation process of the method is described in detail in the embodiment of the image processing method, and is not described herein again.

As can be seen from the above, the image processing apparatus provided in the embodiment of the present application acquires RAW image data by using the image sensor when performing image processing; the RAW image data are processed through the AI processor, and the preset neural network model is obtained through training of the preset RAW image data and the corresponding first label information, so that in an image processing stage, an original RAW image output by the image sensor can be directly detected without being processed by an ISP (internet service provider), and the image processing efficiency is improved.

The embodiment of the present application further provides an electronic device, where the electronic device may be a terminal, and the terminal may be a terminal such as a smart phone, a tablet Computer, a notebook Computer, a touch screen, a game console, a Personal Computer (PC), a Personal Digital Assistant (PDA), and the like. As shown in fig. 8, fig. 8 is a third schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 400 includes a processor 401 having one or more processing cores, a memory 402 having one or more computer-readable storage media, and a computer program stored on the memory 402 and executable on the processor. The processor 401 is electrically connected to the memory 402. Those skilled in the art will appreciate that the electronic device configurations shown in the figures do not constitute limitations of the electronic device, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.

The processor 401 is a control center of the electronic device 400, connects various parts of the entire electronic device 400 using various interfaces and lines, performs various functions of the electronic device 400 and processes data by running or loading software programs and/or modules stored in the memory 402 and calling data stored in the memory 402, thereby integrally monitoring the electronic device 400.

In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402, so as to implement various functions:

acquiring RAW image data through an image sensor;

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

Optionally, as shown in fig. 5, the electronic device 400 further includes: touch-sensitive display screen 403, radio frequency circuit 404, audio circuit 405, input unit 406 and power 407. The processor 401 is electrically connected to the touch display 403, the rf circuit 404, the audio circuit 405, the input unit 406, and the power source 407 respectively. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 5 does not constitute a limitation of the electronic device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.

The touch display screen 403 may be used for displaying a graphical user interface and receiving operation instructions generated by a user acting on the graphical user interface. The touch display screen 403 may include a display panel and a touch panel. The display panel may be used, among other things, to display information entered by or provided to a user and various graphical user interfaces of the electronic device, which may be made up of graphics, text, icons, video, and any combination thereof. Alternatively, the Display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. The touch panel may be used to collect touch operations of a user (for example, operations of the user on or near the touch panel by using a finger, a stylus pen, or any other suitable object or accessory) and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Alternatively, the touch panel may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 401, and can receive and execute commands sent by the processor 401. The touch panel may overlay the display panel and, when the touch panel detects a touch operation thereon or nearby, transmit the touch operation to the processor 401 to determine the type of the touch event, and then the processor 401 provides a corresponding visual output on the display panel according to the type of the touch event. In the embodiment of the present application, the touch panel and the display panel may be integrated into the touch display screen 403 to realize input and output functions. However, in some embodiments, the touch panel and the touch panel can be implemented as two separate components to perform the input and output functions. That is, the touch display screen 403 may also be used as a part of the input unit 406 to implement an input function.

The rf circuit 404 may be used for transceiving rf signals to establish wireless communication with a network device or other electronic devices via wireless communication, and for transceiving signals with the network device or other electronic devices.

The audio circuit 405 may be used to provide an audio interface between the user and the electronic device through a speaker, microphone. The audio circuit 405 may transmit the electrical signal converted from the received audio data to a speaker, and convert the electrical signal into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 405 and converted into audio data, which is then processed by the audio data output processor 401 and then transmitted to, for example, another electronic device via the rf circuit 404, or the audio data is output to the memory 402 for further processing. Audio circuitry 405 may also include an earbud jack to provide communication of peripheral headphones with the electronic device.

The input unit 406 may be used to receive input numbers, character information, or user characteristic information (e.g., fingerprint, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

The power supply 407 is used to power the various components of the electronic device 400. Optionally, the power supply 407 may be logically connected to the processor 401 through a power management system, so as to implement functions of managing charging, discharging, power consumption management, and the like through the power management system. The power supply 407 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

Although not shown in fig. 5, the electronic device 400 may further include an image sensor, a wireless fidelity module, a bluetooth module, etc., which are not described in detail herein.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the related descriptions of other embodiments.

As can be seen from the above, the electronic device provided in this embodiment acquires RAW image data through the image sensor when performing image processing; the RAW image data are processed through the AI processor, and the preset neural network model is obtained through training of the preset RAW image data and the corresponding first label information, so that in an image processing stage, an original RAW image output by the image sensor can be directly detected without being processed through an ISP (internet service provider), and the image processing efficiency is improved.

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer programs can be loaded by a processor to execute the steps in any one of the image processing methods provided by the embodiments of the present application. For example, the computer program may perform the steps of:

acquiring RAW image data through an image sensor;

Alternatively, in another embodiment, the computer program can be loaded by a processor to perform the steps of any one of the neural network model training methods provided in the embodiments of the present application. For example, the computer program may perform the steps of:

acquiring preset RAW image data as input data of a neural network model; acquiring first label information corresponding to the preset RAW image data as output data of the neural network model; and training the preset neural network model according to the input data and the output data to determine model parameters.

Wherein the storage medium may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like.

Since the computer program stored in the storage medium can execute the steps in any image processing method provided in the embodiment of the present application, beneficial effects that can be achieved by any image processing method provided in the embodiment of the present application can be achieved, for details, see the foregoing embodiment, and are not described herein again.

The image processing method, the image processing apparatus, the storage medium, the device, and the model training method provided in the embodiments of the present application are described in detail above, and specific examples are applied in the present application to explain the principles and embodiments of the present application, and the description of the above embodiments is only used to help understanding the method and the core concept of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and the application scope may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An image processing method, comprising:

acquiring RAW image data through an image sensor;

2. The method of claim 1, wherein processing the RAW image data by an AI processor comprises:

converting the RAW image data from single-channel RAW image data into multi-channel RAW image data; and

the multi-channel RAW image data is processed by an AI processor.

3. The method of claim 1, wherein the pre-set neural network model is an image classification model, an object detection model, an image segmentation model, or an instance segmentation model.

4. The method of any of claims 1 to 3, wherein prior to acquiring RAW image data by the image sensor, further comprising:

acquiring the preset RAW image data as input data of the preset neural network model;

5. A neural network model training method is characterized by comprising the following steps:

acquiring preset RAW image data as input data of a neural network model;

6. The method of claim 5, wherein obtaining pre-set RAW image data as input data to the pre-set neural network model comprises:

acquiring first preset RGB image data;

converting the first preset RGB image data into corresponding preset RAW image data according to a preset full convolution neural network, and taking the preset RAW image data as input data of a preset neural network model;

acquiring first tag information corresponding to the preset RAW image data, wherein the first tag information is used as output data of the preset neural network model and comprises the following steps:

acquiring second label information corresponding to the first preset RGB image data;

and mapping second label information of the first preset RGB image data to the preset RAW image data to obtain first label information serving as output data of the preset neural network model.

7. The method of claim 5, wherein obtaining first label information corresponding to the preset RAW image data as output data of the preset neural network model comprises:

acquiring preset RGB image data corresponding to the preset RAW image data and second label information of the preset RGB image data;

8. An image processing apparatus characterized by comprising:

9. A computer-readable storage medium on which a computer program is stored, which, when run on a computer, causes the computer to execute an image processing method according to any one of claims 1 to 4;

alternatively, the computer program, when run on a computer, causes the computer to perform the neural network model training method of any one of claims 5 to 7.

10. An electronic device comprising an image sensor, and an AI processor connected to the image sensor;

the image sensor is used for acquiring RAW image data;

11. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the image processing method according to any one of claims 1 to 4 by calling the computer program;

the processor is configured to execute the neural network model training method according to any one of claims 5 to 7 by calling the computer program.