CN110636216B

CN110636216B - Image processing method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN110636216B
Application number: CN201910802385.6A
Authority: CN
Inventors: 贾玉虎
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-08-28
Filing date: 2019-08-28
Publication date: 2023-04-18
Anticipated expiration: 2039-08-28
Also published as: CN110636216A

Abstract

The application relates to an image processing method and device, an electronic device and a computer readable storage medium, comprising: the method comprises the steps of obtaining RAW image data of an original image shot by a camera module, carrying out electronic anti-shake compensation on the RAW image data of the original image to obtain RAW image data after the electronic anti-shake compensation, and obtaining a target image according to the RAW image data after the electronic anti-shake compensation. Since the RAW image data is data directly output via the image sensor, complete information of a photographed picture is retained without any cropping processing. Compared with RGB image data or YUV image data, the RAW image data has smaller data volume, so that electronic anti-shake compensation is performed on the RAW image data of the original image, the data volume is reduced, a better anti-shake processing effect is obtained, and the definition of the image after anti-shake processing is improved.

Description

Image processing method and device, electronic equipment and computer readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to an image processing method and apparatus, an electronic device, and a computer-readable storage medium.

Background

With the continuous development of the camera technology, the requirement of people on photographing of the camera of the electronic equipment is increasingly improved. The traditional electronic equipment develops from a single camera to a later double camera, and the photographing quality is obviously improved. However, the requirement for taking pictures of electronic devices is increasing day by day, and how to further improve the quality of taking pictures of electronic devices and meet the higher requirements of users for taking pictures is a problem to be solved urgently.

Disclosure of Invention

The embodiment of the application provides an image processing method and device, electronic equipment and a computer readable storage medium, which can improve the photographing quality of the electronic equipment and meet higher photographing requirements.

An image processing method applied to an electronic device includes:

acquiring RAW image data of an original image shot by a camera module;

carrying out electronic anti-shake compensation on RAW image data of the original image to obtain RAW image data after electronic anti-shake compensation;

and obtaining a target image according to the RAW image data after the electronic anti-shake compensation.

An image processing apparatus comprising:

the RAW image data acquisition module is used for acquiring RAW image data of an original image shot by the camera module;

the electronic anti-shake module is used for carrying out electronic anti-shake compensation on the RAW image data of the original image to obtain the RAW image data after EIS electronic anti-shake compensation;

and the target image determining module is used for obtaining a target image according to the RAW image data after the electronic anti-shake compensation.

An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to carry out the steps of the above method.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above method.

The image processing method, the image processing device, the electronic equipment and the computer-readable storage medium are used for acquiring RAW image data of an original image shot by the camera module, performing electronic anti-shake compensation on the RAW image data of the original image to obtain RAW image data after the electronic anti-shake compensation, and obtaining a target image according to the RAW image data after the electronic anti-shake compensation. Since the RAW image data is data directly output via the image sensor, it is not subjected to any cropping processing, and the information of the entirety of the photographed picture is retained. And compared with RGB image data or YUV image data, the RAW image data has smaller data volume, so that electronic anti-shake compensation is performed on the RAW image data of the original image, a better anti-shake processing effect is obtained while the data volume is reduced, and the definition of the image after anti-shake processing is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.

FIG. 1 is a diagram of an exemplary embodiment of an image processing method;

FIG. 2 is a flow diagram of a method of image processing in one embodiment;

FIG. 3 is a flow chart of an image processing method in another embodiment;

fig. 4 is a schematic structural diagram of an OIS anti-shake system of a camera in one embodiment;

FIG. 5 is a flowchart of an image processing method in another embodiment;

FIG. 6 is a diagram illustrating an image processing effect according to an embodiment;

FIG. 7 is a flow diagram of a method of image processing in one particular embodiment;

FIG. 8A is a block diagram showing a configuration of an image processing apparatus according to an embodiment;

FIG. 8B is a block diagram showing a configuration of an image processing apparatus according to another embodiment;

FIG. 9 is a schematic diagram showing an internal configuration of an electronic apparatus according to an embodiment;

FIG. 10 is a schematic diagram of image processing circuitry in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first camera may be referred to as a second camera, and similarly, a second camera may be referred to as a first camera, without departing from the scope of the present application. The first camera and the second camera are both cameras, but they are not the same camera.

Fig. 1 is a schematic diagram of an application environment of an image processing method in an embodiment. As shown in fig. 1, the application environment includes an electronic device 100. The electronic apparatus 100 includes at least two camera modules 110. The electronic device 100 may acquire RAW image data of an original image captured by the camera module; carrying out electronic anti-shake compensation on RAW image data of an original image to obtain RAW image data after electronic anti-shake compensation; and obtaining a target image according to the RAW image data after the electronic anti-shake compensation. It is understood that the electronic device 100 may not be limited to various mobile phones, cameras, computers, portable devices, and the like.

Fig. 2 is a flowchart of an image processing method according to an embodiment, and as shown in fig. 2, the image processing method includes steps 220 to 260.

Step 220, RAW image data of the original image shot by the camera module is obtained.

RAW image data is RAW data in which a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) image sensor converts a captured light source signal into a digital signal. The RAW format is an output format of the image sensor, and is RAW data indicating the intensity of each light received by the image sensor. The RAW image data is output in a certain order, and generally includes four types:

00：GR/BG

01:RG/GB

10:BG/GR

11:GB/RG

the formats of the RAW data are RAW8, RAW10, and RAW12, but other formats are also possible. In which RAW8 is a component of G/R/B/G expressed by 8bits, a crystal is used to express a color in the image sensor in order to reduce power consumption, and then the value of the adjacent pixel is calculated using the difference.

RAW10 is 10 bits to indicate the above-mentioned one G/R/B/G, but 16 bits are used in the data, and the upper 6 bits are not used. The RAW12 indicates the above-mentioned one G/R/B/G by using 12 bits, but the data is 16 bits, and the upper 4 bits are useless. RAW image data herein refers to Bayer pattern image data with RAW as a suffix. The camera module comprises a gyroscope, a main control chip, a lens and the like. Shooting a shooting scene through the electronic equipment, so that the electronic equipment acquires RAW image data of an original image shot by the camera module and presented on the image sensor.

For example, when the YUV image data is in YUV444 format, the bandwidth of the YUV image data is 1.5 times that of the RAW image data, and therefore, the data amount of the RAW image data is smaller relative to the YUV444 format image data. And the data amount of RAW image data relative to RGB image data is also small. Conventionally, image data in RGB format or YUV444 format is generally used for electronic anti-shake, so that image processing is directly performed by using RAW image data, which can reduce the amount of data to be processed and improve efficiency.

And 240, performing electronic anti-shake compensation on the RAW image data of the original image to obtain the RAW image data after the electronic anti-shake compensation.

The EIS (electronic Image Stabilization) uses the detected shaking degree of the body to dynamically adjust ISO, shutter or software for blur correction. EIS anti-jitter is an algorithm operation, which avoids blurring by image clipping compensation, and the anti-jitter effect depends on the design and efficiency of the algorithm. The EIS anti-jitter has the advantages of no additional hardware, low cost and suitability for miniaturization design. And carrying out electronic anti-shake compensation on the RAW image data of the original image to obtain the RAW image data after the electronic anti-shake compensation.

And step 260, obtaining a target image according to the RAW image data after the electronic anti-shake compensation.

And after the RAW image data after the electronic anti-shake compensation is obtained, converting the RAW image data into RGB image data or YUV image data, and displaying a target image on a display screen.

In the embodiment of the application, RAW image data of an original image shot by a camera module is obtained, electronic anti-shake compensation is performed on the RAW image data of the original image to obtain RAW image data after the electronic anti-shake compensation, and a target image is obtained according to the RAW image data after the electronic anti-shake compensation. Since the RAW image data is data directly output via the image sensor, complete information of a photographed picture is retained without any cropping processing. And compared with RGB image data or YUV image data, the RAW image data has smaller data volume, so that electronic anti-shake compensation is performed on the RAW image data of the original image, a better anti-shake processing effect is obtained while the data volume is reduced, and the definition of the image after anti-shake processing is improved.

In one embodiment, the RAW image data includes pixel values.

When the RAW image data is in the RAW8 format, 8bits are used to represent one component in G/R/B/G, and each component represents one pixel value piexl. Therefore, a preview image can be obtained by each pixel value in the RAW image data of the original image.

In one embodiment, before acquiring RAW image data of an original image captured by a camera module, the method includes:

acquiring angular velocity data of a lens in the camera module through a gyroscope;

adjusting shooting parameters of the camera module according to the angular speed data;

step 220, acquiring RAW image data of an original image shot by the camera module, including:

and acquiring RAW image data of the original image obtained by shooting by the camera module according to the adjusted shooting parameters.

Specifically, before the electronic device shoots to obtain an original image, the gyroscope is used for collecting angular velocity data of a lens in any camera module, and shooting parameters of the camera module are adjusted according to the angular velocity data. For example, the influence of shake on shooting is reduced by adjusting the shutter speed and ISO (International Standardization Organization) sensitivity of the image pickup module based on the angular velocity data. Therefore, the camera module shoots the RAW image data of the obtained original image according to the adjusted shooting parameters.

In the embodiment of the application, before the electronic device shoots to obtain an original image, the gyroscope built in the camera module is used for detecting the angular velocity data of the lens, and the angular velocity information is caused by shaking during shooting. Therefore, the shooting parameters of the camera module are adjusted according to the angular velocity information, and the camera module is enabled to shoot RAW image data of the obtained original image according to the adjusted shooting parameters. The interference is performed in advance from the source of acquiring the RAW image data by adjusting the shooting parameters, so that the accuracy of the RAW image data obtained by shooting is higher.

In one embodiment, as shown in fig. 3, step 220 of acquiring RAW image data of an original image captured by a camera module includes:

in step 222, RAW image data of an original image shot by the camera module in the optical anti-shake mode is obtained.

Specifically, when RAW image data of an original image captured by the image capture module is acquired, the RAW image data of the original image may be captured by using optical anti-shake. The OIS anti-shake (Optical image stabilization) refers to that in a camera or other similar imaging apparatuses, the shake phenomenon of the apparatuses occurring in the process of capturing Optical signals is avoided or reduced by setting Optical components, for example, a lens, so as to improve the imaging quality. Electronic equipment in this application includes two at least modules of making a video recording, and every module of making a video recording all has OIS anti-shake function. When the electronic device includes two camera modules, RAW image data of an original image shot by the two camera modules in the optical anti-shake mode is acquired respectively, and the two sets of RAW image data can be synthesized to obtain RAW image data of the synthesized original image.

In the embodiment of the application, when the electronic device includes at least two camera modules and each camera module has an OIS anti-shake function, RAW image data of an original image is obtained by shooting in the OIS anti-shake mode. The RAW image data of the original image is acquired by shooting in the OIS anti-shake mode, so that OIS anti-shake is performed in advance at the source of the acquired data, and the accuracy of the RAW image data acquired by shooting is improved. When the electronic device includes a plurality of camera modules, RAW image data of the original image shot by the plurality of camera modules in the optical anti-shake mode is obtained respectively, a plurality of groups of RAW image data can be synthesized to obtain RAW image data of the synthesized original image, and accuracy of the RAW image data shot and obtained is improved.

In one embodiment, the gyroscope in the optical anti-shake mode collects angular velocity information of the lens, and transmits the angular velocity data to the driver so that the driver drives the motor to control the lens to move to the target position according to the angular velocity data;

step 222, acquiring RAW image data of an original image shot by the camera module in the optical anti-shake mode, including:

and acquiring RAW image data of an original image shot by moving a lens in the camera module to a target position under the control of driving.

Specifically, angular velocity data of a lens in any camera module is collected through a gyroscope, and the angular velocity data is transmitted to a main control chip to calculate jitter compensation information. And sending the shake compensation information to the OIS driving chip so that the OIS driving chip drives the corresponding lens in each camera module to move the same distance to the target position in the same direction according to the shake compensation information. And acquiring RAW image data of an original image shot by moving the lens in each camera module to a target position under the control of driving.

Specifically, as shown in fig. 4, a schematic structural diagram of an OIS anti-shake system of a camera in one embodiment is shown. As shown in fig. 4, in one embodiment, the main control chip 412 and the gyroscope 414 included in the camera anti-shake system are disposed on the main board 410 of the camera anti-shake system, and the anti-shake driving chip 422, the motor 424 and the lens 426 are disposed in the camera module 420 of the camera anti-shake system. The gyroscope 414 and the main control chip 412 may be connected through an SPI (Serial Peripheral Interface); the main control chip 412 and the anti-shake driver chip 422 may be connected via an Inter-Integrated Circuit (IIC) bus. The gyroscope 414 may collect angular velocity information of the lens 426 in any camera module, and send the angular velocity information to the main control chip 412, the main control chip 412 may calculate shake compensation information of the lens 426 according to the angular velocity information, and send the shake compensation information to the anti-shake driving chip 422, and the anti-shake driving chip 422 may control the motor 424 to be powered on according to the shake compensation information, so that the motor 424 drives the lens 426 to move.

Further, an image sensor 428 is further disposed in the camera module 420, and the image sensor 428 may be connected to the main Control chip 412 through a Connection Control Interface (CCI). The main control chip 412 may control the image sensor 428 to be powered on through the CCI interface upon receiving the image capture command, so that the image sensor 428 captures an image based on the moved lens 426.

In the embodiment of the application, the shake compensation information of the lens is calculated by the main control chip according to the angular velocity information acquired by the gyroscope, the anti-shake driving chip controls the motor of each camera module to be powered on to drive the corresponding lens to move to the target position in the same direction by the same distance according to the shake compensation information, and RAW image data of an original image shot by the lens in each camera module moving to the target position is acquired. The anti-shake driving chip is not needed to be used for calculating shake compensation information in the process, the size of the anti-shake driving chip can be reduced, namely, the size of the camera module is reduced, and the reliability of the camera module can be improved. Meanwhile, the RAW image data of the original image is obtained by shooting in the OIS anti-shake mode, so that OIS anti-shake is carried out in advance at the source of the obtained data, and the accuracy of the RAW image data obtained by shooting is improved.

In one embodiment, as shown in fig. 5, after acquiring RAW image data of an original image captured by a camera module in step 502, the method includes:

and step 504, performing subject detection on the original image to obtain a subject detection result.

After RAW image data of an original image shot by a camera module is obtained, subject detection is carried out on the RAW image data of the original image to obtain a subject detection result. Specifically, the subject detection result may be obtained by performing subject detection using a subject detection model. The subject refers to various subjects, such as human, flower, cat, dog, cow, blue sky, white cloud, background, etc.

Step 506, RAW image data corresponding to the subject detection result is obtained from RAW image data of the original image.

After the subject detection result is obtained, RAW image data corresponding to the region where the subject detection result is located is acquired from RAW image data of the original image.

And step 508, performing electronic anti-shake compensation on the RAW image data corresponding to the main body detection result to obtain RAW image data corresponding to the main body detection result after the EIS electronic anti-shake compensation.

In order to highlight the subject in the captured original image and to reduce the amount of image processing calculation, only the RAW image data corresponding to the region where the subject detection result is located may be subjected to electronic anti-shake compensation, and the other regions may not be subjected to electronic anti-shake processing, so that the effect of blurring the background may be achieved. And carrying out electronic anti-shake compensation on the RAW image data corresponding to the main body detection result to obtain the RAW image data corresponding to the main body detection result after the EIS electronic anti-shake compensation.

In step 510, RAW image data of a background region excluding RAW image data corresponding to a subject detection result is acquired from RAW image data of an original image.

The original image generally includes a subject and a background, and when RAW image data corresponding to a subject detection result is removed from RAW image data of the original image, RAW image data of a background area in the original image can be obtained.

And 512, splicing and synthesizing the RAW image data of the background area and RAW image data corresponding to the main body detection result after the electronic anti-shake compensation to obtain a spliced and synthesized target image.

And finally, splicing and synthesizing the RAW image data of the background area and RAW image data corresponding to the main body detection result after the electronic anti-shake compensation, and obtaining a target image after splicing and synthesis, namely obtaining a complete image of a shooting scene.

In the embodiment of the present application, after the subject detection is performed on the RAW image data of the original image to obtain the subject detection result, the RAW image data corresponding to the subject region and the RAW image data corresponding to the background region may be divided from the RAW image data of the original image. Therefore, in order to achieve the effect of background blurring, only RAW image data corresponding to the main area may be subjected to electronic anti-shake, and then RAW image data of the background area and RAW image data corresponding to the main area after electronic anti-shake compensation are subjected to stitching synthesis to obtain a stitched and synthesized target image. Therefore, the calculation amount is greatly reduced while the background blurring effect is realized, and the image processing efficiency is improved.

In one embodiment, the subject detecting the original image to obtain a subject detection result includes:

and carrying out subject detection on the original image by adopting a subject detection model to obtain a subject detection result, wherein the subject detection model is obtained by carrying out deep learning on RAW image data of the sample image.

The subject detection model may be obtained by performing deep learning on RAW image data of a large number of sample images. In deep learning, a convolutional neural network can be used for continuously training samples to detect a main body in the samples, so that a main body detection model is trained.

For example, as shown in fig. 6, the schematic diagram of the image processing effect of performing the subject detection on the original image through the subject detection model to obtain the subject detection result may be that, as shown in fig. 6, one butterfly exists in the RAW map 602, the RAW map is input to the subject detection model to obtain a subject region confidence map 604, then the subject region confidence map 604 is filtered and binarized to obtain a binarized mask map 606, and then the binarized mask map 606 is subjected to morphological processing and guided filtering to realize edge enhancement, so as to obtain a subject mask map 608.

In the embodiment of the application, a subject detection model is used for performing subject detection on an original image to obtain a subject detection result, and the subject detection model is obtained by performing deep learning on RAW image data of a sample image. The method has the advantages that deep learning is carried out on RAW image data of a large number of sample graphs to obtain the main body detection model, the accuracy of the obtained main body detection model is high, main body detection is carried out on the original images through the main body detection model to obtain main body detection results, and therefore the accuracy of the detected main body detection results is finally improved.

In one embodiment, obtaining the target image according to the RAW image data after the electronic anti-shake compensation includes:

converting RAW image data after electronic anti-shake compensation into YUV image data;

and compressing the image of the YUV image data to obtain a target image.

Specifically, for image data in YUV format, the luminance signal of an image is called Y, and the chrominance signal is composed of two independent signals, often called U and V, depending on the color system and format. After RAW Image data subjected to electronic anti-shake compensation is obtained, the RAW Image data is subjected to noise reduction Processing, and format conversion is performed on the RAW Image data subjected to the noise reduction Processing by an Image Signal Processing (ISP) to convert the RAW Image data into YUV Image data. And finally, compressing the image of the YUV image data to obtain a target image for preview display.

In the embodiment of the application, the image format that can be displayed on the display screen of the electronic device is image data in a YUV format, so that RAW image data after electronic anti-shake compensation needs to be converted into YUV image data. And then, compressing the image of the YUV image data to an appropriate preview size according to the size of the display screen for preview display.

In a specific embodiment, there is also provided an image processing method, including:

step 702, acquiring angular velocity data of a lens in a camera module through a gyroscope;

step 704, adjusting shooting parameters of the camera module according to the angular speed data;

step 706, acquiring the adjusted shooting parameters of the camera module and RAW image data of the original image shot in the optical anti-shake mode;

step 708, performing subject detection on RAW image data of the original image to obtain a subject detection result;

step 710, acquiring RAW image data corresponding to a subject detection result from RAW image data of an original image;

step 712, performing electronic anti-shake compensation on RAW image data corresponding to the main body detection result to obtain RAW image data corresponding to the main body detection result after the EIS electronic anti-shake compensation;

step 714, acquiring RAW image data of a background area except for RAW image data corresponding to a subject detection result from RAW image data of an original image;

and 716, splicing and synthesizing the RAW image data of the background area and the RAW image data corresponding to the main body detection result after the electronic anti-shake compensation to obtain a spliced and synthesized target image.

Step 718, converting RAW image data of the spliced and synthesized target image into YUV image data, and compressing the image of the YUV image data to obtain a preview image for displaying.

In the embodiment of the application, because the RAW image data is directly output by the image sensor, the complete information of the shot picture is reserved without any cutting processing. And compared with RGB image data or YUV image data, the RAW image data has smaller data volume, so that electronic anti-shake compensation is performed on the RAW image data of the original image, a better anti-shake processing effect is obtained while the data volume is reduced, and the definition of the image after anti-shake processing is improved.

The interference is performed in advance from the source of obtaining the RAW image data by adjusting the shooting parameters, and the optical anti-shake is added, so that the accuracy of the RAW image data obtained by shooting is higher. When electronic anti-shake is performed, electronic anti-shake can be performed only on the RAW image data corresponding to the main area, and then the RAW image data of the background area and the RAW image data corresponding to the main area after electronic anti-shake compensation are spliced and synthesized to obtain a spliced and synthesized target image. Therefore, the calculation amount is greatly reduced while the background blurring effect is realized, and the image processing efficiency is improved.

It should be understood that, although the steps in the above-described flowcharts are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a part of the steps in the above flowcharts may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or the stages is not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a part of the sub-steps or the stages of other steps.

In one embodiment, as shown in fig. 8A, there is provided an image processing apparatus 800 comprising:

a RAW image data obtaining module 820, configured to obtain RAW image data of an original image captured by the camera module;

the electronic anti-shake module 840 is configured to perform electronic anti-shake compensation on RAW image data of an original image to obtain RAW image data after the electronic anti-shake compensation;

and the target image determining module 860 is used for obtaining a target image according to the RAW image data after the electronic anti-shake compensation.

In one embodiment, as shown in fig. 8B, there is provided an image processing apparatus 800, further comprising: the shooting parameter adjusting module 810 of the camera module is used for acquiring angular velocity data of a lens in the camera module through a gyroscope; adjusting shooting parameters of the camera module according to the angular speed data; the RAW image data obtaining module 820 is further configured to obtain RAW image data of an original image obtained by the camera module performing shooting with the adjusted shooting parameters.

In one embodiment, the RAW image data acquiring module 820 is further configured to acquire RAW image data of an original image captured by the camera module in the optical anti-shake mode.

the RAW image data acquiring module 820 is further configured to acquire RAW image data of an original image captured by the lens in the camera module moving to the target position under the driving control.

In one embodiment, there is provided an image processing apparatus including:

the main body detection module is used for carrying out main body detection on the original image to obtain a main body detection result;

the system comprises a RAW image data acquisition module of a subject, a data acquisition module and a data acquisition module, wherein the RAW image data acquisition module is used for acquiring RAW image data corresponding to a subject detection result from RAW image data of an original image;

the electronic anti-shake module is used for carrying out electronic anti-shake compensation on RAW image data corresponding to the main body detection result to obtain RAW image data corresponding to the main body detection result after EIS electronic anti-shake compensation;

a RAW image data acquisition module of the background area, configured to acquire RAW image data of the background area excluding RAW image data corresponding to the subject detection result from RAW image data of the original image;

and the splicing and synthesizing module is used for splicing and synthesizing the RAW image data of the background area and the RAW image data corresponding to the main body detection result after the electronic anti-shake compensation to obtain a spliced and synthesized target image.

In an embodiment, the subject detection module is further configured to perform subject detection on the original image by using a subject detection model to obtain a subject detection result, where the subject detection model is obtained by performing deep learning on RAW image data of the sample image.

In one embodiment, the target image determining module 860 is further configured to convert the RAW image data after the electronic anti-shake compensation into YUV image data; and compressing the image of the YUV image data to obtain a target image.

The division of the modules in the image processing apparatus is only for illustration, and in other embodiments, the image processing apparatus may be divided into different modules as needed to complete all or part of the functions of the image processing apparatus.

Fig. 9 is a schematic diagram of an internal structure of an electronic device in one embodiment. As shown in fig. 9, the electronic device includes a processor and a memory connected by a system bus. Wherein, the processor is used for providing calculation and control capability and supporting the operation of the whole electronic equipment. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program is executable by a processor to implement an image processing method provided in the following embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium. The electronic device may be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device, etc.

The implementation of each module in the image processing apparatus provided in the embodiments of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program modules constituted by the computer program may be stored on the memory of the terminal or the server. Which when executed by a processor, performs the steps of the method described in the embodiments of the present application.

The embodiment of the application also provides the electronic equipment. The electronic device includes therein an Image Processing circuit, which may be implemented using hardware and/or software components, and may include various Processing units defining an ISP (Image Signal Processing) pipeline. FIG. 10 is a schematic diagram of image processing circuitry in one embodiment. As shown in fig. 10, for convenience of explanation, only aspects of the image processing technique related to the embodiments of the present application are shown.

As shown in fig. 10, the image processing circuit includes a first ISP processor 1030, a second ISP processor 1040, and a control logic 1050. The first camera 1010 includes one or more first lenses 1012 and a first image sensor 1014. First image sensor 1014 may include a color filter array (e.g., a Bayer filter), and first image sensor 1014 may acquire light intensity and wavelength information captured with each imaging pixel of first image sensor 1014 and provide a set of image data that may be processed by first ISP processor 1030. The second camera 1020 includes one or more second lenses 1022 and a second image sensor 1024. The second image sensor 1024 may include a color filter array (e.g., a Bayer filter), and the second image sensor 1024 may acquire light intensity and wavelength information captured with each imaging pixel of the second image sensor 1024 and provide a set of image data that may be processed by the second ISP processor 1040.

The first image acquired by the first camera 1010 is transmitted to the first ISP processor 1030 to be processed, after the first ISP processor 1030 processes the first image, the first ISP processor 1030 may send statistical data (such as brightness of the image, contrast value of the image, color of the image, and the like) of the first image to the control logic 1050, and the control logic 1050 may determine a control parameter of the first camera 1010 according to the statistical data, so that the first camera 1010 may perform operations such as auto focus and auto exposure according to the control parameter. The first image may be stored in the image memory 1060 after being processed by the first ISP processor 1030, and the first ISP processor 1030 may also read the image stored in the image memory 1060 for processing. In addition, the first image may be directly transmitted to the display 1070 to be displayed after being processed by the ISP processor 1030, and the display 1070 may also read and display the image in the image memory 1060.

Wherein the first ISP processor 1030 processes the image data pixel by pixel in a plurality of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the first ISP processor 1030 may perform one or more image processing operations on the image data, collecting statistics about the image data. Wherein the image processing operations may be performed with the same or different bit depth precision.

The image Memory 1060 may be a part of a Memory device, a storage device, or a separate dedicated Memory within an electronic device, and may include a DMA (Direct Memory Access) feature.

Upon receiving an interface from first image sensor 1014, first ISP processor 1030 may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to an image memory 1060 for additional processing before being displayed. The first ISP processor 1030 receives the processed data from the image memory 1060 and performs image data processing in RGB and YCbCr color space on the processed data. The image data processed by the first ISP processor 1030 may be output to a display 1070 for viewing by a user and/or further processed by a Graphics Processing Unit (GPU). Further, the output of the first ISP processor 1030 may also be sent to an image memory 1060, and the display 1070 may read image data from the image memory 1060. In one embodiment, image memory 1060 may be configured to implement one or more frame buffers.

The statistics determined by the first ISP processor 1030 may be sent to the control logic 1050. For example, the statistical data may include first image sensor 1014 statistics such as auto-exposure, auto-white balance, auto-focus, flicker detection, black level compensation, first lens 1012 shading correction, and the like. Control logic 1050 may include a processor and/or microcontroller that executes one or more routines (e.g., firmware) that may determine control parameters for first camera 1010 and control parameters for first ISP processor 1030 based on the received statistical data. For example, the control parameters of the first camera 1010 may include gain, integration time of exposure control, anti-shake parameters, flash control parameters, first lens 1012 control parameters (e.g., focal length for focusing or zooming), or a combination of these parameters, and the like. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (e.g., during RGB processing), as well as first lens 1012 shading correction parameters.

Similarly, the second image captured by the second camera 1020 is transmitted to the second ISP processor 1040 for processing, after the second ISP processor 1040 processes the first image, the statistical data of the second image (such as the brightness of the image, the contrast value of the image, the color of the image, etc.) may be sent to the control logic 1050, and the control logic 1050 may determine the control parameter of the second camera 1020 according to the statistical data, so that the second camera 1020 may perform operations such as auto-focus and auto-exposure according to the control parameter. The second image may be stored in the image memory 1060 after being processed by the second ISP processor 1040, and the second ISP processor 1040 may also read the image stored in the image memory 1060 for processing. The second image may be directly transmitted to the display 1070 to be displayed after being processed by the ISP processor 1040, or the display 1070 may read and display the image in the image memory 1060. The second camera 1020 and the second ISP processor 1040 may also implement the processes described for the first camera 1010 and the first ISP processor 1030.

The image processing circuit provided by the embodiment of the application can realize the image processing method. The electronic equipment can be provided with a plurality of cameras, each camera comprises a lens and an image sensor arranged corresponding to the lens, and the image sensors in the cameras are arranged in a rectangular diagonal mode. The process of the electronic device implementing the image processing method is as in the above embodiments, and is not described herein again.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the image processing method.

A computer program product comprising instructions which, when run on a computer, cause the computer to perform an image processing method.

Any reference to memory, storage, databases, or other media used by embodiments of the application may include non-volatile and/or volatile memory. Suitable non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct bused dynamic RAM (DRDRAM), and Rambus Dynamic RAM (RDRAM).

The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. An image processing method applied to an electronic device, comprising:

acquiring RAW image data of an original image shot by a camera module in an optical anti-shake mode;

carrying out main body detection on the original image to obtain a main body detection result; the main body detection of the original image to obtain a main body detection result includes: inputting an original image into the main body detection model for processing to obtain a main body region confidence map; filtering and binarizing the confidence map of the main body region to obtain a binarized mask map; performing morphological processing and guide filtering on the binary mask image to obtain a main body mask image;

acquiring RAW image data corresponding to the subject detection result from RAW image data of the original image;

performing electronic anti-shake compensation on RAW image data corresponding to the main body detection result to obtain RAW image data corresponding to the main body detection result after the electronic anti-shake compensation;

acquiring RAW image data of a background area except for RAW image data corresponding to the subject detection result from RAW image data of the original image;

splicing and synthesizing the RAW image data of the background area and RAW image data corresponding to the main body detection result after the electronic anti-shake compensation to obtain RAW image data after the electronic anti-shake compensation;

2. The method of claim 1, wherein the RAW image data comprises pixel values.

3. The method according to claim 1, wherein before the acquiring RAW image data of the original image captured by the camera module in the optical anti-shake mode, the method comprises:

the RAW image data of the original image shot by the camera module in the optical anti-shake mode is acquired, and the method comprises the following steps:

and acquiring RAW image data of the original image obtained by shooting the camera module in the optical anti-shake mode according to the adjusted shooting parameters.

4. The method according to claim 1, wherein a gyroscope in the optical anti-shake mode collects angular velocity information of a lens, and transmits the angular velocity data to a driver so that the driver drives a motor to control the lens to move to a target position according to the angular velocity data;

and acquiring RAW image data of an original image shot by the lens in the camera module moving to the target position under the control of the drive.

5. The method according to claim 1, wherein the subject detecting the original image to obtain a subject detection result comprises:

and carrying out subject detection on the original image by adopting a subject detection model to obtain a subject detection result, wherein the subject detection model is obtained by carrying out deep learning on RAW image data of a sample image.

6. The method according to claim 1, wherein obtaining a target image from the RAW image data after the electronic anti-shake compensation comprises:

converting the RAW image data after the electronic anti-shake compensation into YUV image data;

and compressing the image of the YUV image data to obtain a target image.

7. An image processing apparatus characterized by comprising:

the RAW image data acquisition module is used for acquiring RAW image data of an original image shot by the camera module in an optical anti-shake mode;

the electronic anti-shake module is used for carrying out main body detection on the original image to obtain a main body detection result; the main body detection of the original image to obtain a main body detection result includes: inputting an original image into the main body detection model for processing to obtain a main body region confidence map; filtering and binarizing the confidence map of the main body region to obtain a binarized mask map; performing morphological processing and guided filtering on the binary mask image to obtain a main body mask image; acquiring RAW image data corresponding to the subject detection result from RAW image data of the original image; performing electronic anti-shake compensation on RAW image data corresponding to the main body detection result to obtain RAW image data corresponding to the main body detection result after the electronic anti-shake compensation; acquiring RAW image data of a background area except for RAW image data corresponding to the subject detection result from RAW image data of the original image; splicing and synthesizing the RAW image data of the background area and RAW image data corresponding to the main body detection result after the electronic anti-shake compensation to obtain RAW image data after the electronic anti-shake compensation;

8. The apparatus of claim 7, further comprising:

the shooting parameter adjusting module of the camera module is used for acquiring angular velocity data of a lens in the camera module through a gyroscope; adjusting shooting parameters of the camera module according to the angular speed data;

the RAW image data acquisition module is also used for acquiring RAW image data of an original image obtained by shooting the camera module by the adjusted shooting parameters.

9. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the computer program, when executed by the processor, causes the processor to perform the steps of the image processing method according to any of claims 1 to 6.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.