CN114007009A - Electronic device and image processing method - Google Patents

Abstract

The embodiment of the application provides electronic equipment and an image processing method, wherein an image signal pre-processor is additionally arranged, the image signal pre-processor is used for carrying out state information statistics on original image data acquired by an image sensor to correspondingly obtain first state information, and the image signal pre-processor is used for carrying out pre-processing on the original image data to obtain pre-processed image data; then, the image signal processor carries out statistics of state information on the pre-processed image data again to correspondingly obtain second state information; and finally, the application processor fuses the first state information counted by the image signal preprocessor and the second state information counted by the image signal processor to obtain target state information, and the image acquisition parameters of the image sensor are updated by using the target state information. Therefore, the purpose of improving the accuracy of the automatic configuration of the image acquisition parameters of the electronic equipment can be achieved.

Description

Electronic device and image processing method

technical field

The present application relates to the technical field of image processing, and in particular, to an electronic device and an image processing method.

Background technique

At present, the quality of the shooting function has become the key to measure the performance of an electronic device (such as a smartphone, tablet computer, etc.). In order to simplify the user's operation, the electronic device usually automatically configures the image acquisition parameters (such as exposure parameters, focus parameters, etc.) of the image data. However, in the related art, the accuracy of the automatic configuration of the image acquisition parameters by the electronic device is poor.

SUMMARY OF THE INVENTION

The present application provides an electronic device and an image processing method, which can improve the accuracy of the electronic device's automatic configuration of image acquisition parameters.

The present application discloses an electronic device, comprising:

an image sensor, used to collect image data according to the configured image acquisition parameters;

An image signal pre-processor, configured to count state information in the image data for updating the image acquisition parameters of the image sensor, obtain first state information, and perform pre-processing on the image data to obtain pre-processed image data ;

an image signal processor, configured to count state information in the pre-processed image data for updating the image acquisition parameters of the image sensor to obtain second state information;

An application processor, configured to fuse the first state information and the second state information to obtain target state information, and update image acquisition parameters of the image sensor according to the target state information.

The embodiment of the present application also discloses an image processing method, which is suitable for electronic equipment, where the electronic equipment includes an image sensor, an image signal preprocessor, an image signal processor, and an application processor, and the image processing method includes:

Collect image data by the image sensor according to the configured image acquisition parameters;

The image signal pre-processor counts the state information in the image data for updating the image acquisition parameters of the image sensor to obtain first state information, and pre-processes the image data to obtain pre-processed image data ;

Obtain second state information by counting the state information in the pre-processed image data for updating the image acquisition parameters of the image sensor by the image signal processor;

The application processor fuses the first state information and the second state information to obtain target state information, and updates the image acquisition parameters of the image sensor according to the target state information.

In the embodiment of the present application, an image signal pre-processor is additionally added, and the image signal pre-processor performs state information statistics on the raw image data collected by the image sensor, correspondingly obtains the first state information, and the image signal pre-processor analyzes the state information. The original image data is pre-processed to obtain pre-processed image data; then the image signal processor performs state information statistics on the pre-processed image data again, and correspondingly obtains second state information; finally, the application processor performs pre-processing on the image signal. The first state information of the statistics and the second state information of the image signal processor are fused to obtain the target state information, and the image acquisition parameters of the image sensor are updated by using the target state information. In this way, the purpose of improving the accuracy of automatically configuring the image acquisition parameters of the electronic device can be achieved.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that are used in the description of the embodiments.

FIG. 1 is a schematic diagram of a first structure of an electronic device provided by an embodiment of the present application.

FIG. 2 is a schematic structural diagram of the image signal pre-processor in FIG. 1 .

FIG. 3 is a schematic diagram of a data flow involved in an embodiment of the present application.

FIG. 4 is a schematic diagram of a second structure of an electronic device provided by an embodiment of the present application.

FIG. 5 is a schematic diagram of the connection between the image signal pre-processor and the application processor in FIG. 4 .

FIG. 6 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

Detailed ways

The technical solutions provided in the embodiments of the present application can be applied to various scenarios where data communication is required, which is not limited in the embodiments of the present application.

Please refer to FIG. 1 , which is a schematic diagram of a first structure of an electronic device 100 according to an embodiment of the present application. The electronic device 100 is, for example, a mobile electronic device such as a smart phone, a tablet computer, a palmtop computer, a notebook computer, etc., or a non-mobile electronic device such as a desktop computer and a server, which includes an image sensor 110, an image signal preprocessor 120, and an image signal processor. 130 and an application processor 140.

The image sensor 110, or photosensitive element, is a device that converts optical signals into electrical signals. Compared with the photosensitive elements of "point" light sources such as photodiodes and phototransistors, the image sensor 110 divides the light image it feels into many parts. A small unit is then converted into a usable electrical signal to obtain the original image data. It should be noted that the type of the image sensor 110 is not limited in the embodiments of the present application, and it may be a complementary metal-oxide-semiconductor (CMOS) image sensor, or a charge-coupled device (Charge Coupled Device) , CCD) image sensor, etc.

The image signal processor 130 can process the image data collected by the image sensor 110 to improve the quality of the image data. For example, the image signal processor 130 can perform optimization processing methods such as white balance correction, strong light suppression, backlight compensation, color enhancement, lens shadow correction, etc. on the image data. Of course, optimization processing methods not listed in this application may also be included.

Compared with the image signal processor 130, the image signal preprocessor 120 performs some differentiated processing before the image signal processor 130 processes the image data, which can be regarded as the preprocessing before the image signal processor 130 performs processing. For example, optimization processing methods such as dead pixel correction processing, time-domain noise reduction processing, 3D noise reduction processing, linearization processing, and black level correction processing, of course, may also include optimization processing methods not listed in this application.

The application processor 140 is a general-purpose processor, such as a processor designed based on the ARM architecture. .

In this embodiment of the present application, the image sensor 110 is connected to the image signal pre-processor 120 for collecting image data according to the configured image acquisition parameters, and transmitting the collected image data to the image signal pre-processor 120 . It should be noted that the image data collected by the image sensor 110 is image data in RAW format, and the collected image data in original RAW format is transmitted to the image signal pre-processor 120 .

The image acquisition parameters include, but are not limited to, exposure parameters, focus parameters, and white balance parameters. For example, before starting to collect image data, the application processor 140 will configure initial exposure parameters to the image sensor 110, so that the image sensor 110 collects image data according to the initial exposure parameters, and transmits the collected image data to the image sensor 110. Signal preprocessor 120 .

It should be noted that the connection mode between the image signal pre-processor 120 and the image sensor 110 is not specifically limited in the embodiments of the present application. For example, the MIPI (Mobile Industry Processor Interface, Mobile Industry Processor Interface) connection.

Wherein, when the image sensor 110 transmits image data to the image signal pre-processor 120, the image data is encapsulated into a plurality of image data packets, and transmitted to the image signal pre-processor in the form of image data packets. Exemplarily, the image data packet includes a header field, a trailer field and a data field, wherein the header field and the trailer field are used to fill in some necessary control information, such as synchronization information, address information, error control information, etc., and the data field is Used to populate the actual image content.

On the other hand, the image signal preprocessor 120 receives image data from the image sensor 110 . In addition, the image signal pre-processor 120 is also connected to the image signal processor 130, wherein the connection mode of the image signal processor 130 and the image signal pre-processor 120 is not specifically limited in the embodiment of the present application, for example, the image signal processing The connection between the processor 130 and the image signal pre-processor 120 may also be through MIPI.

Wherein, after receiving the image data transmitted from the image sensor 110, the image signal pre-processor 120 uses the image data to count the state information required by the application processor 140 to update the image acquisition parameters of the image sensor 110, including but not limited to brightness information, sharpness information, contrast information, etc., and the state information calculated by using the aforementioned image data is recorded as the first state information. In addition, the image signal pre-processor 120 also pre-processes the image data from the image sensor 110 according to the configured pre-processing strategy, improves the image quality of the image data, and obtains the pre-processed image data accordingly. It should be noted that the preprocessing performed by the image signal preprocessor 120 on the image data does not change the format of the image data, that is, the preprocessed image data obtained by the preprocessing is still in the RAW format.

After completing the pre-processing of the image data and obtaining the pre-processed image data, the image signal pre-processor 120 transmits the pre-processed image data to the image signal processor 130 .

After receiving the pre-processed image data from the image signal pre-processor 120, the image signal processor 130 uses the pre-processed image data to count the state information required by the application processor 140 to update the image acquisition parameters of the image sensor 110, including: However, it is not limited to brightness information, sharpness information, contrast information, etc., and the state information calculated by using the pre-processed image data is recorded as the second state information.

In addition, for the first state information, the image signal pre-processor 120 may directly transmit the first state information to the application processor 140 , or transmit the first state information to the application processor 140 via the image signal processor 130 .

After the image signal preprocessor 120 statistically obtains the first state information, and the image signal processor 130 obtains the second state information through statistics, the application processor 140 further fuses the first state information and the second state information correspondingly according to the configured fusion strategy to obtain A new state information, recorded as the target state information. It can be seen that the target status information carries the status information before and after image data preprocessing to a certain extent, which can more accurately reflect the shooting status of the actual shooting scene, and the image acquisition parameters calculated by using the target status information will be more accurate. . Correspondingly, the application processor 140 calculates a new image acquisition parameter by using the target state information obtained by fusion according to the configured image acquisition parameter calculation algorithm (eg, automatic exposure algorithm, automatic white balance algorithm, autofocus algorithm, etc.).

In addition, the application processor 140 is also connected with the image sensor 110, and controls the image sensor 110 to start collecting image data and end collecting image data. After the new image acquisition parameters are calculated, the application processor updates the image acquisition parameters of the image sensor 110 to the newly calculated image acquisition parameters. In this cycle, by continuously updating the image acquisition parameters of the image sensor 110 until the image acquisition parameters of the image sensor 110 converge, the best image acquisition effect is obtained.

It should be noted that the connection mode between the application processor 140 and the image sensor 110 is not specifically limited in the embodiment of the present application, and can be configured by those of ordinary skill in the art according to the actual situation. For example, in the embodiment of the present application, the application The processor 140 and the image sensor 110 are connected through an I2C bus.

Compared with the related art, the present application additionally adds an image signal pre-processor 120, and the image signal pre-processor 120 performs state information statistics on the raw image data collected by the image sensor 110, correspondingly obtains the first state information, and uses the image signal The signal pre-processor 120 pre-processes the original image data to obtain the pre-processed image data; then the image signal processor 130 performs state information statistics on the pre-processed image data, and correspondingly obtains the second state information; finally, the application processor 140 fuses the first state information counted by the image signal preprocessor 120 and the second state information counted by the image signal processor 130 to obtain target state information, and uses the target state information to update image acquisition parameters of the image sensor. In this way, the purpose of improving the accuracy of automatically configuring the image acquisition parameters of the electronic device can be achieved.

Optionally, please refer to FIG. 2 , the image signal pre-processor 120 includes:

an image signal processing unit 1201, configured to count the first state information of the image data; and

Perform the first preprocessing on the image data;

The neural network processing unit 1202 is configured to perform a second preprocessing on the image data after the first preprocessing to obtain preprocessing image data.

The first image signal processing unit 1201 is connected to the image sensor 110, and is used for performing statistics on the state information of the image data from the image sensor 110, and correspondingly obtaining the first state information. In addition, the image signal preprocessing unit 120 is further configured to perform the first preprocessing on the aforementioned image data according to the configured preprocessing strategy. It should be noted that the first preprocessing performed by the image signal processing unit 1201 is not specifically limited in the embodiments of the present application, including but not limited to dead pixel correction processing, temporal noise reduction processing, 3D noise reduction processing, linear optimization processing methods, such as processing and black level correction processing, of course, may also include optimization processing methods not listed in this application.

The neural network processing unit 1202 is configured to perform a second preprocessing on the image data after the first preprocessing to obtain preprocessing image data. Among them, the neural network processing unit 1202 is solidified with a variety of neural network algorithms (for example, video night scene algorithm based on neural network, video HDR algorithm, video blur algorithm, video noise reduction algorithm and video super-resolution algorithm, etc.), in the first image After the signal processing unit 1201 completes the first pre-processing of the image data, the neural network processing unit 1202 calls the corresponding neural network algorithm to perform the second pre-processing on the image data after the first pre-processing according to the configured pre-processing strategy, Get pre-processed image data.

In layman's terms, the image signal processing unit 1201 uses the non-AI image optimization method to perform preliminary optimization processing on the image data, and then the neural network processing unit 1202 uses the AI image optimization method to further optimize the image data after the preliminary optimization.

Optionally, in one embodiment, the application processor 140 is configured to:

obtaining a first weight value corresponding to the first state information, and obtaining a second weight value corresponding to the second state information;

Weighted summation is performed according to the first state information, the first weight value, the second state information and the second weight value to obtain the target state information.

The present application further provides an optional fusion strategy of the first state information and the second state information. The application processor 140 fuses the first state information and the second state information by means of weighted summation, and uses the weighted result as the target state information obtained by fusion, which can be expressed as:

S'=a*S ₁ +b*S ₂ ;

Wherein, S' represents the target state information obtained by fusion, S ₁ represents the first state information, S ₂ represents the second state information, a represents the weight assigned to the first state information, and b represents the weight assigned to the second state information.

It should be noted that the weight configuration manner of the first state information and the second state information is not specifically limited in the embodiments of the present application, and can be configured by those of ordinary skill in the art according to actual needs. For example, a static weight may be configured for the second state information and the first state information, that is, a fixed weight may be assigned to the second state information and the first state information in advance, or a dynamic weight may be configured for the second state information and the first state information The weight, that is, the second state information and the first state information, changes dynamically according to a specified condition.

For example, the weight of the first state information may be configured as 0.9, and the weight of the second state information may be configured as 0.1;

For another example, the weights of the first state information and the second state information can be dynamically allocated according to the preprocessing performed by the image signal preprocessor, wherein, the greater the image difference between the aforementioned image data before and after the preprocessing, the configuration of the first state information. The larger the weight of , the smaller the weight of configuring the second state information; the smaller the image difference before and after the pre-processing of the image data, the smaller the weight of configuring the first state information, and the greater the weight of configuring the second state information.

It should be noted that the first state information and the second state information include multiple state information of different dimensions, and the application processor 140 fuses the state information in each dimension according to the above fusion strategy provided in this application. The brightness in the state information is fused with the brightness in the second state information, the sharpness in the first state information and the sharpness in the second state information are fused, and so on.

Optionally, in an embodiment, the image signal pre-processor 120 is further configured to transmit the first state information and the pre-processed image data to the image signal processor 130;

The image signal processor 130 is further configured to perform post-processing on the pre-processed image data to obtain post-processed image data; and

Send the post-processed image data, the first state information, the second state information, and the first indication information to the application processor 140

The application processor 140 is further configured to obtain target state information by fusing the first state information and the second state information according to the first indication information.

Referring to FIG. 3 , in the embodiment of the present application, the image signal pre-processor 120 obtains the first state information of the image data from the image sensor 110 according to statistics, and pre-processes the aforementioned image data, and after obtaining the pre-processed image data, The first state information and the pre-processed image data are transmitted to the image signal processor 130 . For example, the image signal pre-processor 120 transmits the first state information obtained by statistics and the pre-processed image data obtained by the pre-processing to the image signal processor 130 based on the MIPI interface between it and the image signal processor 130 .

After the image signal processor 130 receives the first state information from the image signal pre-processor 120 and the pre-processed image data, in addition to performing state statistics on the pre-processed image data to obtain the second state information, the image signal processor 130 also performs state statistics on the pre-processed image data. The data is post-processed to obtain post-processed image data.

The embodiments of the present application do not specifically limit the post-processing performed by the image signal processor 130, and the difference from the pre-processing performed by the image signal pre-processor 120 is a constraint (that is, the image signal pre-processor 120 has performed the same optimization process. , the image signal processor 130 is no longer executed), which can be configured by those of ordinary skill in the art according to actual needs. For example, post-processing performed by the image signal processor 130, such as optimized processing methods such as strong light suppression, backlight compensation, color enhancement, and lens shadow correction, may, of course, also include optimized processing methods not listed in this application.

After the image signal processor 130 completes the post-processing of the pre-processed image data, correspondingly obtains the post-processed image data, and obtains the second state information by statistics, the image signal processor 130 generates the first indication information, and combines the post-processed image data, the first state information, and the second state information. The second state information and the first indication information are sent to the application processor 140 .

Correspondingly, the application processor 140 obtains the target state information by fusing the first state information and the second state information according to the first instruction.

Optionally, in an embodiment, the application processor 140 is further configured to:

When the post-processed image data is dynamic image data, preview the post-processed image data and/or perform video encoding according to the post-processed image data; or,

When the post-processed image data is a still image, image coding is performed according to the post-processed image data.

It should be noted that the image type does not change with the processing of the image data, that is, if the original image data is a static image, the pre-processed image data/post-processed image data obtained by the corresponding processing is also a static image, and the original image If the data is a dynamic image, the pre-processed image data/post-processed image data obtained by corresponding processing is also a dynamic image. Among them, the static image is such as a single frame of image captured in real time, the dynamic image is such as one frame of image in the image sequence collected during preview, and one frame of image in the image sequence collected during video recording.

Optionally, in an embodiment, the image signal pre-processor 120 is further configured to transmit the first state information and the pre-processed image data to the image signal processor 130 when the current transmission mode is configured as the first transmission mode .

It should be noted that two optional transmission modes are provided in this embodiment of the present application, which are a first transmission mode and a second transmission mode, respectively.

The image signal pre-processor 120 is configured as the first transmission mode (or synchronous transmission mode) in the current transmission mode, and transmits the first state information and the pre-processed image data to the image signal processor 130, and processes the image signal through the image signal processor 130. The processor 130 transmits the first state information to the application processor 140. In addition, the graphics signal processor 130 will also perform post-processing image data post-processing on the pre-processed image data, and second state information obtained from statistics on the post-processing image data. The first indication information is transmitted to the application processor 140 along with the first state information. For details, reference may be made to the relevant descriptions in the above embodiments, which will not be repeated here.

Optionally, in an embodiment, the image signal preprocessor 120 is further configured to send the first state information and the second indication information to the application processor 140 when the current transmission mode is configured as the second transmission mode;

The application processor 140 is further configured to obtain target state information by fusing the first state information and the second state information according to the second indication information.

Referring to FIG. 4 , in this embodiment of the present application, the image signal pre-processor 120 is also connected to the application processor 140. When the current transmission mode is configured as the second transmission mode, the image signal pre-processor 120 directly converts the first state information to transmitted to the application processor 140 .

On the other hand, the image signal pre-processor 120 also transmits the pre-processed image data obtained by pre-processing the aforementioned image data to the image signal processor 130 . Correspondingly, in addition to performing state statistics on the pre-processed image data to obtain the second state information, the image signal processor 130 also performs post-processing on the pre-processed image data to obtain post-processed image data.

The embodiments of the present application do not specifically limit the post-processing performed by the image signal processor 130, and the difference from the pre-processing performed by the image signal pre-processor 120 is a constraint (that is, the image signal pre-processor 120 has performed the same optimization process. , the image signal processor 130 is no longer executed), which can be configured by those of ordinary skill in the art according to actual needs. For example, post-processing performed by the image signal processor 130, such as optimized processing methods such as strong light suppression, backlight compensation, color enhancement, and lens shadow correction, may, of course, also include optimized processing methods not listed in this application.

The image signal processor 130 sends the post-processed image data and the second state information to the application processor 140 after completing the post-processing of the pre-processed image data, correspondingly obtaining the post-processed image data, and statistically obtaining the second state information.

The application processor 140 is further configured to obtain target state information by fusing the first state information and the second state information according to the second indication information.

Optionally, please refer to FIG. 5 , a first connection and a second connection are established between the image signal pre-processor 120 and the application processor 140, and the image signal pre-processor 120 is further configured to send the first state information through the first connection. to the application processor 140, and send the second indication information to the application processor 140 through the second connection.

For example, a first connection is established between the image signal pre-processing 120 and the application processor 140 through a serial peripheral interface (Serial Peripheral Interface, SPI), and a second connection is established through a general-purpose input/output (General-purpose input/output, GPIO) interface .

Optionally, in an embodiment, the image signal pre-processor 120 is further configured to:

Analyze according to historical image data to obtain analysis results;

Configure the transmission mode as the first mode or the second mode according to the analysis result.

The historical image data is image data collected before the aforementioned image data. The image signal pre-processor 120 analyzes the historical image data according to the configured analysis strategy according to the historical image data obtained before the aforementioned image data, obtains the analysis result, and then configures the transmission mode as the first mode and the second mode according to the analysis result.

Exemplarily, the historical image data is image data collected by the image sensor 110 before collecting the aforementioned image data. Since the aforementioned historical image data and the aforementioned image data are continuously collected, the aforementioned historical image data and the aforementioned image data have substantially the same image content. The image signal pre-processor 120 obtains the pre-processing duration of a preset number of historical image data, predicts the pre-processing duration of the image data according to the pre-processing duration of the preset number, and determines whether the pre-processing duration of the aforementioned image data reaches The preset duration, if the preset duration is reached, the transmission mode is configured as the second mode, and if the preset duration is not reached, the transmission mode is configured as the first mode. Wherein, the preset duration can be an empirical value obtained by those of ordinary skill in the art according to actual needs.

For example, when the preprocessing duration of the image data is predicted and obtained according to the preset number of preprocessing durations, the image signal preprocessor 120 may obtain the average processing duration of the preset number of preprocessing durations as the preprocessing duration of the aforementioned image data. Processing time.

For another example, when the pre-processing duration of the image data is predicted and obtained according to the pre-processing duration of the preset number, the image signal pre-processor 120 may perform a weighted sum on the pre-processing duration of the preset number, and use the weighted sum as the obtained weighted sum. The preprocessing duration of the aforementioned image data, wherein, for the aforementioned preset number of preprocessing durations, if the corresponding historical image data is collected earlier than the aforementioned image data, the weight is smaller.

Optionally, in an embodiment, there are multiple image sensors 110 , and the application processor 140 is further configured to update the image acquisition parameters of each image sensor 110 synchronously.

In the embodiment of the present application, for each image sensor 110, new image acquisition parameters are obtained according to the calculation method of the image acquisition parameters provided in any of the above embodiments, and then new image acquisition parameters are obtained according to the new image of each image sensor 110. Acquisition parameters, synchronously update the image acquisition parameters of each image sensor.

The present application also provides an image processing method, which is applied to the electronic equipment provided by the present application. Please refer to FIG. 6 and FIG. 1 in combination. The flow of the image processing method can be as follows:

In 201, image data is collected by the image sensor 110 according to the configured image collection parameters;

In 202, the image signal pre-processor 120 counts the state information in the image data for updating the image acquisition parameters of the image sensor 110 to obtain first state information, and pre-processes the image data to obtain pre-processed image data;

In 203, the image signal processor 130 counts the state information used to update the image acquisition parameters of the image sensor 110 in the pre-processed image data to obtain second state information;

In 204, target state information is obtained by fusing the first state information and the second state information by the application processor 140, and the image acquisition parameters of the image sensor 110 are updated according to the target state information.

In this embodiment of the present application, the image sensor 110 is connected to the image signal pre-processor 120 for collecting image data according to the configured image acquisition parameters, and transmitting the collected image data to the image signal pre-processor 120 . It should be noted that the image data collected by the image sensor 110 is image data in RAW format, and the collected image data in original RAW format is transmitted to the image signal pre-processor 120 .

The image acquisition parameters include, but are not limited to, exposure parameters, focus parameters, and white balance parameters. For example, before starting to collect image data, the application processor 140 will configure initial exposure parameters to the image sensor 110, so that the image sensor 110 collects image data according to the initial exposure parameters, and transmits the collected image data to the image sensor 110. Signal preprocessor 120 .

It should be noted that the connection mode between the image signal pre-processor 120 and the image sensor 110 is not specifically limited in the embodiments of the present application. For example, the MIPI (Mobile Industry Processor Interface, Mobile Industry Processor Interface) connection.

Wherein, when the image sensor 110 transmits image data to the image signal pre-processor 120, the image data is encapsulated into a plurality of image data packets, and transmitted to the image signal pre-processor in the form of image data packets. Exemplarily, the image data packet includes a header field, a trailer field and a data field, wherein the header field and the trailer field are used to fill in some necessary control information, such as synchronization information, address information, error control information, etc., and the data field is Used to populate the actual image content.

On the other hand, the image signal preprocessor 120 receives image data from the image sensor 110 . In addition, the image signal pre-processor 120 is also connected to the image signal processor 130, wherein the connection mode of the image signal processor 130 and the image signal pre-processor 120 is not specifically limited in the embodiment of the present application, for example, the image signal processing The connection between the processor 130 and the image signal pre-processor 120 may also be through MIPI.

Wherein, after receiving the image data transmitted from the image sensor 110, the image signal pre-processor 120 uses the image data to count the state information required by the application processor 140 to update the image acquisition parameters of the image sensor 110, including but not limited to brightness information, sharpness information, contrast information, etc., and the state information calculated by using the aforementioned image data is recorded as the first state information. In addition, the image signal pre-processor 120 also pre-processes the image data from the image sensor 110 according to the configured pre-processing strategy, improves the image quality of the image data, and obtains the pre-processed image data accordingly. It should be noted that the preprocessing performed by the image signal preprocessor 120 on the image data does not change the format of the image data, that is, the preprocessed image data obtained by the preprocessing is still in the RAW format.

After completing the pre-processing of the image data and obtaining the pre-processed image data, the image signal pre-processor 120 transmits the pre-processed image data to the image signal processor 130 .

After receiving the pre-processed image data from the image signal pre-processor 120, the image signal processor 130 uses the pre-processed image data to count the state information required by the application processor 140 to update the image acquisition parameters of the image sensor 110, including: However, it is not limited to brightness information, sharpness information, contrast information, etc., and the state information calculated by using the pre-processed image data is recorded as the second state information.

In addition, for the first state information, the image signal pre-processor 120 may directly transmit the first state information to the application processor 140 , or transmit the first state information to the application processor 140 via the image signal processor 130 .

After the image signal preprocessor 120 statistically obtains the first state information, and the image signal processor 130 obtains the second state information through statistics, the application processor 140 further fuses the first state information and the second state information correspondingly according to the configured fusion strategy to obtain A new state information, recorded as the target state information. It can be seen that the target status information carries the status information before and after image data preprocessing to a certain extent, which can more accurately reflect the shooting status of the actual shooting scene, and the image acquisition parameters calculated by using the target status information will be more accurate. . Correspondingly, the application processor 140 calculates a new image acquisition parameter by using the target state information obtained by fusion according to the configured image acquisition parameter calculation algorithm (eg, automatic exposure algorithm, automatic white balance algorithm, autofocus algorithm, etc.).

In addition, the application processor 140 is also connected with the image sensor 110, and controls the image sensor 110 to start collecting image data and end collecting image data. After the new image acquisition parameters are calculated, the application processor updates the image acquisition parameters of the image sensor 110 to the newly calculated image acquisition parameters. In this cycle, by continuously updating the image acquisition parameters of the image sensor 110 until the image acquisition parameters of the image sensor 110 converge, the best image acquisition effect is obtained.

It should be noted that the connection mode between the application processor 140 and the image sensor 110 is not specifically limited in the embodiment of the present application, and can be configured by those of ordinary skill in the art according to the actual situation. For example, in the embodiment of the present application, the application The processor 140 and the image sensor 110 are connected through an I2C bus.

Optionally, in an embodiment, the target state information is obtained by fusing the first state information and the second state information by the application processor 140, including:

Obtain a first weight value corresponding to the first state information, and obtain a second weight value corresponding to the second state information through the application processor 140;

The target state information is obtained by the application processor 140 performing weighted summation according to the first state information, the first weight value, the second state information and the second weight value.

The present application further provides an optional fusion strategy of the first state information and the second state information. The application processor 140 fuses the first state information and the second state information by means of weighted summation, and uses the weighted result as the target state information obtained by fusion, which can be expressed as:

S'=a*S ₁ +b*S ₂ ;

Wherein, S' represents the target state information obtained by fusion, S ₁ represents the first state information, S ₂ represents the second state information, a represents the weight assigned to the first state information, and b represents the weight assigned to the second state information.

It should be noted that the weight configuration manner of the first state information and the second state information is not specifically limited in the embodiments of the present application, and can be configured by those of ordinary skill in the art according to actual needs. For example, a static weight may be configured for the second state information and the first state information, that is, a fixed weight may be assigned to the second state information and the first state information in advance, or a dynamic weight may be configured for the second state information and the first state information The weight, that is, the second state information and the first state information, changes dynamically according to a specified condition.

For example, the weight of the first state information may be configured as 0.9, and the weight of the second state information may be configured as 0.1;

For another example, the weights of the first state information and the second state information can be dynamically allocated according to the preprocessing performed by the image signal preprocessor, wherein, the greater the image difference between the aforementioned image data before and after the preprocessing, the configuration of the first state information. The larger the weight of , the smaller the weight of configuring the second state information; the smaller the image difference before and after the pre-processing of the image data, the smaller the weight of configuring the first state information, and the greater the weight of configuring the second state information.

It should be noted that the first state information and the second state information include multiple state information of different dimensions, and the application processor 140 fuses the state information in each dimension according to the above fusion strategy provided in this application. The brightness in the state information is fused with the brightness in the second state information, the sharpness in the first state information and the sharpness in the second state information are fused, and so on.

Optionally, in an embodiment, the image processing method provided by this application further includes:

The first state information and the pre-processed image data are transmitted to the image signal processor 130 through the image signal pre-processor 120;

Perform post-processing on the pre-processed image data by the image signal processor 130 to obtain post-processed image data;

Send the post-processed image data, the first state information, the second state information and the first indication information to the application processor 140 through the image signal processor 130;

The target state information is obtained by fusing the first state information and the second state information according to the first indication information by the application processor 140 .

Referring to FIG. 3 , in the embodiment of the present application, the image signal pre-processor 120 obtains the first state information of the image data from the image sensor 110 according to statistics, and pre-processes the aforementioned image data, and after obtaining the pre-processed image data, The first state information and the pre-processed image data are transmitted to the image signal processor 130 . For example, the image signal pre-processor 120 transmits the first state information obtained by statistics and the pre-processed image data obtained by the pre-processing to the image signal processor 130 based on the MIPI interface between it and the image signal processor 130 .

After the image signal processor 130 receives the first state information from the image signal pre-processor 120 and the pre-processed image data, in addition to performing state statistics on the pre-processed image data to obtain the second state information, the image signal processor 130 also performs state statistics on the pre-processed image data. The data is post-processed to obtain post-processed image data.

The embodiments of the present application do not specifically limit the post-processing performed by the image signal processor 130, and the difference from the pre-processing performed by the image signal pre-processor 120 is a constraint (that is, the image signal pre-processor 120 has performed the same optimization process. , the image signal processor 130 is no longer executed), which can be configured by those of ordinary skill in the art according to actual needs. For example, post-processing performed by the image signal processor 130, such as optimized processing methods such as strong light suppression, backlight compensation, color enhancement, and lens shadow correction, may, of course, also include optimized processing methods not listed in this application.

After the image signal processor 130 completes the post-processing of the pre-processed image data, correspondingly obtains the post-processed image data, and obtains the second state information by statistics, the image signal processor 130 generates the first indication information, and combines the post-processed image data, the first state information, and the second state information. The second state information and the first indication information are sent to the application processor 140 .

Correspondingly, the application processor 140 obtains the target state information by fusing the first state information and the second state information according to the first instruction.

Optionally, in an embodiment, the image processing method provided by this application further includes:

When the post-processed image data is dynamic image data, preview the post-processed image data through the application processor 140 and/or perform video encoding according to the post-processed image data; or,

When the post-processed image data is a still image, the application processor 140 performs image coding according to the post-processed image data.

It should be noted that the image type does not change with the processing of the image data, that is, if the original image data is a static image, the pre-processed image data/post-processed image data obtained by the corresponding processing is also a static image, and the original image If the data is a dynamic image, the pre-processed image data/post-processed image data obtained by corresponding processing is also a dynamic image. Among them, the static image is such as a single frame of image captured in real time, the dynamic image is such as one frame of image in the image sequence collected during preview, and one frame of image in the image sequence collected during video recording.

Optionally, in an embodiment, the first state information and the pre-processed image data are transmitted to the image signal processor 130 through the image signal pre-processor 120, including:

When the current transmission mode is configured as the first transmission mode, the first state information and the pre-processed image data are transmitted to the image signal processor 130 through the image signal pre-processor 120 .

It should be noted that two optional transmission modes are provided in this embodiment of the present application, which are a first transmission mode and a second transmission mode, respectively.

The image signal pre-processor 120 is configured as the first transmission mode (or synchronous transmission mode) in the current transmission mode, and transmits the first state information and the pre-processed image data to the image signal processor 130, and processes the image signal through the image signal processor 130. The processor 130 transmits the first state information to the application processor 140. In addition, the graphics signal processor 130 will also perform post-processing image data post-processing on the pre-processed image data, and second state information obtained from statistics on the post-processing image data. The first indication information is transmitted to the application processor 140 along with the first state information. For details, reference may be made to the relevant descriptions in the above embodiments, which will not be repeated here.

Optionally, in an embodiment, the image processing method provided by this application further includes:

When the current transmission mode is configured as the second transmission mode, send the first state information and the second indication information to the application processor 140 through the image signal preprocessor 120;

The target state information is obtained by fusing the first state information and the second state information according to the second indication information by the application processor 140 .

Referring to FIG. 4 , in this embodiment of the present application, the image signal pre-processor 120 is also connected to the application processor 140. When the current transmission mode is configured as the second transmission mode, the image signal pre-processor 120 directly converts the first state information to transmitted to the application processor 140 .

On the other hand, the image signal pre-processor 120 also transmits the pre-processed image data obtained by pre-processing the aforementioned image data to the image signal processor 130 . Correspondingly, in addition to performing state statistics on the pre-processed image data to obtain the second state information, the image signal processor 130 also performs post-processing on the pre-processed image data to obtain post-processed image data.

The embodiments of the present application do not specifically limit the post-processing performed by the image signal processor 130, and the difference from the pre-processing performed by the image signal pre-processor 120 is a constraint (that is, the image signal pre-processor 120 has performed the same optimization process. , the image signal processor 130 is no longer executed), which can be configured by those of ordinary skill in the art according to actual needs. For example, post-processing performed by the image signal processor 130, such as optimized processing methods such as strong light suppression, backlight compensation, color enhancement, and lens shadow correction, may, of course, also include optimized processing methods not listed in this application.

The image signal processor 130 sends the post-processed image data and the second state information to the application processor 140 after completing the post-processing of the pre-processed image data, correspondingly obtaining the post-processed image data, and statistically obtaining the second state information.

The application processor 140 is further configured to obtain target state information by fusing the first state information and the second state information according to the second indication information.

Optionally, in an embodiment, the image processing method provided by this application further includes:

Analyze according to historical image data to obtain analysis results;

Configure the transmission mode as the first mode or the second mode according to the analysis result.

The historical image data is image data collected before the aforementioned image data. The image signal preprocessor 120 analyzes the historical image data according to the configured analysis strategy according to the historical image data obtained before the aforementioned image data, obtains the analysis result, and then configures the transmission mode as the first mode and the second mode according to the analysis result.

Exemplarily, the historical image data is image data collected by the image sensor 110 before collecting the aforementioned image data. Since the aforementioned historical image data and the aforementioned image data are continuously collected, the aforementioned historical image data and the aforementioned image data have substantially the same image content. The image signal pre-processor 120 obtains the pre-processing duration of a preset number of historical image data, predicts the pre-processing duration of the image data according to the pre-processing duration of the preset number, and determines whether the pre-processing duration of the aforementioned image data reaches The preset duration, if the preset duration is reached, the transmission mode is configured as the second mode, and if the preset duration is not reached, the transmission mode is configured as the first mode. Wherein, the preset duration can be an empirical value obtained by those of ordinary skill in the art according to actual needs.

For example, when the preprocessing duration of the image data is predicted and obtained according to the preset number of preprocessing durations, the image signal preprocessor 120 may obtain the average processing duration of the preset number of preprocessing durations as the preprocessing duration of the aforementioned image data. Processing time.

For another example, when the pre-processing duration of the image data is predicted and obtained according to the pre-processing duration of the preset number, the image signal pre-processor 120 may perform a weighted sum on the pre-processing duration of the preset number, and use the weighted sum as the obtained weighted sum. The preprocessing duration of the aforementioned image data, wherein, for the aforementioned preset number of preprocessing durations, if the corresponding historical image data is collected earlier than the aforementioned image data, the weight is smaller.

Optionally, in an embodiment, there are multiple image sensors 110, and the image processing method provided by this application further includes:

The image acquisition parameters of each image sensor 110 are synchronously updated by the application processor 140 .

In the embodiment of the present application, for each image sensor 110, new image acquisition parameters are obtained according to the calculation method of the image acquisition parameters provided in any of the above embodiments, and then new image acquisition parameters are obtained according to the new image of each image sensor 110. Acquisition parameters, synchronously update the image acquisition parameters of each image sensor.

The electronic device and the image processing method provided by the embodiments of the present application have been described in detail above. The principles and implementations of the present application are described herein by using specific examples, and the descriptions of the above embodiments are only used to help the understanding of the present application. At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific embodiments and application scope. To sum up, the content of this specification should not be construed as a limitation to the present application.

Claims (10)

1. An electronic device, comprising:

the image sensor is used for acquiring image data according to the configured image acquisition parameters;

the image signal preprocessor is used for counting state information used for updating image acquisition parameters of the image sensor in the image data to obtain first state information, and preprocessing the image data to obtain preprocessed image data;

the image signal processor is used for counting state information used for updating the image acquisition parameters of the image sensor in the pre-processing image data to obtain second state information;

and the application processor is used for fusing the first state information and the second state information to obtain target state information and updating the image acquisition parameters of the image sensor according to the target state information.

2. The electronic device of claim 1, wherein the application processor is configured to:

acquiring a first weight value corresponding to the first state information and acquiring a second weight value corresponding to the second state information;

and carrying out weighted summation according to the first state information, the first weight value, the second state information and the second weight value to obtain the target state information.

3. The electronic device of claim 1, wherein the image signal pre-processor is further configured to transmit the first state information and the pre-processed image data to the image signal processor;

the image signal processor is also used for carrying out post-processing on the pre-processed image data to obtain post-processed image data; and

sending the post-processing image data, the first state information, the second state information and first indication information to the application processor;

the application processor is further configured to obtain the target state information by fusing the first state information and the second state information according to the first indication information.

4. The electronic device of claim 3, wherein the image signal pre-processor is further configured to transmit the first status information and the pre-processed image data to the image signal processor when a current transmission mode is configured as a first transmission mode.

5. The electronic device of claim 4, wherein the image signal pre-processor is further configured to send the first status information and second indication information to the application processor when a current transmission mode is configured as a second transmission mode;

the application processor is further configured to obtain the target state information by fusing the first state information and the second state information according to the second indication information.

6. The electronic device of claim 4 or 5, wherein the image signal pre-processor is further configured to:

analyzing according to the historical image data to obtain an analysis result;

and configuring the transmission mode to be a first mode or a second mode according to the analysis result.

7. The electronic device of any of claims 1-5, wherein the image sensor is multiple, and wherein the application processor is further configured to update image acquisition parameters of each image sensor synchronously.

8. An image processing method applied to an electronic device, wherein the electronic device comprises an image sensor, an image signal pre-processor, an image signal processor and an application processor, the image processing method comprising:

acquiring image data through the image sensor according to the configured image acquisition parameters;

counting state information used for updating image acquisition parameters of the image sensor in the image data through the image signal preprocessor to obtain first state information, and preprocessing the image data to obtain preprocessed image data;

counting state information used for updating image acquisition parameters of the image sensor in the pre-processing image data through the image signal processor to obtain second state information;

and fusing the first state information and the second state information through the application processor to obtain target state information, and updating the image acquisition parameters of the image sensor according to the target state information.

9. The image processing method according to claim 8, further comprising:

transmitting the first state information and the pre-processed image data to the image signal processor through the image signal pre-processor when a current transmission mode is configured as a first transmission mode;

post-processing the pre-processed image data through the image signal processor to obtain post-processed image data; and sending the post-processing image data, the first state information, the second state information and first indication information to the application processor;

and fusing the first state information and the second state information according to the first indication information through the application processor to obtain the target state information.

10. The image processing method according to claim 8, further comprising:

transmitting the first state information and second indication information to the application processor through the image signal pre-processor when the current transmission mode is configured as a second transmission mode;

and fusing the first state information and the second state information according to the second indication information through the application processor to obtain the target state information.

Images