WO2023011032A1

WO2023011032A1 - Image processing method, image processing chip, and electronic device

Info

Publication number: WO2023011032A1
Application number: PCT/CN2022/100535
Authority: WO
Inventors: 吴义孝
Original assignee: 哲库科技(上海)有限公司
Priority date: 2021-08-05
Filing date: 2022-06-22
Publication date: 2023-02-09
Also published as: CN115706768A

Abstract

The present application discloses an image processing method, an image processing chip, and an electronic device. The method comprises: obtaining running state information by monitoring a running state of an image processor; determining, according to the running state information, an optimal working mode matched with the current running state information for the image processor; when the image processor runs ina first working mode, performing frame separation processing on image data; and when the image processor runs in a second working mode, performing downsampling processing and frame separation processing on the image data. Adaptive adjustment of the working mode of the image processor is implemented, such that power consumption of the image processor is optimized, the problem of high power consumption in processing high-frame-rate images for a long time in a single working mode is solved, and the objective of balancing the power consumption of the image processor and an image processing effect is achieved.

Description

Image processing method, image processing chip and electronic equipment

Cross References to Related Applications

This application is based on the prior Chinese patent application with the application number 202110894743.8, the application date is August 5, 2021, and the invention title is "an image processing method, image processing chip and electronic equipment", and requires the prior Chinese patent application Priority of the patent application, the entire content of this prior Chinese patent application is hereby incorporated by reference in its entirety into this application.

technical field

The present application relates to image processing technology, in particular to an image processing method, an image processing chip and electronic equipment.

Background technique

With the development of image processing technology, it is very common for electronic equipment to be equipped with display screens with high resolution and high frame rate, but it is also accompanied by the problem of high power consumption when the image processor of the electronic equipment performs image processing. At present, an image processor is used to preprocess images in electronic equipment, but when processing images with high resolution and high frame rate for a long time, the problem of high power consumption of the image processor is more serious.

Contents of the invention

In order to solve the above technical problems, the embodiments of the present application expect to provide an image processing method, an image processing chip, and an electronic device.

The technical scheme of the present application is realized like this:

In a first aspect, an image processing method is provided, including:

Acquiring the running state information of the image processor;

Determining that the image processor is running in a first working mode or a second working mode according to the running status information, wherein when the image processor is running in the first working mode, image data is processed every other frame; When the image processor operates in the second working mode, it performs down-sampling processing and frame-interval processing on the image data.

In a second aspect, an image processing chip is provided, including: an image processor and an application processor coupled to the image processor, the image processor is configured to process received image data, and the image processor has a first working mode and a second working mode,

Wherein, when the image processor operates in the first working mode, the image processor performs frame-interval processing on the image data to obtain the first image data;

When the image processor is running in the second working mode, the image processor performs down-sampling processing on the image data and then performs frame-interval processing to obtain the first image data;

The application processor is configured to process the first image data.

In a third aspect, an electronic device is provided, and the electronic device includes:

an image acquisition device configured to acquire image data;

An image processor configured to execute the method steps in the aforementioned first aspect implemented by the image processor.

Description of drawings

FIG. 1 is a schematic diagram of a first flow chart of an image processing method in an embodiment of the present application;

Fig. 2 is a second schematic flow chart of the image processing method in the embodiment of the present application;

FIG. 3 is a schematic diagram of classification of image processor working modes in the embodiment of the present application;

FIG. 4 is a schematic diagram of a third flow chart of an image processing method in an embodiment of the present application;

FIG. 5 is a schematic flow chart of frame-interval processing in the embodiment of the present application;

FIG. 6 is a schematic diagram of a fourth flow chart of an image processing method in an embodiment of the present application;

FIG. 7 is a schematic flow chart of downsampling and upsampling processing in the embodiment of the present application;

FIG. 8 is a schematic diagram of a fifth flow chart of the image processing method in the embodiment of the present application;

9 is a schematic diagram of the composition and structure of the image processing chip in the embodiment of the present application;

FIG. 10 is a schematic diagram of the composition and structure of the electronic device in the embodiment of the present application.

Detailed ways

An embodiment of the present application provides an image processing method applied to an image processor, wherein the method includes:

Acquiring the running state information of the image processor;

In some embodiments, when the image processor operates in the first working mode, performing frame-interval processing on the image data includes: performing frame-interval processing on the image data to obtain the first image data and 3A statistical data of the processed image; sending the first image data and the 3A statistical data to an application processor, so that the application processor can analyze the first image data by the 3A statistical data of the processed image Perform image compensation on the unprocessed image to obtain second image data.

In some embodiments, when the image processor operates in the second working mode, performing down-sampling processing and frame-interval processing on the image data includes: performing down-sampling processing on the image data, and performing down-sampling processing on the down-sampled The sampled image data is processed every frame to obtain the first image data and the 3A statistical data of the processed image; the first image data and the 3A statistical data are sent to the application processor for the application processor Perform image compensation on the unprocessed image in the first image data by using the 3A statistical data of the processed image, and perform up-sampling processing on the compensated image data to obtain the second image data.

In some embodiments, the first image data and the 3A statistical data are sent to the application processor, so that the application processor can use the 3A statistical data of the processed image of the previous frame to perform the processing on the unprocessed image of the current frame. Image compensation processing; or, sending the first image data and the 3A statistical data to the application processor, so that the application processor can analyze the 3A statistical data of the processed image of the previous frame and the processed image of the next frame Perform weighting processing to obtain weighted 3A statistical data; perform image compensation processing on the unprocessed image of the current frame through the weighted 3A statistical data.

In some embodiments, the method further includes: determining that the image processor is running in a third working mode according to the running state information, wherein when the image processor is running in the third working mode, the Some modules in the image processor are set to bypass mode.

In some embodiments, the method further includes: from preset operating conditions, determining a target operating condition satisfied by the operating state information; when the target operating condition is the first operating condition, determining that the image processing The processor operates in the first operating mode; when the target operating condition is the second operating condition, determine that the image processor is operating in the second operating mode; when the target operating condition is the third operating condition, determine The image processor operates in the third working mode.

In some embodiments, the operating status information includes at least one of the following: temperature of the image processor, power consumption of the image processor for a preset working time,

The operating conditions include at least one of the following: the temperature of the image processor is within a preset temperature range; the power consumption of the image processor is within a preset power consumption range.

The embodiment of the present application also provides an image processor, the image processor is configured to call and run the computer program stored in the memory, and execute any method steps implemented by the image processor in the embodiments of the present application.

The embodiment of the present application also provides an image processing chip, which includes:

an image processor and an application processor coupled to the image processor, the image processor is configured to process received image data, the image processor has a first mode of operation and a second mode of operation,

The application processor is configured to process the first image data.

In some embodiments, the image processor is configured to process the image data every other frame to obtain the first image data and 3A statistical data of the processed image; the first image data and the 3A statistical data Data is sent to the application processor;

The application processor is configured to perform image compensation on the unprocessed image in the first image data by using the 3A statistical data of the processed image to obtain the second image data.

In some embodiments, the image processor is configured to perform down-sampling processing on the image data, and perform frame-interval processing on the down-sampled image data to obtain the first image data and 3A statistical data of the processed image; sending the first image data and the 3A statistical data to an application processor;

The application processor is configured to perform image compensation on the unprocessed image in the first image data by using the 3A statistical data of the processed image, and perform up-sampling processing on the compensated image data to obtain the second image data.

In some embodiments, the application processor is configured to perform image compensation processing on the unprocessed image of the current frame by using the 3A statistical data of the processed image of the previous frame;

Alternatively, the application processor is configured to perform weighted processing on the 3A statistical data of the processed image of the previous frame and the processed image of the next frame to obtain weighted 3A statistical data; Perform image compensation processing.

The embodiment of the present application also provides an electronic device, and the electronic device includes:

an image acquisition device configured to acquire image data;

An image processor configured to execute any method steps implemented by the image processor in the embodiments of the present application.

In some embodiments, the electronic device further includes an application processor configured to receive the first image data processed by the image processor and process the first image processing data.

The embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium is configured to store a computer program, and the computer program enables the computer to execute the image processor or application processor in the embodiment of the present application. method steps.

In order to understand the characteristics and technical contents of the embodiments of the present application in more detail, the implementation of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. The attached drawings are only for reference and description, and are not intended to limit the embodiments of the present application.

An embodiment of the present application provides an image processing method for adaptively adjusting a working mode of an image processor, and the method is applied to an image processor. Electronic equipment includes an application processor (Application Processor, AP) and an image processor (Image Signal Processor, ISP). The ISP can also be called a front-end image processor (Previous Image Signal Processor, Pre-ISP). The image processor performs front-end image processor operations on the original image data collected by the image acquisition device, and the application processor is used to perform back-end image processing operations.

The electronic equipment described in this application has a shooting function, and the electronic equipment may include mobile phones, tablet computers, notebook computers, palmtop computers, personal digital assistants (Personal Digital Assistant, PDA), portable media players (Portable Media Player, PMP) , wearable devices, cameras, smart cars, etc.

The following provides a detailed illustration of an image processing method in the embodiment of the present application. Fig. 1 is a schematic flow chart of the first image processing method in the embodiment of the present application. As shown in Fig. 1, the method may specifically include:

Step 101: Obtain the running status information of the image processor;

Here, the running state information is used to characterize the power consumption of the image processor, and the best working mode matching the current power consumption of the image processor is determined according to the running state information. Exemplarily, the running state information includes at least one of the following: the temperature of the image processor, and the power consumption of the image processor within a preset working period. The image processor includes a temperature detection device and a power consumption detection device, the temperature detection device is used for detecting temperature, and the power consumption detection device is used for detecting power consumption.

Temperature can be correlated to characterize the power consumption of an image processor, with higher temperatures indicating greater power consumption. The aforementioned preset working duration may be 30 seconds, 60 seconds, 90 seconds, 120 seconds, 5 minutes, etc.

Step 102: Determine whether the image processor is running in the first working mode or the second working mode according to the running state information, wherein when the image processor is running in the first working mode, the image data is isolated Frame processing: when the image processor operates in the second working mode, down-sampling processing and frame-interval processing are performed on the image data.

The working modes of the image processor include at least a first working mode and a second working mode, and the image sensor operates in different working modes under different power consumption.

The first working mode and the second working mode may be referred to as low power consumption modes, and the power consumption of the image processor running the first working mode is greater than the power consumption of running the second working mode. The low power consumption mode can be understood as the simplified processing of the input image data by the ISP, and the ISP in different low power consumption modes adopts different simplified processing strategies to process the image data. In the embodiment of the present application, in the low power consumption mode, the ISP performs down-sampling and frame-interval processing on the high frame rate and high resolution video to reduce the amount of processed data.

Exemplarily, in some embodiments, the method further includes: setting a plurality of operating conditions in advance, and different operating conditions correspond to different working modes; correspondingly, the operating conditions include at least one of the following: the temperature of the image processor is at a preset within the preset temperature range; the power consumption of the image processor is within the preset power consumption range. Different operating conditions correspond to different temperature ranges and different power consumption ranges.

From preset operating conditions, determine a target operating condition satisfied by the operating state information; when the target operating condition is a first operating condition, determine that the image processor is operating in the first working mode; the When the target running condition is the second running condition, it is determined that the image processor runs in the second working mode.

Exemplarily, in some embodiments, the working mode further includes a high-performance mode, and the power consumption of the image processor running in any low-power consumption mode is less than the power consumption of running in the high-performance mode. The high-performance mode can be understood as not making any adjustments to the processing process of the image processor, and it is determined according to the operation state information that the image processor is running in the high-performance mode, and the image processor can normally process the input image data. In practical applications, when it is determined according to the operating state information that the current power consumption of the image sensor is low, the high-performance mode is operated to achieve high-quality image processing effects.

By adopting the above technical solution, the self-adaptive adjustment of the working mode of the image processor is realized, thereby optimizing the power consumption of the image processor, solving the problem of high power consumption in processing high frame rate images for a long time in a single working mode, and achieving a balanced image processing for the purpose of device power consumption and image processing effects.

Based on the above-mentioned embodiments, the image processing method is further illustrated below. FIG. 2 is a second schematic flow diagram of the image processing method in the embodiment of the present application. As shown in FIG. 2 , the processing method specifically includes:

Step 201: Obtain the running status information of the image processor;

Step 202: From the preset operating conditions, determine the target operating conditions that the operating state information satisfies;

Exemplarily, in some embodiments, the method further includes: setting a plurality of operating conditions in advance, and different operating conditions correspond to different working modes;

Correspondingly, the running condition includes at least one of the following: the temperature of the image processor is within a preset temperature range; the power consumption of the image processor is within a preset power consumption range. Different operating conditions correspond to different temperature ranges and different power consumption ranges. The temperature threshold in the temperature range can be preset in the product development stage based on experimental data, and the power consumption threshold in the power consumption range can be preset in the product development stage based on experimental data. Temperature ranges and power dissipation ranges include upper and/or lower limits.

Exemplarily, the running state information includes at least one of the following: the temperature of the image processor, and the power consumption of the image processor within a preset working period. Correspondingly, the running state information includes the temperature and power consumption of the ISP, and the first running condition corresponding to the first working mode includes: the temperature is in the first temperature range, or the power consumption is in the first power consumption range. Specifically, if the temperature is within the first temperature range, or the power consumption is within the first power consumption range, it is determined that the running state information satisfies the first running condition; if the temperature is outside the first temperature range, and the power consumption is within the first If it is outside the power consumption range, it is determined that the running state information does not meet the first running condition. That is to say, when either the temperature or the power consumption is within the preset range, it is determined that the first operating condition is satisfied; otherwise, it is determined that the first operating condition is not satisfied.

In some other embodiments, the first operating condition includes: the temperature is within a first temperature range, and the power consumption is within a first power consumption range. That is to say, when the temperature and power consumption of the ISP meet the first operating condition at the same time, it is determined that the operating state information satisfies the first operating condition; otherwise, it is determined that the operating state information does not meet the first operating condition.

In some other embodiments, the running state information only includes the temperature of the ISP, and the first running condition includes: the temperature is in the first temperature range; or the running state information only includes the power consumption of the ISP, and the first running condition includes: the power consumption is in the first temperature range; A power consumption range.

Step 203: When the target operating condition is the first operating condition, determine that the image processor is operating in the first working mode;

Wherein, when the image processor operates in the first working mode, the image data is processed every other frame.

Step 204: When the target operating condition is the second operating condition, determine that the image processor is operating in the second working mode;

Wherein, when the image processor operates in the second working mode, it performs down-sampling processing and frame-interval processing on the image data.

Referring to the first operating condition, the second operating condition corresponds to a second temperature range and a second power consumption range.

Step 205: When the target operating condition is a third operating condition, determine that the image processor is operating in the third working mode.

Wherein, when the image processor is running in the third working mode, some modules in the image processor are set to bypass mode.

Referring to the first operating condition, the third operating condition corresponds to a third temperature range and a third power consumption range.

It can be understood that the first operating condition includes: the temperature is in the first temperature range, and/or the power consumption is in the first power consumption range. The second operating condition includes: the temperature is within the second temperature range, and/or the power consumption is within the second power consumption range. The third operating condition includes: the temperature is in the third temperature range, and/or the power consumption is in the third power consumption range. In practical applications, the temperature range and power consumption range can be set according to the test results. The first temperature range, the second temperature range, and the third temperature range can be different or partially the same. The first power consumption range, the second power consumption range The range and the third power consumption range may be different or partially the same.

Exemplarily, in some embodiments, the working mode also includes a high-performance mode, and the high-performance mode is set with corresponding high-performance operating conditions, and the power consumption of the image processor running in any low-power mode is less than that of running in all low-power modes. power consumption in high-performance mode. The high-performance mode can be understood as not making any adjustments to the processing process of the image processor, and it is determined according to the operation state information that the image processor is running in the high-performance mode, and the image processor can normally process the input image data. In practical applications, when it is determined according to the operating state information that the current power consumption of the image sensor is low, the high-performance mode is operated to achieve high-quality image processing effects.

Exemplarily, FIG. 3 is a schematic diagram of the classification of image processor working modes in the embodiment of the present application. As shown in FIG. There are four working modes, that is, a high-performance mode, a first working mode, a second working mode and a third working mode. When the image processor runs these four working modes, the power consumption is in order from high to low. The first operation mode, the second operation mode and the third operation mode may be referred to as low power consumption modes.

The control device determines the optimal working mode of the image processor according to the operating state information of the image processor. For example, the operating state information includes temperature and power consumption, and each threshold includes a temperature threshold and a power consumption threshold. When the temperature and power consumption are less than the threshold When 1, the image processor is controlled to enter the high-performance mode; when the temperature or power consumption is greater than or equal to threshold 1, and both are less than threshold 2, the image processor is controlled to enter the first working mode; when the temperature or power consumption is greater than or equal to threshold 2, and When both are less than the threshold 3, the image processor is controlled to enter the second working mode; when the temperature or power consumption is greater than or equal to the threshold 3, the image processor is controlled to enter the third working mode.

Here, the operating conditions corresponding to different working modes are set through threshold 1, threshold 2 and threshold 3, the operating status information is compared with the operating conditions, and the target operating conditions satisfied by the operating status information are judged, so as to determine the target operating conditions corresponding to the target operating conditions. model.

The control device may belong to the image processor, that is, the control device inside the image processor controls the switching of the working mode, and the control device may belong to the application processor or other processors coupled with the image processor.

It should be noted that when the image processor further includes a fourth working mode, the fourth working mode corresponds to a fourth operating condition, and the fourth operating condition corresponds to a fourth temperature range and a fourth power consumption range. Here, the fourth working mode means that other working modes may be included in order to represent the image processor, and it is not intended to limit the number of working modes.

It should be noted that the present application uses the terms first, second, third, etc. to describe various information, but the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another and are not necessarily used to describe a specific order or sequence.

Based on the above-mentioned embodiments, the image processing method when the ISP works in the first working mode is illustrated below. FIG. 4 is a schematic flow chart of the third image processing method in the embodiment of the present application. As shown in FIG. In one working mode, the image processing method specifically includes:

Step 401: Obtain the running status information of the image processor;

Step 402: Determine that the image processor is running in a first working mode according to the running status information;

Taking FIG. 3 as an example, when it is detected that the running state information of the ISP is between the threshold 1 and the threshold 2, it is determined that the best working mode of the ISP is the first working mode.

Step 403: Processing the image data at intervals to obtain the first image data and 3A statistical data of the processed image;

Here, the image data is the original image data collected by the image acquisition device.

Exemplarily, when processing at intervals, the number of interval frames may be one frame or two frames. According to the algorithm requirements, the 3A statistical data of the unprocessed image can be replaced with the 3A statistical data of the processed image before and after, without frame-by-frame statistics or calculation, so as to reduce the pressure of ISP processing, but in order to improve the video image quality, the AP side does not Process the image to compensate. For example, during lens shading correction (Lens Shading Correction, LSC) processing, only the actual LSC processing is done for the interval frame, and the check table calculation based on the image is not performed, and the LSC processing for the interval frame is based on the check table The information can use the checklist information of the previous adjacent frame, so as to ensure the LSC processing effect of the image.

Step 404: Send the first image data and the 3A statistical data to an application processor, so that the application processor can use the 3A statistical data of the processed image to process the unprocessed image in the first image data image compensation to obtain second image data.

It should be noted that after the ISP performs frame-interval processing, if image compensation is not performed, the video quality will be affected. Therefore, when the AP side determines that the ISP is running in the first working mode, the AP side will perform corresponding compensation processing.

Exemplarily, in some embodiments, the first image data and the 3A statistical data are sent to the application processor, so that the application processor can analyze the current frame by using the 3A statistical data of the processed image of the previous frame. performing image compensation processing on the unprocessed image; or, sending the first image data and the 3A statistical data to the application processor for the application processor to process the image of the previous frame and the processed image of the next frame The weighted 3A statistical data is processed to obtain the weighted 3A statistical data; image compensation processing is performed on the unprocessed image of the current frame through the weighted 3A statistical data.

That is to say, when the AP side compensates the unprocessed image of the current frame, it can use the 3A statistical data of the processed image of the previous frame to compensate, or to improve the compensation accuracy, it can firstly combine the processed image of the previous frame with the post-processing image. The 3A statistics of a processed image are weighted and then compensated using the weighted 3A statistics.

Exemplarily, when performing weighting processing, the weight of the 3A statistical data of the processed image in the previous frame is higher than that in the subsequent frame.

Exemplarily, FIG. 5 is a schematic flowchart of frame-interval processing in the embodiment of the present application, as shown in FIG. 5 ,

The image data are: D1, D2, D3, D4, D5, D6, D7...

After the image data enters the ISP, all the data is first processed by the front end module (Front End, FE), and then processed by the back end module (Back End, BE) every frame. The data processed every frame is the gray block in Figure 5 (D1, D3, D5, D7), and the data (D2, D4, D6) that have not been processed by the BE module are directly sent to the AP side;

The 3A statistical data of the data (D1, D3, D5, D7) processed by the ISP every other frame is transmitted to the AP side through the Mobile Industry Processor Interface (MIPI), and the data is shared on the AP side, and the unprocessed images are shared adjacent to each other. The 3A statistical data of the frame, so that the ISP can process frame by frame, reduce the read and write operations of the internal memory of the ISP, and reduce the power consumption of the ISP.

For example, when shooting a video with a high frame rate and high resolution (for example, 4K video), the video preview does not require high image quality, and the AP side can use the 3A statistical data of the processed image of the previous frame to perform image compensation, reducing The amount of calculation saves ISP energy consumption. When saving videos, high image quality is required. To strike a balance between power consumption and image processing effects, the AP side can use weighted 3A statistical data for image compensation.

Based on the above-mentioned embodiments, the image processing method when the ISP works in the second working mode is illustrated below. FIG. 6 is a schematic diagram of the fourth process flow of the image processing method in the embodiment of the present application. As shown in FIG. In the second working mode, the image processing method specifically includes:

Step 601: Obtain the running status information of the image processor;

Step 602: Determine that the image processor is running in a second working mode according to the running state information;

Taking FIG. 3 as an example, when it is detected that the running state information of the ISP is between the threshold 2 and the threshold 3, it is determined that the best working mode of the ISP is the second working mode.

Step 603: Perform down-sampling processing on the image data, and perform frame-interval processing on the down-sampled image data to obtain the first image data and 3A statistical data of the processed image;

Here, the image data is the original image data collected by the image acquisition device. The image acquisition device includes a camera and an image sensor. For example, when the electronic device is a mobile phone, the camera may be a rear main camera, a rear wide-angle camera, a rear telephoto camera or a front camera.

Exemplarily, during the downsampling process, the sampling coefficient k may be 1 or 2.

Step 603: Send the first image data and the 3A statistical data to an application processor, so that the application processor can use the 3A statistical data of the processed image to process the unprocessed image in the first image data Image compensation, and performing up-sampling processing on the compensated image data to obtain second image data.

It should be noted that after the ISP performs down-sampling and frame-interval processing, if no up-sampling and image compensation are performed, the video quality will be affected. Therefore, when the AP side determines that the ISP is running in the second working mode, the AP side will perform corresponding compensation processing and upsampling. Compensation processing is used to improve image quality, and upsampling processing is used to restore the original image size.

Exemplarily, the upsampling processing may be implemented based on a bilinear interpolation method, or may be implemented by a super-resolution technology (Super-Resolution, SR).

Figure 7 is a schematic flow chart of downsampling and upsampling processing in the embodiment of the present application. As shown in Figure 7, the ISP side performs downsampling processing on the received image of height x width, and the downsampling coefficient is 1, and the image size is obtained It is height/2xwidth/2, that is, in the original image, one point is taken every other point in each row and column to form an image, and then the sampled image is processed every other frame to further reduce the calculation amount on the ISP side. ISP energy consumption. Also in order to ensure the image display effect, the image is first up-sampled on the AP side to restore the image size to the height x width, and then the image compensation process is performed.

That is to say, when the AP side compensates the unprocessed image of the current frame, it can use the 3A statistical data of the processed image of the previous frame to compensate, or to improve the compensation accuracy, it can firstly combine the processed image of the previous frame with the post-processing image. The 3A statistics of a processed image are weighted and then compensated using the weighted 3A statistics. For frame-interval processing and compensation processing, refer to FIG. 5 .

Based on the above-mentioned embodiments, the image processing method when the ISP works in the third working mode is described below with an example, and FIG. 8 is a schematic flowchart of a fifth image processing method in the embodiment of the present application.

Exemplarily, as shown in FIG. 8, the image processor includes a front-end module (FE) and a back-end module (BE), and the front-end module is used to perform front-end processing on the image data collected by the image sensor, and process the processed The data is stored in the memory; the back-end module is used to read the image data from the memory for back-end processing. The image processor runs in the third working mode, retains the function of the front-end module (FE), and sets the back-end module (BE) to a bypass mode (also called Bypass mode). The image data is sent to the AP side after being processed by the front-end module (FE) through the processing path 82 in FIG. 8 .

The image processor operates in a high-performance mode. The image data is processed by the front-end module (FE) through the processing path 83 in FIG. Back-end processing, and then send the processed image data to the AP side.

In some embodiments, the working mode further includes a fourth working mode, and when the image processor is running in the third working mode, the image processor is set to a bypass mode.

The fourth working mode is also a low power consumption mode, and the power consumption of the ISP running in the fourth working mode is less than that of running in the third working mode. Run the fourth working mode, set the image processor to bypass mode, taking Figure 8 as an example, setting the image processor to bypass mode includes setting both the front-end module (FE) and the back-end module (BE) to bypass mode, the image data is directly sent to the AP side through the processing path 81 in FIG. 8 .

Compared with the fourth working mode, in the third working mode, setting the back-end module (BE) as the bypass mode retains part of the processing capability of the ISP, and finally obtains a better image processing effect.

Exemplarily, when the ISP operates in the first working mode and the second working mode, the image data may be data processed by the FF module, and the back-end module (BE) performs down-sampling processing and frame-interval processing on the image data. The ISP runs in the third working mode, and the ISP directly sends the image data to the AP.

It should be noted that the ISP may also include other modules, which are not shown in FIG. 8 .

Using the above technical solution, the ISP calculation load can be reduced by sharing the intermediate processing results, thereby reducing power consumption. According to the algorithm requirements, the 3A statistical data of the unprocessed image can be replaced with the 3A statistical data of the processed image before and after, without the need Statistics or calculations are performed frame by frame to reduce the pressure of ISP processing, but in order to improve the quality of video images, the AP side compensates for unprocessed images.

In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an image processor, the image processor is configured to call and run the computer program stored in the memory, and execute the method in the embodiment of the present application The steps of the image processing method.

In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application also provides an image processing chip. As shown in FIG. 9, the image processing chip 90 includes:

an image processor 901a and an application processor 901b coupled to the image processor 901a, the image processor 901a is configured to process received image data, the image processor 901a has a first working mode and a second working mode,

Wherein, when the image processor 901a operates in the first working mode, the image processor 901a performs frame-interval processing on the image data to obtain the first image data;

When the image processor 901a is running in the second working mode, the image processor 901a performs down-sampling processing on the image data and then performs frame-interval processing to obtain the first image data;

The application processor 901b is configured to process the first image data.

In some embodiments, when the image processor 901a operates in the first working mode, the image data is processed every other frame to obtain the first image data and the 3A statistical data of the processed image; the first image data and the 3A statistical data are obtained The statistical data is sent to the application processor 901b, so that the application processor 901b performs image compensation on the unprocessed image in the first image data by using the 3A statistical data of the processed image to obtain the second image data.

In some embodiments, when the image processor 901a operates in the second working mode, the image data is down-sampled, and the down-sampled image data is processed every other frame to obtain the first image data and the 3A statistical data of the processed image ; Send the first image data and the 3A statistical data to the application processor 901b, so that the application processor 901b can perform an unprocessed image in the first image data through the 3A statistical data of the processed image Image compensation, and performing up-sampling processing on the compensated image data to obtain second image data.

In some embodiments, the image processor 901a is configured to send the first image data and the 3A statistical data to the application processor 901b, so that the application processor 901b can use the previous frame processed image 3A statistical data performs image compensation processing on the unprocessed image of the current frame;

Alternatively, the image processor 901a is configured to send the first image data and the 3A statistical data to the application processor 901b, so that the application processor 901b can analyze the processed image of the previous frame and the processed image of the next frame. Perform weighting processing on the 3A statistical data of the processed image to obtain weighted 3A statistical data; perform image compensation processing on the unprocessed image of the current frame through the weighted 3A statistical data.

In some embodiments, the image processor 901a operates in the third working mode, and some modules in the image processor are set to bypass mode.

In some embodiments, the image processor 901a determines the working mode according to the running state information, specifically, determines the target running condition satisfied by the running state information from the preset running conditions; the target running condition is When the first running condition is used, it is determined that the image processor is running in the first working mode; when the target running condition is a second running condition, it is determined that the image processor is running in the second working mode; the When the target operating condition is the third operating condition, it is determined that the image processor is operating in the third working mode.

The operating conditions include at least one of the following:

The temperature of the image processor is within a preset temperature range;

The power consumption of the image processor is within a preset power consumption range.

Optionally, as shown in FIG. 9 , the image processing chip 90 may further include a memory 902 . Wherein, the image processor 901a and the application processor 901b can call and run a computer program from the memory 902, so as to implement the method in the embodiment of the present application.

Wherein, the memory 902 may be a separate device independent of the image processor 901a and the application processor 901b, or may be integrated in the image processor 901a and the application processor 901b.

Optionally, the image processing chip 90 may also include an input interface 903 . Communicate with other devices or chips through the input interface 903 , specifically, information or data sent by other devices or chips can be obtained.

Optionally, the image processing chip 90 may also include an output interface 904 . Communicate with other devices or chips through the output interface 904 , specifically, output information or data to other devices or chips.

Optionally, the image processing chip 90 may be applied to the electronic device in the embodiment of the present application.

It should be understood that the chip mentioned in the embodiment of the present application may also be called a system-on-chip, a system-on-chip, a system-on-a-chip, or a system-on-a-chip.

In practical application, the above-mentioned processor can be application specific integrated circuit (ASIC, Application Specific Integrated Circuit), digital signal processing device (DSPD, Digital Signal Processing Device), programmable logic device (PLD, Programmable Logic Device), on-site At least one of a programmable gate array (Field-Programmable Gate Array, FPGA), a controller, a microcontroller, and a microprocessor. It can be understood that, for different devices, the electronic device used to implement the above processor function may also be other, which is not specifically limited in this embodiment of the present application.

Above-mentioned memory can be volatile memory (volatile memory), such as random access memory (RAM, Random-Access Memory); Or non-volatile memory (non-volatile memory), such as read-only memory (ROM, Read-Only Memory), flash memory (flash memory), hard disk (HDD, Hard Disk Drive) or solid-state drive (SSD, Solid-State Drive); or a combination of the above types of memory, and provide instructions and data to the processor.

The embodiment of the present application also provides an electronic device. The electronic device described in the present application has a shooting function, and the electronic device may include such as a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a personal digital assistant (Personal Digital Assistant, PDA), a portable Media player (Portable Media Player, PMP), wearable device, camera, smart car, etc.

As shown in Figure 10, the electronic device 100 includes:

An image acquisition device 1001 configured to acquire image data;

The image processor 1002 is configured to execute the method steps implemented by the image processor in the foregoing embodiments.

The application processor 1003 is configured to receive the first image data processed by the image processor and process the first image processing data.

Of course, in actual application, as shown in 10 , various components in the electronic device 100 are coupled together through the bus system 1004 . It can be understood that the bus system 1004 is used to realize connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus and a status signal bus. However, the various buses are labeled as bus system 1004 in FIG. 10 for clarity of illustration.

In an exemplary embodiment, the embodiment of the present application also provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by an image processor to complete the steps of the foregoing method.

The embodiment of the present application also provides a computer program product, including computer program instructions.

Optionally, the computer program product can be applied to the image processor in the embodiment of the present application, and the computer program instructions cause the computer to execute the corresponding process implemented by the image processor in each method of the embodiment of the present application. For brevity, the This will not be repeated here.

The embodiment of the present application also provides a computer program.

Optionally, the computer program may be applied to the image processor in the embodiment of the present application, and when the computer program is run on the computer, the computer executes the corresponding process implemented by the image processor in each method of the embodiment of the present application, For the sake of brevity, details are not repeated here.

It should be understood that the terminology used in the present application is for the purpose of describing particular embodiments only, and is not intended to limit the present application. As used in this application and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. The expressions "has", "may have", "comprises" and "comprises", or "may comprise" and "may comprise" in this application may be used herein to indicate the presence of corresponding features (for example, such as values, functions, elements such as operations or components), but does not exclude the presence of additional features.

It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, first information may also be called second information, and similarly, second information may also be called first information.

The technical solutions described in the embodiments of the present application may be combined arbitrarily if there is no conflict.

In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment may be implemented in other ways. The above-described embodiments are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into Another system, or some features may be ignored, or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms of.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be used as a single unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application.

Industrial Applicability

The present application provides an image processing method, an image processing chip and an electronic device. The method obtains the operation state information by monitoring the operation state of the image processor, and determines the best work matching the current operation state information for the image processor according to the operation state information. mode, when the image processor runs in the first working mode, the image data is processed every other frame, and when the image processor runs the second working mode, the image data is down-sampled and processed every other frame. The adaptive adjustment of the working mode of the image processor is realized, thereby optimizing the power consumption of the image processor, solving the problem of high power consumption in processing high frame rate images for a long time in a single working mode, and achieving a balance between the power consumption of the image processor and the image The purpose of processing effects.

Claims

An image processing method applied to an image processor, wherein the method includes:

Acquiring the running state information of the image processor;

Determining that the image processor is running in a first working mode or a second working mode according to the running status information, wherein when the image processor is running in the first working mode, image data is processed every other frame; When the image processor operates in the second working mode, it performs down-sampling processing and frame-interval processing on the image data.
The method according to claim 1, wherein when the image processor is running in the first working mode, performing frame-interval processing on the image data comprises:

Processing the image data at intervals to obtain the first image data and 3A statistical data of the processed image;

sending the first image data and the 3A statistical data to an application processor, so that the application processor can perform image compensation on an unprocessed image in the first image data by using the 3A statistical data of the processed image, Obtain the second image data.
The method according to claim 1, wherein, when the image processor operates in the second working mode, performing downsampling processing and frame-interval processing on the image data includes:

Carrying out down-sampling processing on the image data, and performing frame-interval processing on the down-sampled image data to obtain the first image data and 3A statistical data of the processed image;

sending the first image data and the 3A statistical data to an application processor, so that the application processor can perform image compensation on an unprocessed image in the first image data by using the 3A statistical data of the processed image, And performing up-sampling processing on the compensated image data to obtain second image data.
The method according to claim 2 or 3, wherein,

Sending the first image data and the 3A statistical data to an application processor, so that the application processor performs image compensation processing on the unprocessed image of the current frame through the 3A statistical data of the processed image of the previous frame;

Or, sending the first image data and the 3A statistical data to an application processor, so that the application processor can perform weighting processing on the 3A statistical data of the processed image of the previous frame and the processed image of the next frame, Obtaining weighted 3A statistical data; performing image compensation processing on the unprocessed image of the current frame through the weighted 3A statistical data.
The method according to claim 1, wherein the method further comprises:

According to the operation status information, it is determined that the image processor is running in a third working mode, wherein when the image processor is running in the third working mode, some modules in the image processor are set as bypass road mode.
The method according to claim 5, wherein the method further comprises:

From the preset operating conditions, determine the target operating conditions satisfied by the operating state information;

When the target operating condition is the first operating condition, it is determined that the image processor is operating in the first working mode;

When the target operating condition is a second operating condition, it is determined that the image processor is operating in the second working mode;

When the target operating condition is a third operating condition, it is determined that the image processor is operating in the third working mode.
The method according to claim 6, wherein the operating status information includes at least one of the following: temperature of the image processor, power consumption of the image processor during preset working hours,

The operating conditions include at least one of the following:

The temperature of the image processor is within a preset temperature range;

The power consumption of the image processor is within a preset power consumption range.
An image processor, wherein the image processor is configured to invoke and run a computer program stored in a memory, and execute the method steps implemented by the image processor according to any one of claims 1-7.
An image processing chip, including:

an image processor and an application processor coupled to the image processor, the image processor is configured to process received image data, the image processor has a first mode of operation and a second mode of operation,

Wherein, when the image processor operates in the first working mode, the image processor performs frame-interval processing on the image data to obtain the first image data;

When the image processor is running in the second working mode, the image processor performs down-sampling processing on the image data and then performs frame-interval processing to obtain the first image data;

The application processor is configured to process the first image data.
The image processing chip according to claim 9, wherein,

The image processor is configured to process the image data at intervals to obtain first image data and 3A statistical data of the processed image; and send the first image data and the 3A statistical data to an application processor ;

The application processor is configured to perform image compensation on the unprocessed image in the first image data by using the 3A statistical data of the processed image to obtain the second image data.
The image processing chip according to claim 9, wherein,

The image processor is configured to perform down-sampling processing on the image data, and perform frame-interval processing on the down-sampled image data to obtain the first image data and 3A statistical data of the processed image; the first image data and said 3A statistics are sent to the application processor;

The application processor is configured to perform image compensation on the unprocessed image in the first image data by using the 3A statistical data of the processed image, and perform up-sampling processing on the compensated image data to obtain the second image data.
The image processing chip according to claim 10 or 11, wherein,

The application processor is configured to perform image compensation processing on the unprocessed image of the current frame through the 3A statistical data of the processed image of the previous frame;

Alternatively, the application processor is configured to perform weighted processing on the 3A statistical data of the processed image of the previous frame and the processed image of the next frame to obtain weighted 3A statistical data; Perform image compensation processing.
An electronic device, wherein the electronic device includes:

an image acquisition device configured to acquire image data;

An image processor configured to execute the method steps implemented by the image processor according to any one of claims 1 to 7.
The electronic device of claim 13, wherein the electronic device further comprises an application processor,

An application processor configured to receive the first image data processed by the image processor and process the first image processing data.
A computer-readable storage medium configured to store a computer program, the computer program causing a computer to execute the steps of the method according to any one of claims 1-7.