CN113747145B - Image processing circuit, electronic apparatus, and image processing method - Google Patents
Image processing circuit, electronic apparatus, and image processing method Download PDFInfo
- Publication number
- CN113747145B CN113747145B CN202010478315.2A CN202010478315A CN113747145B CN 113747145 B CN113747145 B CN 113747145B CN 202010478315 A CN202010478315 A CN 202010478315A CN 113747145 B CN113747145 B CN 113747145B
- Authority
- CN
- China
- Prior art keywords
- image
- processing unit
- application processor
- image signal
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 232
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012805 post-processing Methods 0.000 claims description 60
- 238000007781 pre-processing Methods 0.000 claims description 38
- 238000013528 artificial neural network Methods 0.000 claims description 35
- 239000000284 extract Substances 0.000 claims description 8
- 230000003068 static effect Effects 0.000 claims description 5
- 238000004806 packaging method and process Methods 0.000 claims 2
- 238000000034 method Methods 0.000 description 17
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 14
- 238000005457 optimization Methods 0.000 description 13
- 238000012549 training Methods 0.000 description 12
- 230000005540 biological transmission Effects 0.000 description 7
- 238000012937 correction Methods 0.000 description 5
- 238000003384 imaging method Methods 0.000 description 4
- 238000011946 reduction process Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013501 data transformation Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011478 gradient descent method Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N17/00—Diagnosis, testing or measuring for television systems or their details
- H04N17/06—Diagnosis, testing or measuring for television systems or their details for recorders
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
The embodiment of the application provides an image processing circuit, electronic equipment and an image processing method, wherein an image sensor is connected with an image signal preprocessor and is used for acquiring an external image according to configured acquisition parameters and transmitting the external image to the image signal preprocessor; the image signal preprocessor is connected with the application processor and is used for transmitting indication information to the application processor when receiving the packet head of the first image data packet of the single frame image in the external image transmitted by the image sensor; the application processor is connected with the image sensor and is used for updating acquisition parameters of the image sensor according to the indication information. The image signal preprocessor is used for indicating the application processor to update the acquisition parameters of the image sensor, so that the application processor can stably control the image acquisition of the image sensor. When the image processing circuit is applied to the electronic equipment, the stability of the electronic equipment for image acquisition can be effectively improved.
Description
Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing circuit, an electronic device, and an image processing method.
Background
Currently, the quality of a shooting function has become a key for measuring the performance of an electronic device (such as a smart phone, a tablet computer, etc.). The quality of the shooting quality of the electronic equipment is closely related to the image processing capability of the electronic equipment, and whether the electronic equipment can perform image acquisition stably influences whether the electronic equipment can perform image processing normally and effectively.
Disclosure of Invention
The embodiment of the application provides an image processing circuit, electronic equipment and an image processing method, which can improve the stability of the electronic equipment for image acquisition.
The application discloses an image processing circuit, which comprises an image sensor, an image signal preprocessor and an application processor, wherein,
the image sensor is connected with the image signal preprocessor and is used for acquiring external images according to configured acquisition parameters and transmitting the external images to the image signal preprocessor;
the image signal preprocessor is connected with the application processor and is used for transmitting indication information to the application processor when receiving a packet head of a first image data packet of a single frame image in the external image transmitted by the image sensor;
the application processor is connected with the image sensor and used for updating acquisition parameters of the image sensor according to the indication information.
The application also discloses electronic equipment, which comprises the image processing circuit.
The embodiment of the application also discloses an image processing method, which is applicable to an image processing circuit, wherein the image processing circuit comprises an image sensor, an image signal preprocessor and an application processor, and comprises the following steps:
the image sensor acquires an external image according to the configured acquisition parameters and transmits the external image to the image signal preprocessor;
the image signal preprocessor transmits indication information to the application processor when receiving a packet head of a first image data packet of a single frame image in the external image transmitted by the image sensor;
and the application processor updates acquisition parameters of the image sensor according to the indication information.
The image signal preprocessor is used for indicating the application processor to update the acquisition parameters of the image sensor, so that the application processor can stably control the image acquisition of the image sensor. When the image processing circuit is applied to the electronic equipment, the stability of the electronic equipment for image acquisition can be effectively improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that are required to be used in the description of the embodiments will be briefly described below.
Fig. 1 is a schematic diagram of a first configuration of an image processing circuit according to an embodiment of the present application.
FIG. 2 is a state machine diagram of an RTL design according to an embodiment of the present application.
Fig. 3 is a schematic diagram of a second structure of an image processing circuit according to an embodiment of the present application.
Fig. 4 is a schematic diagram of a third structure of an image processing circuit according to an embodiment of the present application.
Fig. 5 is a schematic diagram of a fourth configuration of an image processing circuit according to an embodiment of the present application.
Fig. 6 is a schematic diagram of a fifth configuration of an image processing circuit according to an embodiment of the present application.
Fig. 7 is an exemplary diagram of an enhanced image obtained by enhancing a post-processing image in an embodiment of the present application.
Fig. 8 is a schematic diagram of a sixth configuration of an image processing circuit according to an embodiment of the present application.
Fig. 9 is a flowchart of an image processing method according to an embodiment of the present application.
Detailed Description
The technical scheme provided by the embodiment of the application can be applied to various scenes needing data communication, and the embodiment of the application is not limited to the scenes.
Referring to fig. 1, fig. 1 is a schematic diagram of a first structure of an image processing circuit 100 according to an embodiment of the present disclosure. The image processing circuit 100 may be integrated into an electronic device (e.g., a smart phone, tablet, palm top, notebook, or desktop computer configured with a processor and having processing capabilities), including an image sensor 110, an image signal pre-processor 120, and an application processor 130.
The image sensor 110 or photosensitive element is a device for converting an optical signal into an electrical signal, and compared with a photosensitive element of a "point" light source such as a photodiode or a phototransistor, the image sensor 110 divides an optical image sensed by the photosensitive element into a plurality of small units, and then converts the small units into a usable electrical signal to obtain original image data. It should be noted that, in the embodiment of the present application, the type of the image sensor 110 is not limited, and may be a Complementary Metal Oxide Semiconductor (CMOS) image sensor, a charge coupled device (Charge Coupled Device, CCD) image sensor, or the like.
The application processor 130 is integrated with an image signal processing unit, and can process the image data collected by the image sensor 110, so as to improve the quality of the image data. For example, the image signal processor 130 can perform a dead pixel correction process on the image data to eliminate dead pixels in the image data.
The image signal preprocessor 120 performs some differentiation processing before the image data is processed by the image signal processing unit integrated with the application processor 130, compared to the image signal processing unit integrated with the application processor 130.
In the embodiment of the present application, the image sensor 110 is connected to the image signal preprocessor 120, and is configured to collect external images according to configured collection parameters, and transmit the collected external images to the image signal preprocessor. For example, taking an exposure parameter as an example, in the case of automatic exposure, the application processor 130 calculates a corresponding exposure parameter according to the collected information about the automatic exposure, using a preconfigured automatic exposure control algorithm, and configures the exposure parameter to the image sensor 110, so that the image sensor 110 collects an external image according to the exposure parameter, and transmits the external image to the image signal preprocessor 120.
It should be noted that, in the embodiment of the present application, the connection manner between the image signal preprocessor 120 and the image sensor 110 is not specifically limited, for example, the image signal preprocessor 120 and the image sensor 110 are connected through MIPI (Mobile Industry Processor Interface ).
When the image sensor 110 transmits the external image to the image signal preprocessor 120, the external image is packaged into a plurality of image data packets for each frame, and the image data packets are transmitted to the image signal preprocessor. The image data packet includes a header field, a trailer field, and a data field, where the header field and the trailer field are used to fill some necessary control information, such as synchronization information, address information, error control information, and the like, and the data field is used to fill actual image content.
On the other hand, the image signal preprocessor 120 receives an external image from the image sensor 110. In addition, the image signal preprocessor 120 is further connected to the application processor 130, where in the embodiment of the present application, the connection manner between the application processor 130 and the image signal preprocessor 120 is not specifically limited, for example, the application processor 130 and the image signal preprocessor 120 may also be connected through MIPI.
The image signal preprocessor 120, when receiving the header of the first image data packet of the single frame image in the external image transmitted by the image sensor 110, transmits indication information to the application processor 130, where the indication information is used to instruct the application processor 130 to update the exposure parameters of the image sensor 110. It should be noted that, in the embodiment of the present application, the type of the indication information is not particularly limited, and may be selected by those skilled in the art according to actual needs.
For example, SOF is taken as the indication information.
The image signal preprocessor 120 is connected to the application processor 130 through MIPI, and in the process of designing MIPI by ASIC, the RTL design of the original SoF (Start of Frame) needs to be changed, and the state machine thereof is shown in fig. 2.
The application processor 130 is further connected to the image sensor 110, and is configured to update the acquisition parameters of the image sensor 110 according to the indication information from the image signal preprocessor 120, so that the image sensor 110 performs image acquisition according to the updated acquisition parameters, thereby stably acquiring a high-quality image.
As can be seen from the above, the image processing circuit provided by the present application includes an image sensor, an image signal preprocessor and an application processor, wherein the image sensor is connected with the image signal preprocessor, and is used for acquiring an external image according to configured acquisition parameters and transmitting the external image to the image signal preprocessor; the image signal preprocessor is connected with the application processor and is used for transmitting indication information to the application processor when receiving the packet head of the first image data packet of the single frame image in the external image transmitted by the image sensor; the application processor is connected with the image sensor and is used for updating acquisition parameters of the image sensor according to the indication information. The image signal preprocessor is used for indicating the application processor to update the acquisition parameters of the image sensor, so that the application processor can stably control the image acquisition of the image sensor. When the image processing circuit is applied to the electronic equipment, the stability of the electronic equipment for image acquisition can be effectively improved.
Optionally, in an embodiment, in order to improve the image quality, the image signal preprocessor 120 in the embodiment of the present application further preprocesses a single frame image in the external image from the image sensor 110, so as to obtain a preprocessed image correspondingly.
In order to avoid that the preprocessing of the external image by the image signal preprocessing 120 causes the original data of the external image to be destroyed, the image signal preprocessing 120 extracts the reference parameters of the single frame image before the preprocessing of the single frame image. The reference parameters include relevant parameters required by the application processor 130 for image processing, such as auto-focus parameters, auto-white balance parameters, auto-exposure parameters, and phase-focus parameters.
It should be noted that, in the embodiment of the present application, the preprocessing performed by the image signal preprocessor 120 is not limited, for example, when the electronic device is located in an extremely dark environment with an ambient illuminance lower than 1lux, the image content of the finally obtained imaging image is difficult to be identified, and by using the image signal preprocessor 120 provided in the present application, the original external image collected by the image sensor 110 can be optimized, so that the image content of the finally obtained imaging image can be clearly identified, and the purpose of improving the image quality is achieved.
The application processor 130 is connected to the image signal preprocessor 120, and is configured to further post-process the preprocessed image obtained by the image signal preprocessor 120, and accordingly obtain a post-processed image. It should be noted that, in the embodiment of the present application, the connection manner between the application processor 130 and the image signal preprocessor 120 is not limited, for example, the application processor 130 and the image signal preprocessor 120 may also be connected through MIPI. In addition, in the embodiment of the present application, post-processing performed by the application processor 130 is not limited, and may be configured by a person of ordinary skill in the art according to actual needs, or may be configured by default by the application processor 130 when leaving the factory.
In addition, the application processor 130 may preview and/or store the post-processed image directly after processing the post-processed image, or may preview and/or store the post-processed image after further processing.
Furthermore, the application processor 130 controls the image sensor 110 to start capturing an external image and to end capturing an external image based on its connection with the image sensor 110.
Optionally, in an embodiment, the acquisition parameters include exposure parameters, after the application processor 130 completes the current update of the exposure parameters of the image sensor 110, the image sensor 110 will acquire an external image by using the updated exposure parameters, and accordingly, the application processor 130 collects exposure statistics information required for automatic exposure control, and calculates new exposure parameters by using a configured automatic exposure control algorithm, so as to update the exposure parameters of the image sensor 110 again when receiving the indication information of the image signal preprocessor 120 again. In this manner, the exposure parameters of the image sensor 110 are continuously updated until the exposure parameters converge.
Compared with the related art, the image signal preprocessor 120 is additionally added, and after the image signal preprocessor 120 preprocesses the original image acquired by the image sensor 110, the application processor 130 postprocesses the preprocessed image, so that the image signal preprocessor 120 cooperates with the application processor 130 to process the image. In addition, the application instructs the application processor 130 to update the acquisition parameters of the image sensor 110 by using the image signal preprocessor 120, so that the application processor 130 can stably control the image acquisition of the image sensor 110. When the image processing circuit is applied to the electronic equipment, the image processing capacity of the electronic equipment can be effectively improved.
Alternatively, in one embodiment, the image signal preprocessor 120 and the application processor 130 are directly connected through a GPIO (General purpose input/output) interface.
In this embodiment, the image signal preprocessor 120 is directly connected to the application processor 130 through a GPIO interface, and accordingly, when sending the instruction information to the application processor 130, the second image signal processing unit 1302 sends the instruction information to the application processor 130 through the GPIO interface between itself and the application processor 130.
Referring to fig. 3, fig. 3 is a schematic diagram of a second structure of the image processing circuit 100 according to the embodiment of the present application. The image processing circuit 100 provided in the embodiment of the present application may include an image sensor 110, an image signal preprocessor 120, an application processor 130 and a register 140, where the image sensor 110, the image signal preprocessor 120 and the application processor 130 may refer to the image sensor 110, the image signal preprocessor 120 and the application processor 130 in fig. 1, and are not described herein. Wherein the register 140 is connected to the image signal preprocessor 120 and the application processor 130.
In this embodiment, the indication information of the image signal preprocessor 120 is not directly transmitted to the application processor 130, but is indirectly transmitted by using the third party device "register 140", and accordingly, when the indication information is sent to the application processor 130, the image signal preprocessor 120 is configured to:
the preset flag bit of the register 140 is written as a preset value corresponding to the indication information, so that the application processor 130 obtains the indication information according to the preset value written by the preset flag bit of the register 140, and correspondingly updates the exposure parameter of the image sensor 110.
In this embodiment, which flag bit of the register 140 is selected as the preset flag bit to indicate the transmission of the information is not particularly limited, and may be selected by those skilled in the art according to actual needs. It should be noted that, for the preset flag bit in the register 140, a corresponding relationship between the preset value and the indication information is pre-established in the present application. For example, for the preset flag bit of the register 140, in this embodiment, "1" is used to characterize the foregoing indication information, and accordingly, when the foregoing indication information needs to be sent to the application processor 130, the image signal preprocessor 120 writes the value of the preset flag bit in the register 140 to "1" so that the register 140 sends an interrupt signal to the application processor 130 to inform the application processor 130 that there is currently data writing, and the application processor 130 correspondingly reads the value of the preset flag bit in the register 140, and then "1" is used to obtain the foregoing indication information, so as to update the exposure parameter of the image sensor 110.
It should be noted that, in the embodiment of the present application, a person skilled in the art may select a register type according to actual needs, for example, in the embodiment of the present application, transmission of the indication information is implemented by using PCIE registers.
Optionally, in an embodiment, the image sensor 110 is configured to package each line of image data of the single frame image into an image data packet, and sequentially transmit each image data packet to the image signal preprocessor 120.
It should be noted that, in the embodiment of the present application, for a complete external image, the image sensor 110 does not directly transmit a complete external image to the image signal pre-processor 120, but transmits it to the image signal pre-processor 120 line by line in units of lines.
The image sensor 110 is configured to package each line of image data of the single frame image into an image data packet, and sequentially transmit the packaged image data packet to the image signal preprocessor 120.
Optionally, in an embodiment, the image sensor 110 is configured to, when acquiring image data of a line of external images, encapsulate the acquired line of image data into an image data packet, and transmit the encapsulated image data packet to the image signal preprocessor 120.
It should be noted that, in the embodiment of the present application, the image sensor 110 does not transmit the complete external image to the image signal pre-processor 120 line by line after a frame of the complete external image is acquired, but each time image data of a line of the external image is acquired, that is, one line of the acquired image data is transmitted to the image signal pre-processor 120.
For example, after the image sensor 110 acquires the first line of image data of the single frame image, the image data of the first line is packaged into an image data packet and transmitted to the image signal pre-processor 120, and at the same time, the image data of the next line of the single frame image is continuously acquired and transmitted again, and the process is circulated until the image data of the last line of the single frame image is acquired, and the image data of the last line is packaged into an image data packet and transmitted to the second image signal processing unit 1302, so as to complete the transmission of the single frame image.
Referring to fig. 4, fig. 4 is a schematic diagram illustrating a third configuration of an image processing circuit 100 according to an embodiment of the present application. The image processing circuit 100 includes an image sensor 110, an image signal pre-processor 120, and an application processor 130. Wherein the image signal pre-processor 120 comprises a first image signal processing unit 1201 and a neural network processing unit 1202. In addition, the image sensor 110 and the application processor 130 may refer to the image sensor 110 and the application processor 130 in fig. 1, respectively, and will not be described herein.
In this embodiment of the present application, the first image signal processing unit 1201 is configured to extract the reference parameters of the single frame image, and transmit the extracted reference parameters to the application processor 130, and in addition, perform optimization processing on the single frame image to obtain an optimized external image. After extracting the reference parameters of the single frame image, the first image signal processing unit 1201 further performs optimization processing on the single frame image according to a pre-configured optimization policy, so as to obtain an optimized single frame image. It should be noted that, in the embodiment of the present application, the optimization performed by the first image signal processing unit 1201 is not specifically limited, and may include at least one of a dead point correction process, a time domain noise reduction process, a 3D noise reduction process, a linearization process, and a black level correction process, and of course, may also include an optimization process manner not listed in the present application.
The neural network processing unit 1202 is configured to perform preprocessing on the optimized single-frame image to obtain a preprocessed image. The neural network processing unit 1202 is configured to cure a plurality of neural network algorithms (for example, a video night scene algorithm, a video HDR algorithm, a video blurring algorithm, a video noise reduction algorithm, a video super-resolution algorithm, etc. based on the neural network), after the first image signal processing unit 1201 completes the optimization processing on the single frame image, the neural network processing unit 1202 invokes a corresponding neural network algorithm to further perform preprocessing on the optimized single frame image according to a configured preprocessing strategy, so as to obtain a preprocessed image.
After the preprocessing of the optimized single frame image is completed to obtain a preprocessed image, the neural network processing unit 1202 also transmits the preprocessed image to the application processor 130.
For example, the neural network processing unit 1202 may transmit the pre-processed image to the first image signal processing unit 1201, and the first image signal processing unit 1201 transmits the extracted reference parameter to the application processor 130 together with the pre-processed image processed by the neural network processing unit 1202.
Optionally, the first image signal processing unit 1201 is further configured to transmit the extracted reference parameters to the application processor 130 when the reference parameters are extracted.
Referring to fig. 5, fig. 5 is a schematic diagram of a fourth configuration of an image processing circuit 100 according to an embodiment of the present application. The image processing circuit 100 includes an image sensor 110, an image signal pre-processor 120, and an application processor 130. Wherein the image signal pre-processor 120 comprises a first image signal processing unit 1201 and a neural network processing unit 1202. In addition, the image signal pre-processor 120 further includes a central processing unit 1203, and the image sensor 110, the application processor 130, the first image signal processing unit 1201 and the neural network processing unit 1202 may refer to the image sensor 110, the application processor 130, the first image signal processing unit 1201 and the neural network processing unit 1202 in fig. 4, which are not described herein.
In this embodiment, the first image signal processing unit 1201 is further configured to send the foregoing indication information to the central control processing unit 1203 synchronously when transmitting the indication information to the application processor 130.
On the other hand, the central processing unit 1203 is configured to control preprocessing of the neural network processing unit 1202 according to the foregoing instruction information. The control of the central processing unit 1203 to the neural network processing unit includes, but is not limited to, timing and type of preprocessing, and the like.
Referring to fig. 6, fig. 6 is a schematic diagram of a fifth configuration of an image processing circuit 100 according to an embodiment of the present application. The image processing circuit 100 includes an image sensor 110, an image signal pre-processor 120, and an application processor 130. Wherein the image signal pre-processor 120 includes a first image signal processing unit 1201 and a neural network processing unit 1202, and the application processor 130 includes an application processing unit 1301 and a second image signal processing unit 1302.
The second image signal processing unit 1302 is connected to the first image signal processing unit 1201, and is configured to obtain a reference parameter and a pre-processed image transmitted by the first image signal processing unit 1201, and perform post-processing on the pre-processed image according to the reference parameter to obtain a post-processed image;
The application processing unit 1301 is configured to:
when the post-processing image is a dynamic image, previewing the post-processing image or performing video coding according to the post-processing image; or,
when the post-processing image is a still image, image encoding is performed based on the post-processing image.
The post-processing performed by the second image signal processing unit 1302 in the embodiment of the present application is not particularly limited, and may be configured by one of ordinary skill in the art according to actual needs.
For example, the second image signal processing unit 1302 may perform image enhancement processing on the pre-processed image. The image enhancement means that data transformation is carried out on image data to be enhanced, and interested features in the image are selectively highlighted and some unwanted features in the image are restrained, so that the visual effect of the enhanced image quality is improved. Supervised learning is a machine learning task that infers model parameters from a labeled training dataset. In supervised learning, sample pairs include input objects and desired outputs. In the embodiment of the application, the input object is a training image in a sample pair, and the expected output is a paired target image. The initial image enhancement model is one that requires further model training to adjust model parameters. The type of the initial image enhancement model can be set according to the needs, and can be a deep convolutional neural network model or a residual convolutional network model, for example. The goal of model training is to obtain better model parameters to improve the image enhancement effect. When training is carried out, a training image is input into an initial image enhancement model to obtain an output model enhancement image, then model parameters are adjusted according to the difference between the model enhancement image and a matched target image, so that the model enhancement image obtained by image enhancement according to the adjusted model parameters is more and more close to the target image, for example, the model parameters are adjusted by adopting a gradient descent method towards the direction of descending the loss value corresponding to the model until convergence, and an image enhancement model is obtained.
In this embodiment, after the second image signal processing unit 1302 acquires the pre-processing image, that is, the reference parameter, the pre-trained image enhancement model is called to perform the image enhancement processing on the pre-processing image, so as to obtain the enhanced image (that is, the post-processing image).
For example, referring to fig. 7, the left side of fig. 7 shows a pre-processed image obtained by the second image signal processing unit 1302, the second image signal processing unit 1302 invokes a pre-trained image enhancement model to enhance the brightness and sharpness of the pre-processed image, so as to obtain an enhanced image (i.e. a post-processed image) after enhancement shown on the right side, and it can be seen that the brightness and sharpness of the obtained enhanced image after image enhancement processing are significantly improved compared with those of the pre-processed image.
It should be noted that the image type does not change with the processing of the external image, that is, the original external image is a still image, the post-processing image obtained by the corresponding processing is also a still image, the original external image is a moving image, and the post-processing image obtained by the corresponding processing is also a moving image. The static image is a single-frame image shot in real time, the dynamic image is a frame image in an image sequence acquired during preview, and the dynamic image is a frame image in an image sequence acquired during video recording.
For example, when the image processing circuit 100 provided in the present application is applied to a photographing scene, the external image collected by the image sensor 110 is a frame of still image;
when the image processing circuit 100 provided by the application is applied to a preview scene, an external image acquired by the image sensor 110 is a frame of dynamic image in a preview image sequence;
when the image processing circuit 100 provided by the application is applied to video scenes, the external image acquired by the image sensor 110 is a frame of dynamic image in a video image sequence.
In this embodiment of the present application, after the second image signal processing unit 1302 completes post-processing on the pre-processed image and obtains the post-processed image accordingly, the application processing unit 1301 is configured to obtain the foregoing post-processed image from the second image signal processing unit 1302, and perform a corresponding operation according to the type of the post-processed image. It should be noted that, in the embodiment of the present application, the type of the pre-processed image is not changed by post-processing the pre-processed image, for example, the pre-processed image is a dynamic image, and the post-processed image obtained by corresponding post-processing is also a dynamic image.
When the post-processing image is a moving image, the application processing unit 1301 may preview the aforementioned post-processing image or perform video encoding according to the aforementioned post-processing image. For example, when the current processed image is a dynamic image of a frame in the preview image sequence, after the corresponding post-processing obtains a post-processed image, the application processing unit 1301 previews the post-processed image; when the current processed image is a dynamic image of a frame in the video image sequence, after the corresponding processing obtains a post-processed image, the application processing unit 1301 performs video encoding according to the post-processed image.
When the post-processing image is a still image, the application processing unit 1301 performs image encoding according to the aforementioned post-processing image, resulting in an encoded image. For example, the pre-processing image is a RAW format image, the post-processing image obtained by the corresponding post-processing is also a RAW format image, and the application processing unit 1301 may perform JPEG image encoding according to the foregoing post-processing image to obtain an encoded image in JPEG format.
Referring to fig. 8, fig. 8 is a schematic diagram of a sixth configuration of an image processing circuit 100 according to an embodiment of the present application. The difference from the image processing circuit 100 shown in fig. 6 is that the image signal preprocessor 120 further includes a memory 1204 in the embodiment of the present application.
It should be noted that, in the embodiment of the present application, the type of the memory 1204 is not limited, and may be selected by those skilled in the art according to actual needs, and may be a static memory or a dynamic memory. For example, in the embodiment of the application, the memory 1204 is a DDR (Double Data Rate) type dynamic memory.
In the embodiment of the present application, the first image signal processing unit 1201 provides the pre-processed image and the reference parameter to the second image signal processing unit 1302 in different transmission modes according to the image type of the pre-processed image.
When the currently processed image is a moving image, the first image signal processing unit 1201 is configured to directly transmit the reference parameter and the pre-processed image to the second image signal processing unit 1302. Since the moving image needs to consider the real-time nature of the processing, the pre-processed image and the aforementioned reference parameters can be directly transmitted to the second image signal processing unit 1302 by the first image signal processing unit 1201 based on a direct connection (such as MIPI connection) between it and the second image signal processing unit 1302.
When the currently processed image is a still image, the first image signal processing unit 1201 is configured to write the reference parameter and the previously processed image into the memory 1204, and the second image signal processing unit 1302 is configured to extract the previously processed image and the reference parameter from the memory 1204.
The present application also provides an electronic device for processing image data, the electronic device comprising the image processing circuit provided by the present application.
The present application further provides an image processing method, referring to fig. 9, the flow of the image processing method may be:
in 201, the image sensor collects an external image according to the configured collection parameters and transmits the external image to the image signal preprocessor;
In 202, the image signal preprocessor transmits indication information to the application processor when receiving a packet header of a first image data packet of an external image transmitted by the image sensor;
in 203, the application processor updates acquisition parameters of the image sensor according to the indication information.
Taking the image processing method provided by the application as an example, the image processing circuit shown in fig. 1 is suitable for use.
In the embodiment of the present application, the image sensor 110 is connected to the image signal preprocessor 120, and is configured to collect external images according to configured collection parameters, and transmit the collected external images to the image signal preprocessor. For example, taking an exposure parameter as an example, in the case of automatic exposure, the application processor 130 calculates a corresponding exposure parameter according to the collected information about the automatic exposure, using a preconfigured automatic exposure control algorithm, and configures the exposure parameter to the image sensor 110, so that the image sensor 110 collects an external image according to the exposure parameter, and transmits the external image to the image signal preprocessor 120.
It should be noted that, in the embodiment of the present application, the connection manner between the image signal preprocessor 120 and the image sensor 110 is not specifically limited, for example, the image signal preprocessor 120 and the image sensor 110 are connected through MIPI (Mobile Industry Processor Interface ).
When the image sensor 110 transmits the external image to the image signal preprocessor 120, the external image is packaged into a plurality of image data packets for each frame, and the image data packets are transmitted to the image signal preprocessor. The image data packet includes a header field, a trailer field, and a data field, where the header field and the trailer field are used to fill some necessary control information, such as synchronization information, address information, error control information, and the like, and the data field is used to fill actual image content.
On the other hand, the image signal preprocessor 120 receives an external image from the image sensor 110. In addition, the image signal preprocessor 120 is further connected to the application processor 130, where in the embodiment of the present application, the connection manner between the application processor 130 and the image signal preprocessor 120 is not specifically limited, for example, the application processor 130 and the image signal preprocessor 120 may also be connected through MIPI.
The image signal preprocessor 120, when receiving the header of the first image data packet of the single frame image in the external image transmitted by the image sensor 110, transmits indication information to the application processor 130, where the indication information is used to instruct the application processor 130 to update the exposure parameters of the image sensor 110. It should be noted that, in the embodiment of the present application, the type of the indication information is not particularly limited, and may be selected by those skilled in the art according to actual needs.
For example, SOF is taken as the indication information.
The image signal preprocessor 120 is connected to the application processor 130 through MIPI, and in the process of designing MIPI by ASIC, the RTL design of the original SoF (Start of Frame) needs to be changed, and the state machine thereof is shown in fig. 2.
The application processor 130 is further connected to the image sensor 110, and is configured to update the acquisition parameters of the image sensor 110 according to the indication information from the image signal preprocessor 120, so that the image sensor 110 performs image acquisition according to the updated acquisition parameters, thereby stably acquiring a high-quality image.
Optionally, in an embodiment, the image processing method provided in the present application further includes:
the image signal preprocessor 120 extracts reference parameters of the single frame image and preprocesses the single frame image to obtain a preprocessed image; and
transmitting the reference parameters and the pre-processed image to the application processor 130;
the application processor 130 performs post-processing on the pre-processed image according to the reference parameters to obtain a post-processed image.
In order to improve the image quality, the image signal preprocessor 120 in the embodiment of the present application also preprocesses the external image from the image sensor 110, so as to obtain a preprocessed image.
In order to avoid that the preprocessing of the external image by the image signal preprocessing 120 causes the original data of the external image to be destroyed, the image signal preprocessing 120 also extracts the reference parameters of the external image before the external image is preprocessed. The reference parameters include relevant parameters required by the application processor 130 for image processing, such as auto-focus parameters, auto-white balance parameters, auto-exposure parameters, and phase-focus parameters.
It should be noted that, in the embodiment of the present application, the preprocessing performed by the image signal preprocessor 120 is not limited, for example, when the electronic device is located in an extremely dark environment with an ambient illuminance lower than 1lux, the image content of the finally obtained imaging image is difficult to be identified, and by using the image signal preprocessor 120 provided in the present application, the original external image collected by the image sensor 110 can be optimized, so that the image content of the finally obtained imaging image can be clearly identified, and the purpose of improving the image quality is achieved.
The application processor 130 is connected to the image signal preprocessor 120, and is configured to further post-process the preprocessed image obtained by the image signal preprocessor 120, and accordingly obtain a post-processed image. It should be noted that, in the embodiment of the present application, the connection manner between the application processor 130 and the image signal preprocessor 120 is not limited, for example, the application processor 130 and the image signal preprocessor 120 may also be connected through MIPI. In addition, in the embodiment of the present application, post-processing performed by the application processor 130 is not limited, and may be configured by a person of ordinary skill in the art according to actual needs, or may be configured by default by the application processor 130 when leaving the factory.
In addition, the application processor 130 may preview and/or store the post-processed image directly after processing the post-processed image, or may preview and/or store the post-processed image after further processing.
Furthermore, the application processor 130 controls the image sensor 110 to start capturing an external image and to end capturing an external image based on its connection with the image sensor 110.
Optionally, in an embodiment, the acquisition parameters include exposure parameters, after the application processor 130 completes the current update of the exposure parameters of the image sensor 110, the image sensor 110 will acquire an external image by using the updated exposure parameters, and accordingly, the application processor 130 collects exposure statistics information required for automatic exposure control, and calculates new exposure parameters by using a configured automatic exposure control algorithm, so as to update the exposure parameters of the image sensor 110 again when receiving the indication information of the image signal preprocessor 120 again. In this manner, the exposure parameters of the image sensor 110 are continuously updated until the exposure parameters converge.
Optionally, referring to fig. 3, the image processing circuit 100 further includes a register 140, where the register 140 is connected to the image signal preprocessor 120 and the application processor 130, and transmits instruction information to the application processor, and includes:
The image signal preprocessor 120 writes a preset flag bit of the register 140 as a preset value corresponding to the indication information, so that the application processor 130 acquires the indication information according to the preset value.
In this embodiment, the indication information of the image signal preprocessor 120 is not directly transmitted to the application processor 130, but is indirectly transmitted by using the third party device "register 140", and accordingly, when the indication information is sent to the application processor 130, the image signal preprocessor 120 is configured to:
the preset flag bit of the register 140 is written as a preset value corresponding to the indication information, so that the application processor 130 obtains the indication information according to the preset value written by the preset flag bit of the register 140, and correspondingly updates the exposure parameter of the image sensor 110.
In this embodiment of the present application, which flag bit of the PCIE register is selected as the preset flag bit to indicate the transfer of information is not specifically limited, and may be selected by a person of ordinary skill in the art according to actual needs. It should be noted that, for the preset flag bit in the register 140, a corresponding relationship between the preset value and the indication information is pre-established in the present application. For example, for the preset flag bit of the register 140, in this embodiment, "1" is used to characterize the foregoing indication information, and accordingly, when the foregoing indication information needs to be sent to the application processor 130, the image signal preprocessor 120 writes the value of the preset flag bit in the register 140 to "1" so that the register 140 sends an interrupt signal to the application processor 130 to inform the application processor 130 that there is currently data writing, and the application processor 130 correspondingly reads the value of the preset flag bit in the register 140, and then "1" is used to obtain the foregoing indication information, so as to update the exposure parameter of the image sensor 110.
It should be noted that, in the embodiment of the present application, a person skilled in the art may select a register type according to actual needs, for example, in the embodiment of the present application, transmission of the indication information is implemented by using PCIE registers.
Optionally, in an embodiment, transmitting the external image to the image signal preprocessor 120 includes:
the image sensor 110 encapsulates each line of image data of the single frame image into one image data packet, and sequentially transmits each image data packet to the image signal preprocessor 120.
It should be noted that, in the embodiment of the present application, for a complete external image, the image sensor 110 does not directly transmit a complete external image to the image signal pre-processor 120, but transmits it to the image signal pre-processor 120 line by line in units of lines.
The image sensor 110 is configured to package each line of image data of the single frame image into an image data packet, and sequentially transmit the packaged image data packet to the image signal preprocessor 120.
Optionally, in an embodiment, encapsulating each line of image data of the aforementioned image into one image data packet, and sequentially transmitting each image data packet to the image signal preprocessor 120 includes:
When the image sensor 110 acquires one line of image data of the single frame image, the acquired one line of image data is encapsulated into an image data packet, and the encapsulated image data packet is transmitted to the image signal preprocessor 120.
It should be noted that, in the embodiment of the present application, the image sensor 110 does not transmit the complete external image to the image signal pre-processor 120 line by line after a frame of the complete external image is acquired, but each time image data of a line of the external image is acquired, that is, one line of the acquired image data is transmitted to the image signal pre-processor 120.
For example, after the image sensor 110 acquires the first line of image data of the single frame image, the image data of the first line is packaged into an image data packet and transmitted to the image signal pre-processor 120, and at the same time, the image data of the next line of the single frame image is continuously acquired and transmitted again, and the process is circulated until the image data of the last line of the single frame image is acquired, and the image data of the last line is packaged into an image data packet and transmitted to the second image signal processing unit 1302, so as to complete the transmission of the single frame image.
Referring to fig. 4, the image signal preprocessor 120 includes a first image signal processing unit 1201 and a neural network processing unit 1202, extracts reference parameters of the external image, and preprocesses the external image to obtain a preprocessed image; and transmitting the reference parameters and the pre-processed image to the application processor comprises:
the first image signal processing unit 1201 extracts reference parameters of the external image, and transmits the extracted reference parameters to the application processor 130; performing optimization processing on the external image to obtain an optimized external image;
the neural network processing unit is used for preprocessing the optimized external image to obtain a preprocessed image; and transmitting the pre-processed image to the application processor 130.
In this embodiment of the present application, the first image signal processing unit 1201 is configured to extract the reference parameters of the single frame image, and transmit the extracted reference parameters to the application processor 130, and in addition, perform optimization processing on the single frame image to obtain an optimized external image. After extracting the reference parameters of the single frame image, the first image signal processing unit 1201 further performs optimization processing on the single frame image according to a pre-configured optimization policy, so as to obtain an optimized single frame image. It should be noted that, in the embodiment of the present application, the optimization performed by the first image signal processing unit 1201 is not specifically limited, and may include at least one of a dead point correction process, a time domain noise reduction process, a 3D noise reduction process, a linearization process, and a black level correction process, and of course, may also include an optimization process manner not listed in the present application.
The neural network processing unit 1202 is configured to perform preprocessing on the optimized single-frame image to obtain a preprocessed image. The neural network processing unit 1202 is configured to cure a plurality of neural network algorithms (for example, a video night scene algorithm, a video HDR algorithm, a video blurring algorithm, a video noise reduction algorithm, a video super-resolution algorithm, etc. based on the neural network), after the first image signal processing unit 1201 completes the optimization processing on the single frame image, the neural network processing unit 1202 invokes a corresponding neural network algorithm to further perform preprocessing on the optimized single frame image according to a configured preprocessing strategy, so as to obtain a preprocessed image.
After the preprocessing of the optimized single frame image is completed to obtain a preprocessed image, the neural network processing unit 1202 also transmits the preprocessed image to the application processor 130.
For example, the neural network processing unit 1202 may transmit the pre-processed image to the first image signal processing unit 1201, and the first image signal processing unit 1201 transmits the extracted reference parameter to the application processor 130 together with the pre-processed image processed by the neural network processing unit 1202.
Optionally, the first image signal processing unit 1201 is further configured to transmit the extracted reference parameters to the application processor 130 when the reference parameters are extracted.
Referring to fig. 5, the image signal preprocessor 120 further includes a central processing unit 1203, and the image processing method provided in the present application further includes:
the first image signal processing unit 1201 synchronously transmits the instruction information to the central control processing unit 1203 when transmitting the instruction information to the application processor 130;
the central processing unit 1203 controls preprocessing of the neural network processing unit 1202 according to the instruction information.
In this embodiment, the first image signal processing unit 1201 is further configured to send the foregoing indication information to the central control processing unit 1203 synchronously when transmitting the indication information to the application processor 130.
On the other hand, the central processing unit 1203 is configured to control preprocessing of the neural network processing unit 1202 according to the foregoing instruction information. The control of the central processing unit 1203 to the neural network processing unit includes, but is not limited to, timing and type of preprocessing, and the like.
Referring to fig. 6, the application processor 130 includes an application processing unit 1301 and a second image signal processing unit 1302, and performs post-processing on a pre-processed image according to a reference parameter to obtain a post-processed image, including:
the second image signal processing unit 1302 is configured to obtain the reference parameter and the pre-processed image transmitted by the first image signal processing unit 1201, and perform post-processing on the pre-processed image according to the reference parameter to obtain a post-processed image;
The image processing method provided by the application further comprises the following steps:
when the post-processing image is a moving image, the application processing unit 1301 previews the post-processing image or performs video encoding according to the post-processing image; or,
when the post-processing image is a still image, the application processing unit 1301 performs image encoding according to the post-processing image.
The post-processing performed by the second image signal processing unit 1302 in the application embodiment is not particularly limited, and may be configured by one of ordinary skill in the art according to actual needs.
For example, the second image signal processing unit 1302 may perform image enhancement processing on the pre-processed image. The image enhancement means that data transformation is carried out on image data to be enhanced, and interested features in the image are selectively highlighted and some unwanted features in the image are restrained, so that the visual effect of the enhanced image quality is improved. Supervised learning is a machine learning task that infers model parameters from a labeled training dataset. In supervised learning, sample pairs include input objects and desired outputs. In the embodiment of the application, the input object is a training image in a sample pair, and the expected output is a paired target image. The initial image enhancement model is one that requires further model training to adjust model parameters. The type of the initial image enhancement model can be set according to the needs, and can be a deep convolutional neural network model or a residual convolutional network model, for example. The goal of model training is to obtain better model parameters to improve the image enhancement effect. When training is carried out, a training image is input into an initial image enhancement model to obtain an output model enhancement image, then model parameters are adjusted according to the difference between the model enhancement image and a matched target image, so that the model enhancement image obtained by image enhancement according to the adjusted model parameters is more and more close to the target image, for example, the model parameters are adjusted by adopting a gradient descent method towards the direction of descending the loss value corresponding to the model until convergence, and an image enhancement model is obtained.
In this embodiment, after the second image signal processing unit 1302 acquires the pre-processing image, that is, the reference parameter, the pre-trained image enhancement model is called to perform the image enhancement processing on the pre-processing image, so as to obtain the enhanced image (that is, the post-processing image).
For example, referring to fig. 7, the left side of fig. 7 shows a pre-processed image obtained by the second image signal processing unit 1302, the second image signal processing unit 1302 invokes a pre-trained image enhancement model to enhance the brightness and sharpness of the pre-processed image, so as to obtain an enhanced image (i.e. a post-processed image) after enhancement shown on the right side, and it can be seen that the brightness and sharpness of the obtained enhanced image after image enhancement processing are significantly improved compared with those of the pre-processed image.
It should be noted that the image type does not change with the processing of the external image, that is, the original external image is a still image, the post-processing image obtained by the corresponding processing is also a still image, the original external image is a moving image, and the post-processing image obtained by the corresponding processing is also a moving image. The static image is a single-frame image shot in real time, the dynamic image is a frame image in an image sequence acquired during preview, and the dynamic image is a frame image in an image sequence acquired during video recording.
For example, when a scene is photographed, the external image collected by the image sensor 110 is a frame of still image;
when a scene is previewed, the external image collected by the image sensor 110 is a frame of dynamic image in the preview image sequence;
when a scene is recorded, an external image acquired by the image sensor 110 is a frame of dynamic image in a video image sequence.
In this embodiment of the present application, after the second image signal processing unit 1302 completes post-processing on the pre-processed image and obtains the post-processed image accordingly, the application processing unit 1301 is configured to obtain the foregoing post-processed image from the second image signal processing unit 1302, and perform a corresponding operation according to the type of the post-processed image. It should be noted that, in the embodiment of the present application, the type of the pre-processed image is not changed by post-processing the pre-processed image, for example, the pre-processed image is a dynamic image, and the post-processed image obtained by corresponding post-processing is also a dynamic image.
When the post-processing image is a moving image, the application processing unit 1301 may preview the aforementioned post-processing image or perform video encoding according to the aforementioned post-processing image. For example, when the current processed image is a dynamic image of a frame in the preview image sequence, after the corresponding post-processing obtains a post-processed image, the application processing unit 1301 previews the post-processed image; when the current processed image is a dynamic image of a frame in the video image sequence, after the corresponding processing obtains a post-processed image, the application processing unit 1301 performs video encoding according to the post-processed image.
When the post-processing image is a still image, the application processing unit 1301 performs image encoding according to the aforementioned post-processing image, resulting in an encoded image. For example, the pre-processing image is a RAW format image, the post-processing image obtained by the corresponding post-processing is also a RAW format image, and the application processing unit 1301 may perform JPEG image encoding according to the foregoing post-processing image to obtain an encoded image in JPEG format.
Referring to fig. 8, the image signal preprocessor 120 further includes a memory 1204 for transmitting the reference parameters and the preprocessed image to the application processor, including:
when the currently processed image is a moving image, the first image signal processing unit 1201 is configured to directly transmit the reference parameter and the pre-processed image to the second image signal processing unit;
when the currently processed image is a still image, the first image signal processing unit 1201 is configured to write the reference parameters and the pre-processed image into the memory, so that the second image signal processing unit 1302 extracts the reference parameters and the pre-processed image from the memory 1204.
It should be noted that, in the embodiment of the present application, the type of the memory 1204 is not limited, and may be selected by those skilled in the art according to actual needs, and may be a static memory or a dynamic memory. For example, in the embodiment of the application, the memory 1204 is a DDR (Double Data Rate) type dynamic memory.
In the embodiment of the present application, the first image signal processing unit 1201 provides the pre-processed image and the reference parameter to the second image signal processing unit 1302 in different transmission modes according to the image type of the pre-processed image.
When the currently processed image is a moving image, the first image signal processing unit 1201 is configured to directly transmit the reference parameter and the pre-processed image to the second image signal processing unit 1302. Since the moving image needs to consider the real-time nature of the processing, the pre-processed image and the aforementioned reference parameters can be directly transmitted to the second image signal processing unit 1302 by the first image signal processing unit 1201 based on a direct connection (such as MIPI connection) between it and the second image signal processing unit 1302.
When the currently processed image is a still image, the first image signal processing unit 1201 is configured to write the reference parameter and the previously processed image into the memory 1204, and the second image signal processing unit 1302 is configured to extract the previously processed image and the reference parameter from the memory 1204.
The image processing circuit, the circuit board, the electronic device and the image processing method provided in the embodiments of the present application are described in detail above. Specific examples are set forth herein to illustrate the principles and embodiments of the present application, with the description of the examples given above only to assist in understanding the present application. Meanwhile, those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, and the present description should not be construed as limiting the present application in view of the above.
Claims (11)
1. An image processing circuit is characterized by comprising an image sensor, an image signal preprocessor and an application processor, wherein,
the image sensor is connected with the image signal preprocessor and is used for acquiring external images according to configured acquisition parameters and transmitting the external images to the image signal preprocessor; the image sensor is used for packaging each line of image data of a single frame image into an image data packet and sequentially transmitting each image data packet to the image signal preprocessor;
the image signal preprocessor is connected with the application processor and is used for transmitting indication information to the application processor when receiving a packet head of a first image data packet of a single frame image in the external image transmitted by the image sensor;
the application processor is connected with the image sensor and is used for updating acquisition parameters of the image sensor according to the indication information, wherein the acquisition parameters comprise exposure parameters, the application processor collects exposure statistical information required by automatic exposure control, and a configured automatic exposure control algorithm is adopted to calculate new exposure parameters.
2. The image processing circuit according to claim 1, wherein the image signal preprocessor is further configured to extract a reference parameter of the single frame image, and perform preprocessing on the single frame image to obtain a preprocessed image; and
transmitting the reference parameters and the pre-processed image to the application processor;
the application processor is also used for carrying out post-processing on the pre-processing image according to the reference parameter to obtain a post-processing image.
3. The image processing circuit of claim 1, further comprising a register coupled to the image signal preprocessor and the application processor, the image signal preprocessor to:
writing a preset flag bit of the register as a preset value corresponding to the indication information, so that the application processor obtains the indication information according to the preset value.
4. The image processing circuit of claim 1, wherein the image signal preprocessor comprises a first image signal processing unit and a neural network processing unit, the first image signal processing unit to:
Extracting reference parameters of the single-frame image, and transmitting the reference parameters to the application processor;
optimizing the single-frame image to obtain an optimized single-frame image;
the neural network processing unit is used for preprocessing the optimized single-frame image to obtain a preprocessed image;
and transmitting the pre-processed image to the application processor.
5. The image processing circuit of claim 4, wherein the image signal preprocessor further comprises a central processing unit, the first image signal processing unit further configured to synchronously send the indication information to the central processing unit when transmitting the indication information to the application processor;
the central control processing unit is used for controlling the preprocessing of the neural network processing unit according to the indication information.
6. The image processing circuit of claim 4, wherein the application processor comprises an application processing unit and a second image signal processing unit, wherein,
the second image signal processing unit is used for performing post-processing on the pre-processing image according to the reference parameters to obtain a post-processing image;
The application processing unit is used for:
when the post-processing image is a dynamic image, previewing the post-processing image or performing video coding according to the post-processing image; or,
and when the post-processing image is a static image, performing image coding according to the post-processing image.
7. The image processing circuit of claim 6, wherein the image signal preprocessor further comprises a memory, wherein,
when the pre-processing image is a dynamic image, the first image signal processing unit is used for directly transmitting the reference parameter to the second image signal processing unit;
the first image signal processing unit is configured to write the reference parameter into the memory when the pre-processed image is a still image;
the second image signal processing unit is configured to extract the reference parameter from the memory.
8. An electronic device for processing image data, characterized in that the electronic device comprises an image processing circuit as claimed in any of claims 1-6.
9. An image processing method applied to an image processing circuit, wherein the image processing circuit comprises an image sensor, an image signal preprocessor and an application processor, the image processing method comprising:
The image sensor acquires an external image according to the configured acquisition parameters and transmits the external image to the image signal preprocessor; the image sensor is used for packaging each line of image data of a single frame image into an image data packet and sequentially transmitting each image data packet to the image signal preprocessor;
the image signal preprocessor transmits indication information to the application processor when receiving a packet head of a first image data packet of a single frame image in the external image transmitted by the image sensor;
the application processor updates the acquisition parameters of the image sensor according to the indication information, wherein the acquisition parameters comprise exposure parameters, the application processor collects exposure statistical information required by automatic exposure control, and a configured automatic exposure control algorithm is adopted to calculate new exposure parameters.
10. The image processing method according to claim 9, characterized by further comprising:
the image signal preprocessor extracts the reference parameters of the single frame image and preprocesses the single frame image to obtain a preprocessing image; and
transmitting the reference parameters and the pre-processed image to the application processor;
And the application processor performs post-processing on the pre-processing image according to the reference parameter to obtain a post-processing image.
11. The image processing method according to claim 10, wherein the image signal preprocessor includes a first image signal processing unit and a neural network processing unit, wherein,
the first image signal processing unit extracts a reference parameter of the external image; and
optimizing the external image to obtain an optimized external image;
the neural network processing unit performs preprocessing on the optimized external image to obtain a preprocessed image;
the first image signal processing unit transmits the reference parameter and the pre-processed image to the application processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010478315.2A CN113747145B (en) | 2020-05-29 | 2020-05-29 | Image processing circuit, electronic apparatus, and image processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010478315.2A CN113747145B (en) | 2020-05-29 | 2020-05-29 | Image processing circuit, electronic apparatus, and image processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113747145A CN113747145A (en) | 2021-12-03 |
| CN113747145B true CN113747145B (en) | 2024-02-27 |
Family
ID=78724940
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010478315.2A Active CN113747145B (en) | 2020-05-29 | 2020-05-29 | Image processing circuit, electronic apparatus, and image processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113747145B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015081556A1 (en) * | 2013-12-06 | 2015-06-11 | 华为终端有限公司 | Photographing method for dual-camera device and dual-camera device |
| CN107578453A (en) * | 2017-10-18 | 2018-01-12 | 北京旷视科技有限公司 | Compressed image processing method, apparatus, electronic equipment and computer-readable medium |
| CN108364267A (en) * | 2018-02-13 | 2018-08-03 | 北京旷视科技有限公司 | Image processing method, device and equipment |
| CN109801209A (en) * | 2019-01-29 | 2019-05-24 | 北京旷视科技有限公司 | Parameter prediction method, artificial intelligence chip, equipment and system |
| CN110602415A (en) * | 2019-09-30 | 2019-12-20 | 杭州海康威视数字技术股份有限公司 | Exposure control device, method and camera |
-
2020
- 2020-05-29 CN CN202010478315.2A patent/CN113747145B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015081556A1 (en) * | 2013-12-06 | 2015-06-11 | 华为终端有限公司 | Photographing method for dual-camera device and dual-camera device |
| CN107578453A (en) * | 2017-10-18 | 2018-01-12 | 北京旷视科技有限公司 | Compressed image processing method, apparatus, electronic equipment and computer-readable medium |
| CN108364267A (en) * | 2018-02-13 | 2018-08-03 | 北京旷视科技有限公司 | Image processing method, device and equipment |
| CN109801209A (en) * | 2019-01-29 | 2019-05-24 | 北京旷视科技有限公司 | Parameter prediction method, artificial intelligence chip, equipment and system |
| CN110602415A (en) * | 2019-09-30 | 2019-12-20 | 杭州海康威视数字技术股份有限公司 | Exposure control device, method and camera |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113747145A (en) | 2021-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109636754B (en) | Extremely-low-illumination image enhancement method based on generation countermeasure network | |
| US10846551B2 (en) | Video data processing | |
| JP2008544673A (en) | Digital image processing | |
| US10600170B2 (en) | Method and device for producing a digital image | |
| KR100601460B1 (en) | Apparatus and method for interfacing between image sensor and image processor | |
| US20160150158A1 (en) | Photographing apparatus and method for controlling thereof | |
| WO2021238522A1 (en) | Multimedia processing chip, electronic device, and image processing method | |
| CN113810593B (en) | Image processing method, device, storage medium and electronic equipment | |
| US20230086519A1 (en) | Multimedia processing chip, electronic device, and method for dynamic-image processing | |
| KR100770835B1 (en) | Image data processing system and it's method | |
| US7406548B2 (en) | Systems and methods for responding to a data transfer | |
| CN104811601B (en) | A kind of method and apparatus for showing preview image | |
| CN112822371B (en) | Image processing chip, application processing chip, data statistical system and method | |
| CN113747145B (en) | Image processing circuit, electronic apparatus, and image processing method | |
| CN113744119A (en) | Multimedia processing chip and electronic equipment | |
| US9088699B2 (en) | Image communication method and apparatus which controls the output of a captured image | |
| CN113873142B (en) | Multimedia processing chip, electronic device, and moving image processing method | |
| CN114298889A (en) | Image processing circuit and image processing method | |
| CN113766142B (en) | Image processing apparatus, image signal preprocessing module, device, and processing method | |
| CN113873143B (en) | Multimedia processing chip and electronic equipment | |
| CN113873178B (en) | Multimedia processing chip, electronic device and image processing method | |
| KR100932721B1 (en) | How to set color processing parameters and digital imaging equipment | |
| CN113923341A (en) | Electronic device | |
| CN114007009B (en) | Electronic device and image processing method | |
| CN113873141B (en) | Electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |