WO2023020503A1

WO2023020503A1 - Image processing chip, method, application processing chip, and electronic device

Info

Publication number: WO2023020503A1
Application number: PCT/CN2022/112845
Authority: WO
Inventors: 崔惠婷
Original assignee: Oppo广东移动通信有限公司
Priority date: 2021-08-19
Filing date: 2022-08-16
Publication date: 2023-02-23
Also published as: CN115714898A

Abstract

An image processing chip, a method, an application processing chip, and an electronic device. The image processing chip comprises a data packing unit. N frames of obtained RAW images are recombined by the data packing unit according to image processing requirement information of the application processing chip to obtain M data blocks, the M data blocks are encapsulated into L data packets, and the encapsulated L data packets are transmitted to the application processing chip. N and M are positive integers greater than 1, and L is a positive integer less than N.

Description

Image processing chip, method, application processing chip and electronic device

This application claims the priority of the Chinese patent application with the application number 202110955052.4 and the application title "image processing chip, method, application processing chip and electronic equipment" filed with the China Patent Office on August 19, 2021, the entire contents of which are incorporated by reference incorporated in this application.

technical field

The present application relates to the technical field of image processing, in particular to an image processing chip, method, application processing chip and electronic equipment.

Background technique

At present, users usually use electronic devices with cameras (such as digital cameras, smart phones, etc.) to take images, so as to record what happens around them, the scenery they see, etc. anytime and anywhere. In related technologies, in order to improve the quality of captured images, electronic devices usually perform specific image processing on the captured original images. For example, it is possible to capture multiple frames of images, perform high dynamic composite processing on the multiple frames of images to obtain a high dynamic composite image, or perform multi-frame composite noise reduction processing on the multiple frames of images to obtain a noise-reduced composite image, and the like. When multiple frames of images need to be processed, the image sensor in the camera needs to occupy multiple data transmission channels of the application processing chip to transmit the captured multiple frames of images to the application processing chip for image processing.

Contents of the invention

The present application provides an image processing chip, an application processing chip, electronic equipment, and an image processing method, which can reduce the occupation of image data on the data transmission channel of the application processing chip, thereby reducing the impact of image data on other types of data transmission and processing.

The application discloses an image processing chip, including:

The data packaging unit is used to acquire N frames of RAW images, where N is a positive integer greater than 1; and according to the image processing requirement information of the application processing chip, the image data in the acquired N frames of RAW images are reorganized into M data blocks, and M is a positive integer greater than 1; and encapsulates M data blocks into L data packets, where L is a positive integer less than N;

The image processing chip is used to transmit the L data packets to the application processing chip.

The present application also discloses an application processing chip for transmitting image processing requirement information to the image processing chip, and obtaining L data packets corresponding to the image processing requirement information from the image processing chip, including:

an image processing unit, configured to generate image processing requirement information;

The data unpacking unit is used to split L data packets into M data blocks and perform parallel parsing to obtain image data. L data packets are obtained by reassembling the image data in N frames of RAW images into M data blocks and then encapsulating them. , N and M are positive integers greater than 1, and L is a positive integer smaller than N;

The image processing unit is also used to perform image processing on the analyzed image data.

The present application discloses an electronic device, including an image processing chip and an application processing chip, wherein,

The application processing chip is used to generate image processing requirement information, and transmit the image processing requirement information to the image processing chip;

The image processing chip is used to obtain N frames of RAW images, N being a positive integer greater than 1; and reorganizing the image data in the N frames of RAW images into M data blocks according to the image processing requirement information, and M being a positive integer greater than 1; and Encapsulate M data blocks into L data packets and transmit them to the application processing chip, where L is a positive integer smaller than N;

The application processing chip is also used to split the L data packets into M data blocks, perform parallel parsing to obtain image data, and perform image processing on the parsed image data.

The present application discloses an image processing method applied to an image processing chip, including:

Get N frames of RAW images, N is a positive integer greater than 1;

According to the image processing demand information of the application processing chip, the image data in the N frames of RAW images are reorganized into M data blocks, and M is a positive integer greater than 1;

Encapsulate M data blocks into L data packets, where L is a positive integer smaller than N;

Transmitting the L data packets to the application processing chip.

The present application discloses an image processing method applied to an application processing chip, including:

Generate image processing requirement information, and transmit the image processing requirement information to the image processing chip;

Obtaining L data packets corresponding to image processing demand information from the image processing chip;

Split L data packets into M data blocks and perform parallel analysis to obtain image data. L data packets are obtained by reorganizing image data in N frames of RAW images into M data blocks and encapsulating them. N and M are greater than 1 A positive integer, L is a positive integer less than N;

Image processing is performed on the analyzed image data.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings that need to be used in the description of the embodiments.

FIG. 1 is a schematic structural diagram of an image processing chip provided by an embodiment of the present application.

FIG. 2 is an example diagram of reorganizing 3 frames of RAW images into 2 data blocks by the data packing unit in the embodiment of the present application.

Fig. 3 is an example diagram of reorganizing 2 frames of RAW images into 2 data blocks by the data packing unit in the embodiment of the present application.

FIG. 4 is a schematic diagram of a comparison between the packetized transmission mode provided by the embodiment of the present application and related technologies.

FIG. 5 is an example diagram of unpacking, parsing and image processing of data packets transmitted by the image processing chip by using the processing chip in the embodiment of the present application.

FIG. 6 is an exemplary diagram of an ROI of a shooting scene in an embodiment of the present application.

FIG. 7 is an example diagram of reorganizing image data into data blocks by the data packing unit in the embodiment of the present application.

FIG. 8 is a schematic diagram of block division of a row of image data in a RAW image in an embodiment of the present application.

FIG. 9 is an example diagram of encapsulating reassembled data blocks into data packets in the embodiment of the present application.

FIG. 10 is a schematic structural diagram of an application processing chip provided by an embodiment of the present application.

FIG. 11 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

FIG. 12 is a schematic flow chart of an image processing method adopted in the embodiment of the present application.

FIG. 13 is another schematic flowchart of the image processing method provided by the embodiment of the present application.

Detailed ways

It should be noted that the principle of the present application is illustrated by being implemented in an appropriate computing environment. The following description is based on illustrated specific embodiments of the present application, which should not be construed as limiting other specific embodiments of the present application that are not described in detail here.

The relational terms such as first and second involved in the following embodiments of the present application are only used to distinguish one object or operation from another object or operation, and are not used to limit the actual relationship between these objects or operations. sequence relationship.

At present, users usually use electronic devices with cameras (such as digital cameras, smart phones, etc.) to take images, so as to record what happens around them, the scenery they see, etc. anytime and anywhere. In related technologies, in order to improve the quality of captured images, electronic devices usually perform specific image processing on the captured original images. For example, it is possible to capture multiple frames of images, perform high dynamic composite processing on the multiple frames of images to obtain a high dynamic composite image, or perform multi-frame composite noise reduction processing on the multiple frames of images to obtain a noise-reduced composite image, and the like. When multi-frame images need to be processed, the image sensor in the camera needs to occupy multiple data transmission channels of the application processing chip to transmit the captured multi-frame images to the application processing chip for image processing. However, the data transmission of the application processing chip The number of channels is limited. When the data transmission channel is occupied by too much image data, it will affect the transmission and processing of other types of data.

Please refer to FIG. 1. FIG. 1 is a schematic structural diagram of an image processing chip 100 provided by an embodiment of the present application. As shown in FIG. 1, the image processing chip 100 includes a first data interface unit 110, a data packing unit 120 and a second data interface Unit 130.

It should be noted that the image processing chip 100 provided in this application can be configured in an electronic device equipped with an image sensor and an application processing chip (which may be integrated with a central processing unit, a modem, and an image signal processor, etc.), for processing according to the application The image processing demand information of the chip, the image data in N (a positive integer greater than 1) frames of RAW images from the image sensor is reorganized into M (a positive integer greater than 1) data blocks, and the M data The block is encapsulated into L (a positive integer less than N) data packets and then transmitted to the application processing chip for image processing on demand by the application processing chip. Therefore, by encapsulating N frames of RAW images into L data packets smaller than N, this application needs to occupy at most L data transmission channels to realize the transmission of N frames of RAW images. Compared with related technologies that need to occupy N data transmission channels to To realize the transmission mode of N frames of RAW images, the present application can reduce the number of data transmission channels required for the transmission of N frames of RAW images. Wherein, the physical type of the electronic device may be a mobile electronic device such as a smart phone, a tablet computer, a palmtop computer, or a notebook computer, or may be a fixed electronic device such as a desktop computer or a television, which is not specifically limited in this application.

In this embodiment, the first data interface unit 110 is used to provide a data transmission channel between the image sensor and the image processing chip 100 to realize data transmission between the image sensor and the image processing chip 100 . The entity type of the data interface unit 110 is not specifically limited to the embodiment of the present application of the first data interface unit 110 here, including but not limited to Mobile Industry Processor Interface (Mobile Industry Processor Interface, MIPI), PCI-E interface wait. For example, in this application, the entity type of the first data interface unit 110 is a mobile industry processor interface. The image processing chip 100 can acquire N frames of unprocessed original RAW images collected by the image sensor from the image sensor through the first data interface unit 110 , where N is a positive integer greater than 1.

In an optional embodiment, the image processing chip 100 may further include an image preprocessing unit configured to perform image preprocessing on the N frames of RAW images acquired by the image processing chip 100 from the image sensor to obtain preprocessed The next N frames of RAW images.

It should be noted that the image preprocessing performed by the image preprocessing unit can be understood as the image processing performed before the application processing chip performs image processing, which is constrained to be different from the image processing performed by the application processing chip, and can be of any type The image processing can be specifically configured by those skilled in the art according to actual needs.

For example, the image preprocessing unit may include two image processing subunits, respectively a first image processing subunit for traditional image processing, and a second image processing subunit for artificial intelligence-based image processing, which can be configured according to actual needs The image processing required by the first image processing subunit (including but not limited to image processing methods such as bad pixel correction processing, temporal noise reduction processing, 3D noise reduction processing, linearization processing, and black level correction processing), and configuration The image processing required by the second image processing sub-unit (including but not limited to image processing methods such as super-resolution processing and noise reduction processing based on artificial intelligence).

It should be noted that the image processing chip 100 can perform image preprocessing on the N frames of RAW images from the image sensor through the image preprocessing unit as required, or not perform image preprocessing on the N frames of RAW images from the image sensor.

The data packaging unit 120 is configured to acquire N frames of RAW images, and reorganize the acquired N frames of RAW images into M data blocks according to the image processing requirement information of the application processing chip, and package the M data blocks into L data blocks. Bag.

Wherein, the image processing requirement information is used to describe the image processing requirement of the application processing chip, including but not limited to what type of image processing to perform and how to perform image processing.

In this embodiment, the N frames of RAW images acquired by the data packing unit 120 may be the unprocessed N frames of RAW images directly acquired by the image processing chip 100 from the image sensor, or the N frames of RAW images preprocessed by the image preprocessing unit. image. For the acquired N frames of RAW images, the data packaging unit 120 reorganizes the image data in the N frames of RAW images according to the image processing requirement information of the application processing chip, and correspondingly obtains M frames matching the image processing requirements of the application processing chip. Data blocks, the image data in each data block can be independently used in the image processing of the application processing chip. For the recombined M data blocks, the data packing unit 120 further packs the M data blocks into L data packets according to the configured data packing strategy. There is no specific limitation on the configuration of the data encapsulation strategy here, and it can be configured by those skilled in the art according to actual needs, where L is constrained to be less than N, and L can be dynamically set to any positive integer, such as 1.

For example, image processing requirement information can describe:

The application processing chip needs to use 3 frames of RAW images of the same shooting scene for high dynamic range synthesis processing, and it is necessary to perform high dynamic range synthesis processing on the region of interest (Region of Interest, ROI) of the shooting scene first, and then perform high dynamic range synthesis processing on the non-identical images of the shooting scene. Regions of interest are processed for high dynamic range synthesis.

Correspondingly, the image sensor can collect the RAW images of the aforementioned three frames of the shooting scene with different exposure parameters, which are the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image. Correspondingly, the image processing chip 100 acquires a long-exposure RAW image, a medium-exposure RAW image and a short-exposure RAW image from the image sensor through the first data interface unit 110 .

Referring to FIG. 2, the data packing unit 120 reorganizes the image data of the region of interest in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image into a data block (ie, data block 1) according to the above image processing requirement information, The image data of the non-interest regions in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image are reorganized into another data block (ie, data block 2). In this way, the image data in the 3 frames of RAW images are reorganized into 2 data blocks, and each data block includes all the image data required by the application processing chip for high dynamic range synthesis processing.

For another example, image processing requirement information can describe:

The application processing chip needs to use 2 frames of RAW images of the same shooting scene to perform composite noise reduction processing, and different degrees of composite noise reduction processing need to be performed on the in-focus area and non-focus area of the shooting scene.

Correspondingly, the image sensor can collect two frames of RAW images of the aforementioned shooting scene, which are RAW image 1 and RAW image 2 respectively. Correspondingly, the image processing chip 100 acquires the RAW image 1 and the RAW image 2 from the image sensor through the first data interface unit 110 .

Please refer to FIG. 3 , the data packing unit 120 reorganizes the image data of the in-focus areas in the RAW image 1 and the RAW image 2 into one data block (that is, data block 3) according to the above image processing requirement information, and combines the RAW image 1 and the RAW image 2 The image data of the central non-focus area is reorganized into another data block (ie, data block 4). In this way, the image data in the 2 frames of RAW images are reorganized into 2 data blocks, and each data block includes all the image data required by the application processing chip for synthetic noise reduction processing.

The second data interface unit 130 is used to provide a data transmission channel between the application processing chip and the image processing chip 100 to realize data transmission between the application processing chip and the image processing chip 100 . Here, the entity type of the second data interface unit 130 is not specifically limited, including but not limited to Mobile Industry Processor Interface (MIPI), PCI-E interface, and the like. For example, in the embodiment of the present application, the entity type of the second data interface unit 130 is a mobile industry processor interface.

In this embodiment, the image processing chip 100 transmits the L data packets packaged by the data packaging unit 120 to the application processing chip through the second data interface unit 130 for the application processing chip to perform image processing. It can be understood that since N frames of RAW images are packaged into L data packets, it only needs to occupy at most L data transmission channels of the application processing chip for transmission. For example, please refer to Figure 4 (where the arrow represents the data transmission channel), when the image sensor directly transmits N frames of RAW images to the application processing chip, it needs to occupy N data transmission channels of the application processing chip to realize the transmission of N frames of RAW images , and when the image processing chip 100 provided by the present application is used to package the N frames of RAW images collected by the image sensor into L data packets and transmit them to the application processing chip, if the N frames of RAW images are packaged into one data packet by the data packaging unit 120 , then only one data transmission channel of the application processing chip needs to be occupied.

In addition, it should be noted that in the embodiment of the present application, the data packing unit 120 does not directly pack the N frames of RAW images into data packets, but reorganizes the image data in the N frames of RAW images into M data blocks, and The M data blocks are matched with the image processing required by the application processing chip, and then the M data blocks are packaged into L data packets. Correspondingly, after the application processing chip receives the data packet transmitted by the image processing chip 100 through the second data interface unit 130, it can split the data packet into M data blocks and perform parallel analysis, and analyze the M data blocks The obtained image data is subjected to image processing.

For example, please refer to FIG. 2 and FIG. 5 together. On the one hand, the image processing chip 100 reorganizes 3 frames of RAW images into 2 data blocks through the data packaging unit 120 according to the reorganization method shown in FIG. 2 , and packages the 2 data blocks. is a data packet, and the encapsulated data packet is transmitted to the application processing chip through the second data interface unit 130 .

On the other hand, after receiving the data packet, the application processing chip splits the data packet into two data blocks, that is, data block 1 and data block 2, and then performs parallel parsing on data block 1 and data block 2, from Data block 1 is analyzed to obtain the image data of the region of interest in the long exposure RAW image (represented by A in Figure 5), the image data of the region of interest in the medium exposure RAW image (represented by B in Figure 5) and the short exposure RAW image The image data of the region of interest in the medium (represented by C in FIG. 5 ), and the image data of the non-interest region in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image are analyzed from the data block 2. Afterwards, the application processing chip performs high dynamic range synthesis processing on the image data obtained by analyzing data block 1 to obtain high dynamic range region-of-interest image data (indicated by D in FIG. 5 ), and performs image data analysis on data block 2. High dynamic range synthesis processing to obtain high dynamic range non-interest region image data. Finally, the application processing chip stitches the high dynamic range ROI image data and the non-ROI image data to obtain a high dynamic range composite image.

It can be seen from the above that the image processing chip 100 provided by the present application includes a data packing unit 120, through which N frames of RAW images are obtained, and according to the image processing requirement information of the application processing chip, the obtained N frames of RAW images are processed. Recombine to obtain M data blocks, and encapsulate the M data blocks into L data packets and transmit them to the application processing chip. Wherein, N and M are positive integers greater than 1, and L is a positive integer smaller than N. Therefore, by encapsulating N frames of RAW images into L data packets smaller than N, the image processing chip 100 needs to occupy at most L data transmission channels of the application processing chip to realize the transmission of N frames of RAW images. Occupying N data transmission channels to realize the transmission mode of N frames of RAW images, this application can reduce the occupation of image data on the data transmission channels of the application processing chip, thereby reducing the impact of image data on other types of data transmission and processing. In addition, the reorganized M data blocks can be analyzed in parallel by the application processing chip, which can also reduce the time required for analysis.

In an optional embodiment, the data packing unit 120 is configured to assign priorities to image data at different positions in each RAW image according to image processing requirement information; Reorganize into data blocks to obtain M data blocks.

In this embodiment, in addition to describing what type of image processing is performed by the application processing chip and how to perform image processing, the image processing requirement information may also describe the importance information defined by the application processing chip for different regions in the shooting scene of the image sensor . Correspondingly, when the data packing unit 120 allocates the priority, it can allocate according to the importance of the image data. Wherein, when the image processing requirement information describes that the application processing chip needs to use image data at the same position in different RAW images for image processing, in order to match the image processing requirements of the application processing chip, the image data at the same position in different RAW images will be The data packing unit 120 assigns the same priority.

For example, referring to FIG. 6, for a shooting scene, the application processing chip identifies two regions of interest, which are respectively region of interest 1 (represented by ROI1 in FIG. 6 ) and region of interest 2 (represented by ROI2 in FIG. 6 ). , and it is defined that the importance of ROI 1 is greater than that of ROI 2, and the importance of ROI 2 is greater than that of non-ROIs (that is, in Figure 6 except ROI 1 and ROI 2 area) importance.

Image processing requirements information can describe:

The application processing chip needs to use 3 frames of RAW images of the same shooting scene for high dynamic range synthesis processing. In the shooting scene, the importance of the area of interest 1 is greater than that of the area of interest 2, and the importance of the area of interest 2 is greater than that of the non-interesting area. region, and it is necessary to perform a high degree of high dynamic range synthesis processing on the region of interest 1 in the shooting scene, perform a moderate degree of high dynamic range synthesis processing on the region of interest 2 in the shooting scene, and perform non-interesting Areas are processed with a low degree of HDR compositing.

Correspondingly, the image sensor can collect the RAW images of the aforementioned three frames of the shooting scene with different exposure parameters, which are the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image. Correspondingly, the image processing chip 100 acquires a long-exposure RAW image, a medium-exposure RAW image and a short-exposure RAW image from the image sensor through the first data interface unit 110 . Image preprocessing is performed on the acquired long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image through an image preprocessing unit.

The data packaging unit 120 obtains the preprocessed long exposure RAW image, medium exposure RAW image and short exposure RAW image, and according to the above image processing requirement information, obtains the preprocessed long exposure RAW image, medium exposure RAW image and short exposure RAW image The image data corresponding to ROI 1 in the RAW image is assigned a priority of "high", and the image data corresponding to ROI 2 in the long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image is assigned a priority of "medium". The image data corresponding to the non-interest region in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image is assigned a priority of “low”. Then, the data packing unit 120 reorganizes the image data whose priority is “high” in the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image into one data block, and combines the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image into one data block. Image data with a priority of "Medium" in the RAW image is reorganized into one data block, and image data with a priority of "Low" in the long-exposure RAW image, medium-exposure RAW image, and short-exposure RAW image are reorganized into one data block, A total of 3 data blocks are obtained. These 3 data blocks include the image data corresponding to the region of interest 1 in the long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image, and the long-exposure RAW image, medium-exposure RAW image and short The image data corresponding to the region of interest 2 in the exposure RAW image, and the image data corresponding to the non-interest region in the long exposure RAW image, the medium exposure RAW image and the short exposure RAW image.

It should be noted that, in other embodiments, when the data packing unit 120 assigns priorities, it can also assign them according to attribute information other than the importance of the image data. For example, the data packing unit 120 can also assign the priorities according to the The exposure of the data is assigned priority, etc.

In an optional embodiment, the data block includes a plurality of data rows, the data row includes a header part and a row body part, and the data packing unit 120 is configured to sequentially write image data of the same priority in N frames of RAW images into the same The row body part of different data rows of the data block; and the first description information of the image data written in the row body part of each data row is obtained, and the first description information corresponding to each data row is written into the row header of each data row part.

It should be noted that the first description information includes at least description information used to describe how the data packing unit reorganizes the image data, for example, the first description information may include the source RAW image of the image data written in the body part of the data row Information (used to describe which RAW image the image data comes from), position information in the source RAW image, data size information, priority information, etc.

Wherein, when performing image data reorganization, the data packing unit 120 may alternately write image data of the same priority in N frames of RAW images into the row body parts of different data rows of the same data block in sequence. That is to say, in the same data block, the image data written in the body part of all data lines has the same priority, but the image data written in the body part of two adjacent data lines comes from different RAW images.

For example, according to the priority allocation method corresponding to the region of interest shown in FIG. 6 (for details, please refer to the relevant descriptions in the above embodiments), the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image with a priority of “high " image data reorganization as an example to illustrate.

Please refer to FIG. 7 , where the value of O depends on the data size of the image data to be written. The data packing unit 120 identifies whether there is image data with “high” priority in the first row of image data in the short-exposure RAW image, and if so, writes the image data with “high” priority into the row body in data row 1 If it does not exist, continue to identify whether there is image data with a priority "high" in the image data of the next row until it is recognized that the image data with a priority "high" is written into the row body part in data row 1; then, the data Packing unit 120 identifies whether there is image data with a priority of "high" in the first line of image data in the mid-exposure RAW image, and if so, writes the image data with "high" priority into the line body part of data line 2 , if it does not exist, continue to identify whether there is image data with a priority "high" in the image data of the next line until it is recognized that the image data with a priority "high" is written into the line body part in data line 2; then, the data Packing unit 120 identifies whether there is image data with “high” priority in the first row of image data in the long-exposure RAW image, and if so, writes the image data with “high” priority into the row body part in data row 3 , if it does not exist, continue to identify whether there is image data with a priority "high" in the image data of the next row until it is recognized that the image data with a priority "high" is written into the row body part in data row 3. In this way, the data packing unit 120 alternately packs the short-exposure RAW image, the medium-exposure RAW image, and the long-exposure RAW image in the sequence of the short-exposure RAW image, the medium-exposure RAW image, and the long-exposure RAW image with “high” priority Data is sequentially written to the row body part of different data rows of the same data block. In addition, for each data row, the data packing unit 120 also obtains the first description information of the image data written in the row body part in each data row, and writes the first description information corresponding to each data row into each data row. The line header part of the line. For example, the first description information acquired at this time may include: source RAW image information (the image data used to describe the writing here is from a short-exposure RAW image, a medium-exposure RAW image, or a long-exposure RAW image), Position information in (used to describe the number of rows where the written image data is located in the entire source RAW image), data size information, priority information (used to describe the priority of the written image data as high, medium or low) , whether it belongs to the mark information of the region of interest, the number of blocks (describe how many blocks of a line of image data in the source RAW image are included in the written image data), the starting abscissa of the region of interest, and the region of interest The end abscissa of , etc.

Please continue to refer to FIG. 8 to illustrate the number of blocks by taking the reconstruction of a line of image data in a long-exposure RAW image as an example. The row of image data can be divided into 5 blocks, namely Block 1, Block 2, Block 3, Block 4, and Block 5, of which Block 1, Block 3, and Block 5 are not of interest Area, block 2 belongs to ROI 1, and block 4 belongs to ROI 2. Correspondingly, block 1, block 3 and block 5 will be reorganized into the same data line of the same data block by the data packing unit 120, and for this data line, the number of blocks in the first description information is 3; Block 2 will be reorganized into a data line of a data block by the data packing unit 120, and for this data line, the number of blocks in its first description information is 1; block 2 will be reorganized by the data packing unit 120 into In a data row of a data block, the number of blocks in the first description information is 1 for the data row.

In an optional embodiment, the data packet includes a packet header and a packet body, and when M is greater than or equal to N, L is equal to 1, and the data packing unit 120 is configured to, according to the respective priorities of the M data blocks, M data blocks are sequentially written into the packet body part of the same data packet, and the second description information of the M data blocks is obtained, and the second description information is written into the header part of the same data packet to obtain 1 data packet; or,

When M is less than N, L is equal to M, and the data packing unit 120 is configured to respectively write M data blocks into the body parts of M data packets, and obtain the data blocks written in the body parts of each data packet The third descriptive information is to write the third descriptive information corresponding to each data packet into the packet header of each data packet to obtain M data packets.

Wherein, the second description information includes the number information of the data blocks written in the package body (that is, the value of M), the priority information of each data block, and the position information (used to describe the position of the data block in the entire data package) ) and at least one of the data size information, the third description information includes the serial number information of the data block written in the package body (for describing which data block the written data block is in the M data blocks), At least one of priority information and data size information.

For example, referring to FIG. 9 , the data packing unit 120 reorganizes the N frames of RAW images obtained by the first data interface unit 110 into M (3) data blocks, which are respectively data block 1, data block 2 and data block 3, and Assign priority "high" to data block 1, assign priority "middle" to data block 2, and assign priority "low" to data block 3. Correspondingly, since M is equal to N at this time, the data packing unit 120 sequentially packs the data Block 1, data block 2, and data block 3 are written into the packet body of the same data packet, and then, the data packing unit 120 obtains the second description information of the three data blocks and writes them into the packet header, including the written data block The number information "3", the location information, priority information, and data size information of data block 1, the location information, priority information, and data size information of data block 2, and the location information, priority information, and data size information of data block 3 size information.

In an optional embodiment, the image processing chip 100 is configured to send the packet header part of the data packet to the application processing chip; and send the packet body The M data blocks in the section are sequentially sent to the application processing chip.

For example, the second data interface unit 130 first sends the header part of the data packet to the application processing chip, and then according to the priority of the M data blocks written in the packet body part, according to the order of priority from high to low, sequentially Send the M data blocks in the packet body to the application processing chip.

In this embodiment, the application processing chip can perform parallel parsing and image processing on the M data blocks after receiving a complete data packet, or can perform processing on the received data blocks after receiving a data block each time. parsing and image processing.

In an optional embodiment, the image processing chip 100 is configured to send M data packets to the application processing chip in parallel.

For example, assuming that the data packaging unit 120 encapsulates and obtains three data packets, the image processing chip 100 sends the three data packets obtained by encapsulation to the application processing chip in parallel through the three data transmission channels provided by the second data interface unit 130. , supplying the application processing chip for image processing.

In an optional embodiment, the data packing unit 120 is configured to obtain N frames of RAW images when the number of idle data transmission channels of the application processing chip is less than N, and according to the image processing requirement information of the application processing chip, N The image data in the frame RAW image is reorganized into M data blocks and then encapsulated into L data packets.

In this embodiment, the image processing chip 100 does not always package and transmit N frames of images from the image sensor. Wherein, the data packaging unit 120 recognizes the number of idle data transmission channels of the application processing chip in real time, and when the number of idle data transmission channels of the application processing chip is less than N, acquires N frames of RAW images, and performs processing according to the image processing requirements of the application processing chip information, reorganize the image data in N frames of RAW images into M data blocks and then encapsulate them into L data packets. For details, please refer to the relevant descriptions in the above embodiments, and details will not be repeated here.

In this embodiment, the second data interface unit 130 is configured to transmit the data packet to the application processing chip through an idle data transmission channel of the application processing chip.

For example, when the application processing chip chip has only one idle data transmission channel, the image processing chip 100 can transmit the data packet to the application processing chip through the unique data transmission channel provided by the second data interface unit 130;

For another example, when the idle data transmission channel of the application processing chip is not unique, the image processing chip 100 can transmit the data packet to the application processing chip through any idle data transmission channel provided by the second data interface unit 130, and can also transmit the data packet to the application processing chip according to the idle data The number of transmission channels splits the data packet into the same number of sub-packets, and equalizes the split sub-packets to each idle data transmission channel for transmission to the application processing chip.

In other embodiments, when the number of idle data channels of the application processing chip is greater than or equal to N, the image processing chip 100 is configured to transmit N frames of RAW images to the application processing chip in parallel.

Referring to FIG. 10 , the present application also provides an application processing chip 200 . As shown in FIG. 10 , the application processing chip 200 includes a third data interface unit 210 , a data unpacking unit 220 and an image processing unit 230 .

It should be noted that the application processing chip 200 provided in this application can be configured in an electronic device equipped with an image sensor and an image processing chip provided in this application to implement image processing. Among them, the entity type of the electronic equipment may be a mobile electronic equipment such as a smart phone, a tablet computer, a handheld computer, or a notebook computer, or a fixed electronic equipment such as a desktop computer or a television, and this application does not make specific limitations on this

In this embodiment, the third data interface unit 210 is used to provide a data transmission channel between the application processing chip 200 and the image processing chip, so as to realize data transmission between the application processing chip 200 and the image processing chip. Here, there is no specific limitation on the entity type of the third data interface unit 210, including but not limited to Mobile Industry Processor Interface (MIPI), PCI-E interface, etc. For example, in the embodiment of the present application, the entity type of the third data interface unit 210 is a mobile industry processor interface.

The image processing unit 220 is configured to generate image processing requirement information, which is used to describe the image processing requirement of the image processing unit 230 , including but not limited to what type of image processing to perform and how to perform image processing.

The application processing chip 200 is configured to transmit the image processing requirement information of the image processing unit 230 to the image processing chip through the third data interface unit 210, so that the image processing chip can convert the image in the acquired N frames of RAW images according to the image processing requirement information The data is reorganized into M data blocks and then encapsulated into L data packets, where N and M are positive integers greater than 1, and L is a positive integer smaller than N. For specific description, please refer to the relevant description in the above embodiment of the image processing chip, which will not be repeated here. In addition, the application processing chip 200 is further configured to acquire L data packets corresponding to image processing requirement information from the image processing chip. Wherein, the application processing chip 200 may receive L data packets sent by the image processing chip through the third data interface unit 210 .

The data unpacking unit 220 is configured to split the L data packets into M data blocks and perform parallel parsing to obtain image data. Wherein, the data unpacking unit 220 reversely parses the L data packets into M data blocks after the application processing chip 200 obtains the L data packets corresponding to the image processing demand information, and performs parallel parsing on the M data blocks, Get the corresponding image data.

The image processing unit 230 is configured to perform image processing on the image data obtained by analyzing the M data blocks.

For example, image processing requirement information can describe:

The image processing unit 230 needs to use 3 frames of RAW images of the same shooting scene to perform high dynamic range synthesis processing, and firstly perform high dynamic range synthesis processing on the region of interest (Region of Interest, ROI) of the shooting scene, and then perform high dynamic range synthesis processing on the region of interest (ROI) of the shooting scene. Regions of non-interest are processed for high dynamic range compositing.

Correspondingly, the image sensor can collect the RAW images of the aforementioned three frames of the shooting scene with different exposure parameters, which are the long-exposure RAW image, the medium-exposure RAW image and the short-exposure RAW image. Correspondingly, the image processing chip acquires a long-exposure RAW image, a medium-exposure RAW image and a short-exposure RAW image from the image sensor. And according to the above image processing requirement information, the image data of the region of interest in the long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image are reorganized into a data block, and the long-exposure RAW image, medium-exposure RAW image and short-exposure RAW image Image data for areas not of interest in the image are reorganized into another data block. In this way, the image data in the 3 frames of RAW images are reorganized into 2 data blocks, and each data block includes all the image data required by the image processing unit 230 for the high dynamic range synthesis process. Afterwards, the image processing chip packs the two data blocks obtained through recombination into two data packets and transmits them to the application processing chip 200 . On the other hand, the application processing chip 200 will receive the two data packets transmitted by the image processing chip through the third data interface unit 210 .

Afterwards, the data unpacking unit 220 splits the two data packets into two data blocks, and then analyzes the two data blocks in parallel, and obtains a long-exposure RAW image, a medium-exposure RAW image, and a short-exposure RAW image from one of the data blocks. The image data of the region of interest in the image is quickly analyzed from another data to obtain the image data of the non-interest region in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image.

Afterwards, the image processing unit 230 performs high dynamic range synthesis processing on the image data of the region of interest in the long-exposure RAW image, the medium-exposure RAW image, and the short-exposure RAW image to obtain high dynamic range image data of the region of interest. The image data of the non-interest region in the image, the medium-exposure RAW image and the short-exposure RAW image are subjected to high dynamic range synthesis processing to obtain high dynamic range non-interest region image data. Finally, the image processing unit 230 stitches the high dynamic range ROI image data and the non-ROI image data to obtain a high dynamic range composite image.

In an optional embodiment, the data unpacking unit 220 is configured to invoke M idle parsing threads from a preset thread pool to perform parallel parsing on the M data blocks.

Among them, the thread pool includes multiple different types of threads, which are obtained through pre-construction.

For example, assuming that the image processing chip reassembles the acquired 3 frames of RAW images into 2 data blocks, the data unpacking unit 220 may call 2 idle parsing threads from the preset thread pool to analyze the 2 data blocks in parallel.

In an optional embodiment, the data unpacking unit 220 is also configured to call the idle parsing thread in the thread pool according to the priority corresponding to each data block when the number of idle parsing threads in the thread pool is less than M The M data blocks are parsed in sequence.

In this embodiment, the data unpacking unit 220 will analyze the data blocks sequentially according to the priorities of the data blocks when there are not enough free parsing threads to perform parallel parsing on the split data blocks.

For example, when there is only one idle parsing thread, the data unpacking unit 220 may call the only idle parsing thread to parse the M data blocks obtained by splitting the data packet in sequence according to the order of priority from high to low;

For another example, when there are K idle parsing threads (K is a positive integer greater than 1 and less than M), the data unpacking unit 220 may call the K idle parsing threads first in order of priority from high to low. Parallel parsing is performed on the K data blocks with the highest priority; if the remaining unparsed data blocks are greater than K, K idle parsing threads can be called again to perform parallel parsing on the remaining K data blocks with the highest priority until the remaining The number of unresolved data blocks is less than K; at this time, according to the number of remaining unresolved data blocks, the corresponding number of idle parsing threads can be called to analyze the remaining unresolved data blocks. To parse the data block, call 1 idle parsing thread to parse the remaining 1 unparsed data block; Parallel parsing.

In an optional embodiment, the image processing unit 230 is further configured to perform image processing on the analyzed image data when the data unpacking unit 220 finishes analyzing a data block in the M data blocks each time.

It should be noted that different data blocks have different data sizes. When the data unpacking unit 220 parses the data blocks, some data blocks will be parsed first, and some data blocks will be parsed later. Since the image data in each data block can be independently used for image processing, in order to improve the image processing efficiency, the image processing unit 230 can complete the analysis of a data block in the M data blocks each time when the data unpacking unit 220, that is Image processing is performed on the analyzed image data.

Please refer to FIG. 11 , the present application also provides an electronic device 10, including an image processing chip 100 and an application processing chip 200, wherein,

The application processing chip 200 is configured to generate image processing requirement information, and transmit the image processing requirement information to the image processing chip 100;

The image processing chip 100 is configured to acquire N frames of RAW images, where N is a positive integer greater than 1; and reorganize the image data in the N frames of RAW images into M data blocks according to the image processing requirement information, and M is a positive integer greater than 1 ; and encapsulate the M data blocks into L data packets and transmit them to the application processing chip 200;

The application processing chip 200 is further configured to split the L data packets into M data blocks, perform parallel parsing to obtain image data, and perform image processing on the parsed image data.

In an optional embodiment, the image processing chip 100 is configured to assign priorities to image data at different positions in each RAW image according to image processing requirement information; and assign image data with the same priority in N frames of RAW images Reorganize into data blocks to obtain M data blocks.

In an optional embodiment, the data block includes a plurality of data rows, the data row includes a row header part and a row body part, and the image processing chip 100 is configured to sequentially write image data of the same priority in N frames of RAW images into the same The row body part of different data rows of the data block; and the first description information of the image data written in the row body part of each data row is obtained, and the first description information corresponding to each data row is written into the row header of each data row part.

In an optional embodiment, the data packet includes a packet header and a packet body. When M is greater than or equal to N, L is equal to 1, and the image processing chip 100 is configured to assign The M data blocks are sequentially written into the packet body; and the second description information of the M data blocks is acquired, and the second description information is written into the packet header.

In an optional embodiment, the image processing chip 100 is configured to send the packet header part to the application processing chip; and send the M data blocks in the packet body part to the application processing chip in sequence according to the respective priorities of the M data blocks. Application processing chip 200 .

In an optional embodiment, when M is less than N, L is equal to M, and the image processing chip 100 is configured to respectively write M data blocks into the packet bodies of M data packets, and obtain The third descriptive information of the data block written in the packet body, the third descriptive information corresponding to each data packet is written into the packet header of each data packet.

For specific description, please refer to the relevant embodiments above, and details are not repeated here.

Please refer to Figure 12, the present application also provides an image processing method, which is applied to an image processing chip, including:

In 410, N frames of RAW images are acquired, where N is a positive integer greater than 1.

In 420, according to the image processing requirement information of the application processing chip, reorganize the image data in the N frames of RAW images into M data blocks, where M is a positive integer greater than 1.

In 430, the M data blocks are encapsulated into L data packets, where L is a positive integer smaller than N.

At 440, L data packets are transmitted to the application processing chip.

In an optional embodiment, according to the image processing demand information of the application processing chip, assign priorities to image data in different positions in each RAW image; and recombine image data with the same priority in N frames of RAW images into data blocks to get M data blocks.

In an optional embodiment, the data block includes a plurality of data lines, and the data line includes a line header part and a line body part, and the image data of the same priority in N frames of RAW images are sequentially written into different data lines of the same data block The row body part: acquiring the first description information of the image data written in the row body part in each data row, and writing the first description information corresponding to each data row into the row header part of each data row.

In an optional embodiment, the first description information includes at least one of source RAW image information, position information in the source RAW image, data size information, and priority information of the image data written in the row body part of the data row. kind.

In an optional embodiment, the data packet includes a packet header and a packet body. When M is greater than or equal to N, L is equal to 1. According to the respective priorities of the M data blocks, the M data blocks are sequentially written to the packet body; acquire the second description information of the M data blocks, and write the second description information into the packet header.

In an optional embodiment, the packet header is sent to the application processing chip; and the M data blocks in the packet body are sequentially sent to the application processing chip according to the respective priorities of the M data blocks.

In an optional embodiment, when M is less than N, L is equal to M, M data blocks are respectively written into the body parts of M data packets, and the data written in the body parts of each data packet is acquired For the third description information of the block, write the third description information corresponding to each data packet into the packet header of each data packet.

In an optional embodiment, the second description information includes at least one of the number information of the data blocks written in the packet body, the priority information of each data block, the location information and the data size information; and/ Alternatively, the third description information includes at least one of sequence number information, priority information, and data size information of the data blocks written in the packet body.

It should be noted that the image processing method provided by the embodiment of the present application belongs to the same concept as the above image processing chip, and for detailed description, please refer to the relevant description of the image processing chip in the above embodiment, and details will not be repeated here.

Please refer to Figure 13, the present application also provides an image processing method, which is applied to an application processing chip, including:

In 510, image processing requirement information is generated, and the image processing requirement information is transmitted to the image processing chip.

In 520, L data packets corresponding to image processing requirement information are acquired from the image processing chip.

In 530, the L data packets are split into M data blocks and then analyzed in parallel to obtain image data. The L data packets are obtained by reassembling the image data in N frames of RAW images into M data blocks and then encapsulating them. N, M is a positive integer greater than 1, and L is a positive integer less than N.

In 540, image processing is performed on the analyzed image data.

In an optional embodiment, M idle parsing threads are invoked from a preset thread pool to perform parallel parsing on M data blocks.

In an optional embodiment, when the number of idle parsing threads in the thread pool is less than M, according to the priority corresponding to each data block, the idle parsing threads in the thread pool are called to parse the M data blocks in turn.

In an optional embodiment, each time the analysis of a data block among the M data blocks is completed, image processing is performed on the image data obtained through analysis.

It should be noted that the image processing method provided in the embodiment of the present application is based on the same idea as the application processing chip provided above. For detailed description, please refer to the relevant description of the application processing chip in the above embodiment, and details will not be repeated here.

The image processing chip, the application processing chip, the electronic device and the image processing method provided in the embodiments of the present application have been introduced in detail above. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application, and the descriptions of the above embodiments are only used to help understand the present application. At the same time, for those skilled in the art, based on the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the application.

Claims

An image processing chip, including:

The data packaging unit is used to acquire N frames of RAW images, where N is a positive integer greater than 1; and according to the image processing requirement information of the application processing chip, reorganize the image data in the N frames of RAW images into M data blocks, M is a positive integer greater than 1; and encapsulates the M data blocks into L data packets, where L is a positive integer less than N;

The image processing chip is used to transmit the L data packets to the application processing chip.
The image processing chip according to claim 1, wherein the data packing unit is used for assigning priorities to image data in different positions in each RAW image according to the image processing requirement information; and the N frames of RAW Image data of the same priority in the image are reorganized into data blocks to obtain the M data blocks.
The image processing chip according to claim 2, wherein the data block includes a plurality of data lines, the data lines include a line header part and a line body part, and the data packing unit is used to pack the N frames of RAW images into Image data of the same priority is sequentially written into the row body part of different data rows of the same data block; The first description information is written in the row header of each data row.
The image processing chip according to claim 3, wherein the first description information includes source RAW image information, position information in the source RAW image, data size information, and priority At least one of the level information.
The image processing chip according to claim 2, wherein the data packet includes a packet header and a packet body, and when M is greater than or equal to N, L is equal to 1, and the data packing unit is used to The corresponding priority of each block, writing the M data blocks into the packet body part in turn; and obtaining the second description information of the M data blocks, and writing the second description information into the packet header part.
The image processing chip according to claim 5, wherein the image processing chip is configured to send the packet header part to the application processing chip; and according to the respective priorities of the M data blocks, the The M data blocks in the packet body are sequentially sent to the application processing chip.
The image processing chip according to claim 5, wherein, when M is less than N, L is equal to M, and the data packing unit is used to write the M data blocks into the body parts of M data packets respectively, and The third description information of the data block written in the body part of each data packet is obtained, and the third description information corresponding to each data packet is written into the header part of each data packet.
The image processing chip according to claim 7, wherein the second description information includes information on the number of data blocks written in the package body, priority information of each data block, position information, and data size information. at least one; and/or

The third description information includes at least one of sequence number information, priority information and data size information of the data blocks written in the packet body.
An application processing chip, wherein the application processing chip is used to transmit image processing requirement information to the image processing chip, and obtain L data packets corresponding to the image processing requirement information from the image processing chip, including:

an image processing unit, configured to generate the image processing requirement information;

A data unpacking unit, configured to split the L data packets into M data blocks and perform parallel parsing to obtain image data, and the L data packets are reorganized into M data from image data in N frames of RAW images Obtained by encapsulation after the block, N and M are positive integers greater than 1, and L is a positive integer less than N;

The image processing unit is also used to perform image processing on the analyzed image data.
The application processing chip according to claim 9, wherein the data unpacking unit is configured to invoke M idle parsing threads from a preset thread pool to perform parallel parsing on the M data blocks.
The application processing chip according to claim 10, wherein the data unpacking unit is further configured to, when the number of idle parsing threads in the thread pool is less than M, according to the priority corresponding to each data block, call the The idle parsing threads in the thread pool parse the M data blocks in turn.
The application processing chip according to claim 11, wherein the image processing unit is further configured to, when the data unpacking unit finishes analyzing a data block in the M data blocks, analyze the obtained The image data is subjected to image processing.
An electronic device, including an image processing chip and an application processing chip, wherein,

The application processing chip is used to generate image processing requirement information, and transmit the image processing requirement information to the image processing chip;

The image processing chip is used to acquire N frames of RAW images, where N is a positive integer greater than 1; and reorganizes the image data in the N frames of RAW images into M data blocks according to the image processing requirement information, and M is greater than A positive integer of 1; and the M data blocks are encapsulated into L data packets and then transmitted to the application processing chip, where L is a positive integer smaller than N;

The application processing chip is further configured to split the L data packets into the M data blocks, perform parallel parsing to obtain image data, and perform image processing on the parsed image data.
The electronic device according to claim 13, wherein the image processing chip is configured to assign priorities to image data at different positions in each RAW image according to the image processing requirement information; and divide the N frames of RAW images Image data of the same priority in the image data are reorganized into data blocks to obtain the M data blocks.
The electronic device according to claim 14, wherein the data block includes a plurality of data rows, the data rows include a row header part and a row body part, and the image processing chip is used to convert the same The priority image data is sequentially written into the row body part of different data rows of the same data block; and the first description information of the image data written in the row body part in each data row is obtained, which will correspond to the first description information of each data row A description information is written in the row header portion of each data row.
The electronic device according to claim 14, wherein the data packet includes a packet header and a packet body, and when M is greater than or equal to N, L is equal to 1, and the image processing chip is used to Writing the M data blocks into the packet body in turn according to their respective priorities; and obtaining the second description information of the M data blocks, and writing the second description information into the header part .
The electronic device according to claim 16, wherein the image processing chip is configured to send the header portion of the packet to the application processing chip; The M data blocks in the body part are sequentially sent to the application processing chip.
The electronic device according to claim 16, wherein, when M is less than N, L is equal to M, and the image processing chip is used to respectively write the M data blocks into the packet bodies of M data packets, and obtain The third description information of the data block written in the body part of each data packet is written into the header part of each data packet with the third description information corresponding to each data packet.
An image processing method applied to an image processing chip, including:

Get N frames of RAW images, N is a positive integer greater than 1;

According to the image processing demand information of the application processing chip, the image data in the N frames of RAW images are reorganized into M data blocks, and M is a positive integer greater than 1;

Encapsulating the M data blocks into L data packets, where L is a positive integer less than N;

and transmitting the L data packets to the application processing chip.
An image processing method applied to an application processing chip, including:

generating image processing requirement information, and transmitting the image processing requirement information to an image processing chip;

Obtaining L data packets corresponding to the image processing requirement information from the image processing chip;

The L data packets are split into M data blocks and then analyzed in parallel to obtain image data. The L data packets are obtained by reassembling the image data in N frames of RAW images into M data blocks and then encapsulating them. N, M is a positive integer greater than 1, L is a positive integer less than N;

Image processing is performed on the analyzed image data.