CN115118886A - Image processing chip, application processing chip, electronic device and image processing method - Google Patents
Image processing chip, application processing chip, electronic device and image processing method Download PDFInfo
- Publication number
- CN115118886A CN115118886A CN202110304163.9A CN202110304163A CN115118886A CN 115118886 A CN115118886 A CN 115118886A CN 202110304163 A CN202110304163 A CN 202110304163A CN 115118886 A CN115118886 A CN 115118886A
- Authority
- CN
- China
- Prior art keywords
- image
- information
- processing chip
- exposure
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 214
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000005286 illumination Methods 0.000 claims description 75
- 238000004364 calculation method Methods 0.000 claims description 35
- 238000000034 method Methods 0.000 claims description 28
- 239000011159 matrix material Substances 0.000 claims description 19
- 238000010606 normalization Methods 0.000 claims description 7
- 235000019557 luminance Nutrition 0.000 description 28
- 230000008569 process Effects 0.000 description 19
- 238000010586 diagram Methods 0.000 description 7
- 238000012937 correction Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Landscapes
- Studio Devices (AREA)
Abstract
The embodiment of the application provides an image processing chip, an application processing chip, electronic equipment and an image processing method. When the embodiment of the application is applied to the electronic equipment, the target data required by calculating the exposure parameters can be obtained through the image processing chip before image acquisition, the exposure parameters are calculated by the application processing chip according to the target data during image acquisition, the camera is controlled to perform image acquisition according to the exposure parameters, and the speed of exposure control in image acquisition is effectively improved.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing chip, an application processing chip, an electronic device, and an image processing method.
Background
With the rapid development of electronic device manufacturing technology, cameras are configured on electronic devices such as mobile phones and tablet computers. The camera can be used for shooting delicacies and landscapes, and the daily life of people is enriched.
At present, electronic devices equipped with a camera all have an exposure control function. The exposure control function means that when an image is collected, the electronic equipment automatically adjusts the exposure according to the intensity of light to prevent overexposure or underexposure.
However, in the related art, the electronic apparatus has a problem that the exposure control speed is slow.
Disclosure of Invention
The embodiment of the application provides an image processing chip, an electronic device and an image processing method, which can effectively improve the speed of exposure control in image acquisition.
In a first aspect, an embodiment of the present application provides an image processing chip, including:
the first image signal processor is used for acquiring an image of an image acquisition scene and acquiring exposure state information of the image acquisition scene according to the image;
and the first central processing unit is used for acquiring target data required by the application processing chip for calculating the exposure parameters according to the exposure state information.
In a second aspect, an embodiment of the present application provides an application processing chip, including:
and the second image signal processor is used for acquiring target data required by calculating exposure parameters from an image processing chip when the camera is started to acquire images, acquiring the exposure parameters according to the target data and acquiring images of image acquisition scenes according to the exposure parameters, wherein the image processing chip acquires images of the image acquisition scenes and acquires the target data according to the images.
In a third aspect, an embodiment of the present application provides an electronic device, including:
the image processing chip is used for acquiring an image of an image acquisition scene, acquiring target data required by the application processing chip for calculating exposure parameters according to the image, and transmitting the target data to the application processing chip;
and the application processing chip is used for calculating exposure parameters according to the target data and carrying out image acquisition on the image acquisition scene according to the exposure parameters during image acquisition.
In a fourth aspect, an embodiment of the present application provides an image processing method, including:
acquiring an image of an image acquisition scene;
acquiring exposure state information of the image acquisition scene according to the image;
when a camera is started to collect images, calculating exposure parameters of the camera according to the exposure state information;
and controlling the camera to acquire the image of the image acquisition scene according to the exposure parameters.
The embodiment of the application provides an image processing chip, an application processing chip, an electronic device and an image processing method.
The image processing chip in the embodiment of the application can acquire the exposure state information before image acquisition, calculate the target data required by the exposure parameters according to the exposure state information, and quickly calculate the exposure parameters according to the target data during image acquisition, thereby effectively improving the speed of exposure control in the image acquisition process.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a schematic structural diagram of an image processing chip according to an embodiment of the present application.
Fig. 2 is a schematic flow chart of a pipeline process provided in an embodiment of the present application.
Fig. 3 is a flowchart of data processing in an image processing chip according to an embodiment of the present application.
Fig. 4 is a calculation manner based on a region of interest according to an embodiment of the present application.
Fig. 5 is a schematic diagram of brightness change of a preview screen of an electronic device after a camera is turned on according to an embodiment of the present application.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Fig. 7 is another schematic structural diagram of an electronic device provided in an embodiment of the present application.
Fig. 8 is a schematic flowchart of the operation of an image processing chip in an electronic device according to an embodiment of the present application.
Fig. 9 is a schematic flowchart of the operation of an application processing chip in the electronic device according to the embodiment of the present application.
Fig. 10 is a schematic flowchart of an image processing method according to an embodiment of the present application.
Detailed Description
Referring to the drawings, wherein like reference numbers refer to like elements, the principles of the present application are illustrated as being implemented in a suitable computing environment. The following description is based on illustrated embodiments of the application and should not be taken as limiting the application with respect to other embodiments that are not detailed herein.
The embodiment of the application provides an image processing chip, and the image processing chip can be integrated on an electronic device to be realized. The electronic device may be a smart phone, a tablet computer, a Personal Digital Assistant (PDA), or the like. The following is a detailed explanation of the analysis.
For example, when an electronic device integrated with the image processing chip provided in the embodiment of the present application is used, the first image signal processor of the image processing chip may obtain an image of an image capturing scene, and obtain exposure state information of the image capturing scene according to the image, and then the first central processing unit of the image processing chip may obtain target data required by an application processing chip of the electronic device when calculating an exposure parameter according to the exposure state information, and transmit the target data to the application processing chip, so that when the camera is started, the application processing chip calculates the exposure parameter according to the target data, and controls the camera to capture an image of the image capturing scene according to the exposure parameter.
As shown in fig. 1, fig. 1 is a schematic block diagram of an image processing chip 100 provided in the embodiment of the present application, and the image processing chip 100 may include a first image signal processor 110, a first central processing unit 120, and a memory 130.
The first image signal processor 110 is configured to acquire an image of an image capturing scene and acquire exposure state information of the image capturing scene according to the image.
And a first central processing unit 120, configured to obtain, according to the exposure state information, target data required for calculating the exposure parameter by the application processing chip.
And a memory 130 for storing the target data. In some embodiments, the first central processor 120 stores the target data into the memory 130 according to the acquisition time of the target data, so that the application processing chip acquires the target data from the memory 130 according to the acquisition time. The memory 130 may be in various forms, such as a Double Data Rate (DDR SDRAM) SDRAM.
As shown in fig. 1, the image processing chip 100 further includes a first system bus 160, wherein the first image signal processor 110, the first central processor 120 and the memory 130 are respectively connected to the first system bus 160 to implement interaction between data.
For example, when the camera and the light sensor operate in a low power consumption mode (low resolution, low frame rate) in the background, real-time illumination information and exposure state information can be acquired. The first central processing unit can output the real-time information to a queue of the double-rate synchronous dynamic random access memory after acquiring the real-time information, a plurality of historical data and timestamp information can be stored in the queue, and the application processing chip can select to read the historical data according to the timestamp.
For another example, when the second camera needs to capture a High-Dynamic Range image (HDR image), the application processing chip may obtain a plurality of historical data from the queue according to the timestamp. For another example, when an image needs to be acquired quickly, multiple historical data can be acquired from the queue according to the time stamps to achieve quick automatic exposure convergence, so that the camera can acquire the image quickly.
As shown in fig. 1, the image processing chip 100 may further include a first interface 140, where the first interface 140 is configured to be connected to a camera and the first image signal processor 110, so that the first image signal processor 110 obtains an image of an image capturing scene from the camera. The first Interface 140 can have various forms, for example, the first Interface 140 can be a Mobile Industry Processor Interface (MIPI), and a specific Interface can be selected according to actual needs.
As shown in fig. 1, the image processing chip 100 may further include a second interface 150, where the second interface 150 is used for connecting with a light sensor to obtain illumination information of an image capturing scene. In some embodiments, the second interface 150 may be connected to the first central processor 120 through an I2C (Inter-Integrated Circuit) bus, so that the first central processor 120 obtains the illumination information.
In some embodiments, the first image signal processor 110 may be configured to:
processing the image to obtain a processed image; and carrying out state statistics on the processed image to obtain exposure state information.
In some embodiments, Raw image Data (Raw Data) of an image may be acquired from a camera, and then the Raw image Data of the image may be subjected to image processing.
The image processing may have a variety of representations, among others. For example, the image processing may be a pipeline processing flow (an integrated processing procedure, which refers to a flow of performing various image processing in order) in the image signal processing.
As shown in fig. 2, fig. 2 is a schematic flow chart of a pipeline process provided in the embodiment of the present application. The pipeline process may be an image processing process including processing steps such as Black Level Correction (BLC) in step 200, data linearization (Degamma) in step 201, Noise Reduction (NR) in step 202, Lens Correction (LSC) in step 203, White Balance (WB) in step 204, Demosaic (Demosaic) in step 205, Color Correction (CCM) in step 206, Gamma Correction (Gamma) in step 207, Sharpening (Sharpening) in step 208, Color Space Conversion (CSM) in step 209, Color Space Matrix in step 210, and image Noise Reduction/Sharpening (NR/YUV) in step 210, or may be an image processing process including a part of the processing steps listed above, and the processing steps may be specifically set according to actual requirements.
The Exposure state information may be automatic Exposure statistical information (AE Stats, Auto Exposure Stats) of the electronic device when shooting an image of a current image capture scene, and the automatic Exposure statistical information may include various information, for example, Bayer image Grid statistical information (BG Stats, Bayer Grid Stats), Histogram statistical information (Hist Stats, Histogram Stats), and the like.
The bayer image grid statistical information is obtained by dividing and counting Raw image Data (Raw Data) of an image according to a luminance value, a oversaturated pixel proportion, a red-green proportion and a blue-green proportion. For example, the brawdata is divided into 64 × 48 luminance values, the oversaturated pixel ratio, the red-green ratio and the blue-green ratio, and then the statistics is performed to obtain BG Stats.
The histogram statistical information is statistical information obtained from a histogram obtained by counting the number of different color level pixels in original image data of an image.
For example, as shown in fig. 2, after the first image signal processor 110 acquires original image data of an image from a camera, the original image data is processed through a pipeline process shown in fig. 2, so as to obtain exposure state information.
In some embodiments, as shown in fig. 3, fig. 3 is a data processing flow chart of the image processing chip 100 provided in the embodiments of the present application. The first image signal processor 110 acquires an image as shown in step 301, and then the first central processor 120 in the image processing chip 100 acquires illumination information of the current image capturing scene after the first image signal processor 110 acquires exposure state information according to the image as shown in steps 302 and 303, and then further calculates brightness information of the image capturing scene according to the illumination information and the exposure state information as shown in step 304a, and stores the brightness information as target data in the memory 130.
In some embodiments, the first central processor 120 may be configured to:
acquiring illumination information of an image acquisition scene; and calculating according to the illumination information and the exposure state information to obtain the brightness information of the image acquisition scene, and taking the brightness information as target data.
The illumination information may be ambient light information, or may be simply referred to as illumination. The illuminance is measured in units of "lux", abbreviated as "lux", and in unit symbols of "lx" or "lux". For example, the ambient light of the image capture scene may be obtained from a light sensor as 12000 lux.
In some embodiments, the first central processor 120 may be configured to:
calculating exposure state information through a preset weight matrix to obtain average brightness information of the image;
respectively carrying out normalization processing on the average brightness information and the illumination information to obtain corresponding normalized brightness information and normalized illumination information; and carrying out weighted calculation on the normalized brightness information and the normalized illumination information to obtain the brightness information of the image acquisition scene.
The exposure state information may be calculated by a preset weight matrix in various ways. For example, referring to fig. 4, fig. 4 illustrates a Region Of Interest (ROI) based calculation method provided in an embodiment Of the present application. The region of interest is an image region selected from the image, which is a major point of interest in image processing, such as a person, an object, and the like in the image. After the region of interest is determined, when the average brightness of the image is calculated based on the weight matrix, weighting calculation is carried out on each block in the region of interest to obtain the sum of the brightness of the image. Then, the average luminance of the image can be obtained by averaging the sum of the luminances.
In some embodiments, the first central processor 120 may be configured to:
determining a region of interest of an image; and calculating the exposure state information of the region of interest through a preset weight matrix to obtain the average brightness information of the image.
For example, the preset weight matrix is Score [64 × 48], each weight in the weight matrix is obtained in advance through a tuning process in the image processing chip, and the weight range is [0,1.0 ]. The calculation formula Of the image brightness based on the Region Of Interest (ROI) can be:
wherein, sumofroiliuma refers to the sum of the brightness of all blocks in the region of interest, i.e. the sum of the brightness of the image, ROICount refers to the number of blocks in the region of interest, roilium [ i ] refers to the ith block in the region of interest, and Score [ i ] refers to the ith weight in the preset weight matrix.
After the sum of the luminances of all the blocks in the region of interest is calculated, the average luminance of the image is obtained by taking an average value, and then the average luminance calculation formula of the image may be:
averageLuma＝SumOfROILuma/ROICount
wherein averageLuma refers to the average brightness of an image.
After the average brightness information and the illumination information of the acquired image are normalized, weighting calculation is performed again, and then the brightness information of the image acquisition scene can be obtained, and then the normalization processing formula of the average brightness information can be as follows:
SensorLuxIdx＝L 0 +log(averageLuma)
and the normalization processing formula of the illumination information can be as follows:
LightLuxIdx＝K 0 +log(Light_Lux)
where sensorlluxidx refers to normalized luminance information, LightLuxIdx refers to normalized illumination information, and Light _ Lux refers to illumination information obtained from the Light sensor.
After the normalized luminance information and the normalized illumination information are obtained, the luminance information of the image acquisition scene can be obtained by performing weighting calculation, and then the calculation formula of the luminance information may be as follows:
LightIntensityIdx＝SensorLuxIdx*(1-w)+LightLuxIdx*w
wherein, LightIntensityIdx refers to luminance information of an image capturing scene.
As can also be seen from the calculation formula of the luminance information, when w is 0, the luminance information of the image capturing scene can be calculated according to the luminance of the image; when w is 1, the brightness information of the image acquisition scene can be calculated according to illumination. Therefore, in some embodiments, the brightness information of the image capturing scene can be directly calculated when the exposure state information is acquired from the image, or the brightness information can be directly calculated when the illumination information is acquired from the light sensor.
The calculation is carried out based on the interested region, so that the whole image is prevented from being calculated, the calculation amount can be effectively reduced, the calculation aiming at the interested region can avoid the interference caused by the noise of the non-interested region, and the precision can be effectively improved.
It can be understood that any module including the first central processing unit can calculate the brightness information of the image capturing scene according to the illumination information and the exposure state information provided in the embodiment of the present application. That is, when the application processing chip includes a hardware structure with a function similar to that of the first central processing unit, as shown in step 304b in fig. 3, the illumination information and the exposure state information may be used as target data and then stored in the memory 130, and the application processing chip may obtain data from the memory 130 and calculate brightness information of the image capturing scene accordingly.
In some embodiments, the first central processor 120 is configured to:
acquiring illumination information of an image acquisition scene from a light sensor; the illumination information and the exposure state information are taken as target data.
It can be understood that after the image signal processor processes the image to obtain the exposure state information and the central processing unit obtains the illumination information from the light sensor, the exposure state information and the illumination information may be actually processed inside the image processing chip to obtain the brightness information of the image capturing scene. On the other hand, the information may be transmitted to the application processing chip, and the application processing chip may process the information to obtain the luminance information.
For example, when the first central processing unit acquires the illumination information and the exposure state information, it may be used as target data and stored in the memory according to the acquisition time of the data. The application processing chip can directly acquire the illumination information and the exposure state information, acquire the brightness information required by calculating the exposure parameter according to the illumination information and the exposure state information in the application processing chip, and then calculate the exposure parameter according to the brightness information.
For another example, in some embodiments, after the first central processing unit 120 acquires the exposure state information, the exposure state information may be stored in the memory 130 as target data according to the acquisition time sequence, and the application processing chip may directly acquire the exposure state information and the illumination information and directly calculate the brightness information according to the exposure state information. Then, the application processing chip of the electronic device can calculate the exposure parameter according to the brightness information.
The brightness information is calculated by the image processing chip, or the brightness information is calculated by the application processing chip, which can be selected according to the operation condition, power consumption condition, etc. of the electronic device. For example, when the electronic device has less power, since the power consumption of the image processing chip is lower than that of the application processing chip, in order to save power, the calculation may be performed by the image processing chip. For another example, when fewer applications are currently running on the electronic device, then the calculations may be performed by the application processing chip.
In some embodiments, the first central processor 120 may be further configured to:
acquiring illumination information of an image acquisition scene; calculating according to the illumination information and the exposure state information to obtain the brightness information of the image acquisition scene;
the first image signal processor 110 may be further configured to:
and calculating the exposure parameters according to the brightness information, and storing the exposure parameters into the memory, so that the application processing chip acquires the exposure parameters from the memory.
For example, when more applications are running in the application processing chip, the processing speed of the application processing chip side is affected by transmitting the brightness information to the application processing chip for processing. The exposure parameters may be further calculated from the brightness information in the central processor and stored in the memory. When the camera needs to be started for image acquisition, the exposure parameters are transmitted to the application processing chip from the memory, and then the camera can be directly controlled to acquire images according to the exposure parameters. Therefore, the exposure control time is shorter, and the image acquisition rate is effectively improved.
When a user uses an electronic device such as a smart phone and a tablet to shoot, as a certain time is needed in an exposure process to enable an output picture to reach normal brightness, a long time is needed to wait when a camera is opened to shoot. For example, as shown in fig. 5, a diagram of brightness change of a preview screen after a camera of a common electronic device is turned on is shown. It can be seen that the picture will gradually change from dark to bright, and the first few frames cannot be used as preview images due to being too dark. It can be understood that in the conventional exposure control technique, it takes a long time for the brightness of the output image to coincide with the actual scene.
The image processing chip in the embodiment of the application can acquire the exposure state information before image acquisition, calculate the target data required by the exposure parameters according to the exposure state information, and quickly calculate the exposure parameters according to the target data during image acquisition, thereby effectively improving the speed of exposure control in the image acquisition process.
When the embodiment of the application is applied to the electronic equipment, the image processing chip acquires target data required by calculating the exposure parameters, when the camera is started to acquire images, the application processing chip calculates the exposure parameters according to the target data and controls the camera to acquire the images according to the exposure parameters, and the speed of exposure control in image acquisition can be effectively improved.
In order to better implement the image processing chip provided by the embodiment of the present application, an application processing chip is also provided by the embodiment of the present application. The application processing chip includes a second image processor.
And the second image signal processor is used for acquiring target data required by calculating exposure parameters from an image processing chip when the camera is started to acquire images, acquiring the exposure parameters according to the target data and acquiring the images of the image acquisition scene according to the exposure parameters.
For example, when the camera is not started to capture an image, the image processing chip has already calculated the target data required for calculating the exposure parameters. When image acquisition is needed, the second image signal processor in the application processing chip can directly calculate according to the target data to obtain the exposure parameters, and the camera is controlled to acquire the image according to the exposure parameters. The target data may be brightness information of an image acquisition scene, and when the second image signal processor acquires the brightness information, a photometric algorithm may be performed according to the brightness information to obtain an exposure parameter.
Since different processors in the application processing chip are required to process different target data. Then, in some embodiments, when the target data is luminance information of the image capture scene, the second image signal processor may be configured to: and calculating according to the brightness information to obtain the exposure parameters.
In some embodiments, the application processing chip may further include a second central processor, and when the target data acquired from the image processing chip is illumination information and exposure state information of the image capturing scene, the second central processor is configured to:
acquiring the exposure state information and the illumination information; calculating according to the exposure state information and the illumination information to obtain brightness information of the image acquisition scene, and taking the brightness information as the target data; transmitting the target data to the second image signal processor.
For example, the target data obtained in the image processing chip may be brightness information or illumination information and exposure state information of an image capture scene, and if the exposure parameters are obtained according to the illumination information and the exposure state information, the second central processing unit in the application processing chip is required to process the target data to obtain the brightness information, and then the second image signal processor calculates according to the brightness information to obtain the exposure parameters and controls the camera to capture the image according to the exposure parameters.
As can be seen from the above, the application processing chip provided in the embodiment of the present application can quickly calculate the exposure parameters according to the obtained target data, thereby reducing the time for calculating the exposure parameters in the exposure control and effectively improving the efficiency of the exposure control.
In order to better implement the image processing chip provided in the embodiment of the present application, an electronic device is further provided in the embodiment of the present application, and as shown in fig. 6, the electronic device 10 includes a camera, a light sensor, an image processing chip 100, and an application processing chip 200. The image processing chip 100 includes a first image signal processor 110, a first central processing unit 120, a memory 130, a first interface 140, a second interface 150, a first system bus 160, and a third interface 170. The application processing chip 200 includes a second image signal processor 210, a second central processor 220, a second system bus 230, and a fourth interface 240.
As shown in fig. 6, the first image signal processor 110 acquires an image of an image capturing scene from a camera through the first interface 140, and calculates exposure state information. And the first central processor 120 acquires illumination information of the image capturing scene from the light sensor through the second interface 150. The first video signal processor 110, the first central processing unit 120 and the memory 130 are respectively connected to the first system bus 160. The first cpu 120 may calculate brightness information of the image capturing scene according to the illumination information and the exposure status information, and store the brightness information in the memory 130 through the first system bus 160. Alternatively, the light irradiation information and the exposure state information may be stored in the memory 130 without being processed.
Then, the third interface 170 in the image processing chip 100 is connected to the fourth interface 240 in the application processing chip, so that the image processing chip 100 and the application processing chip 200 implement data intercommunication through the third interface 170 and the fourth interface 240.
The second image signal processor 210 in the application processing chip 200 obtains data required for calculating the exposure parameters from the memory 130 through the second system bus 230, the fourth interface 240, the third interface 170 and the first system bus 160. When the brightness information is stored in the memory 130, the second image signal processor 210 calculates an exposure parameter according to the brightness information. Or, when the illumination information and the exposure state information are stored in the memory 130, the second central processor 220 performs calculation according to the illumination information and the exposure state information to obtain the brightness information, and then the second image signal processor 210 performs calculation according to the brightness information to obtain the exposure parameter. When the exposure parameters are acquired, the application processing chip 200 controls the camera to acquire images according to the exposure parameters.
Wherein, there are multiple arrangement mode in the position of light sensor in this application. For example, a light sensor may be disposed within the electronic device 10 as shown in FIG. 6. For another example, the light sensor may be disposed in the application processing chip 200, as shown in fig. 7, fig. 7 is another schematic structural diagram of the electronic device provided in the embodiment of the present application. The image processing chip 100 further includes a fifth interface 180, and the light sensor disposed in the application processing chip 200 is connected to the image processing chip 100 through the fifth interface 180, so as to transmit the illumination information of the image capturing scene to the first central processing unit 120 through the first system bus 160.
In order to better implement the image processing chip provided by the embodiment of the present application, an embodiment of the present application further provides an electronic device, where the electronic device includes a first camera, a second camera, a light sensor, an image processing chip, and an application processing chip. The electronic device may be a smart phone, a tablet computer, a Personal Digital Assistant (PDA), or the like.
For example, as shown in fig. 8, fig. 8 is a schematic flowchart of the operation of the image processing chip. When the electronic device is in a working state, the application processing chip can power on the image processing chip and send a command, so that the image processing chip turns on the first camera and the light sensor, and the application processing chip controls the image processing chip to set the frame rate and the resolution when the first camera collects images, so that the first camera and the light sensor operate in a low-power-consumption operation mode (low frame rate and low resolution) at the background. As shown in step 401, the first camera acquires an image in an image acquisition scene and outputs original image data of the image, the first image signal processor of the image processing chip acquires the original image data, then as shown in step 402, the first image signal processor performs pipeline processing on the original image data and performs statistics to obtain exposure state information, as shown in step 403, the light sensor outputs illumination information of the image acquisition scene, and then the first central processing unit acquires the illumination information.
In step 404, it is determined whether the image processing chip performs the operation of calculating the brightness information, and when the image processing chip needs to perform the calculation, in step 405a, the exposure state information and the illumination information acquired by the light sensor are input into the first central processing unit of the image processing chip, and the brightness information of the scene is output using a light metering algorithm. When the image processing chip is not needed for calculation, in step 405b, the original exposure state information and illumination information are stored in the memory. Then, as shown in step 406, after the brightness information of the scene, or the original exposure state information and the illumination information are output to the queue of the memory, a plurality of historical data and time stamp information thereof are stored in the queue, and the application processing chip can select the data to be read according to the time stamp.
As shown in fig. 9, fig. 9 is a schematic flowchart of the operation of the application processing chip. As shown in step 501, when the application processing chip turns on the second camera, the application processing chip performs calculation according to the information provided by the image processing chip. In step 502, determining how to perform calculation according to the type of the provided information, and when the image processing chip provides the brightness information of the scene, as shown in step 503a, directly reading the brightness information for use; when the image processing chip provides the original exposure state information and illumination information, then scene brightness information is calculated from the data, as shown in step 503 b. When the brightness information of the scene is obtained through calculation, as shown in step 504, the exposure parameter (i.e., shutter time, gain) can be calculated by querying the exposure table according to the brightness information, so that the second camera can rapidly converge when acquiring the image.
In order to better implement the image processing chip provided in the embodiment of the present application, an embodiment of the present application provides an electronic device, where the electronic device includes:
the image processing chip is used for acquiring an image of an image acquisition scene, acquiring target data required by the application processing chip for calculating exposure parameters according to the image, and transmitting the target data to the application processing chip;
and the application processing chip is used for calculating exposure parameters according to the target data and carrying out image acquisition on the image acquisition scene according to the exposure parameters during image acquisition.
In some embodiments, the image processing chip is to: performing image processing on the image to obtain a processed image; and carrying out state statistics on the processed image to obtain the exposure state information.
In some embodiments, the image processing chip is to: acquiring illumination information of the image acquisition scene; and calculating according to the illumination information and the exposure state information to obtain brightness information of the image acquisition scene, and taking the brightness information as the target data. Reference may be made to the above embodiments, which are not described herein again.
In some embodiments, the electronic device further includes a first camera and a second camera, where the first camera acquires an image of the image capture scene, so that the image processing chip acquires the target data according to the image; and the second camera is used for carrying out image acquisition on the image acquisition scene according to the exposure parameters during image acquisition.
For common electronic equipment, the brightness of the acquired environment cannot be determined when a camera is just opened, exposure parameters need to be continuously adjusted, and meanwhile, the brightness of the environment can be gradually calculated according to the brightness of an output picture, so that automatic exposure convergence is achieved. Due to the influence of the automatic exposure convergence process, as shown in fig. 5, the brightness of the first few frames of images collected by the camera changes from dark to bright, so that the user feels that the opening speed of the camera is slow.
For example, the first camera may operate in a low power consumption mode at the background, acquire an image with a low resolution so that the image processing chip acquires the exposure state information, and when the second camera is turned on, the application processing chip may obtain the exposure parameters according to the exposure state information and control the second camera to shoot the image with the high resolution, thereby reducing the waiting time for exposure convergence. When the exposure state information is acquired, the first camera is controlled to operate in a low power consumption mode, and a low-resolution image is acquired, so that the power consumption of the electronic equipment can be effectively reduced, and when the second camera is controlled to acquire the high-resolution image, the use requirements of a user can be effectively met.
Then in the electronic device, the first camera performs image acquisition with a first resolution, and the second camera performs image acquisition with a second resolution, wherein the first resolution is lower than the second resolution.
In some embodiments, the first camera operates at a first power consumption and the second camera operates at a second power consumption, wherein the first power consumption is lower than the second power consumption.
In some embodiments, the electronic device may further include: display, radio frequency circuit, audio circuit and power. The display, the radio frequency circuit, the audio circuit and the power supply are electrically connected with the application processing chip and the image processing chip respectively.
The display may be used to display information input by or provided to the user as well as various graphical user interfaces, which may be made up of graphics, text, icons, video, and any combination thereof. The Display may include a Display panel, and in some embodiments, the Display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.
The radio frequency circuit can be used for transceiving radio frequency signals so as to establish wireless communication with a network device or other electronic devices through wireless communication and to transceive signals with the network device or other electronic devices.
The audio circuit may be used to provide an audio interface between a user and an electronic device through a speaker, microphone.
The power supply may be used to power various components of the electronic device. In some embodiments, the power supply may be logically connected to the processor through a power management system, such that the power management system may manage charging, discharging, and power consumption management functions.
Although not shown, the electronic device may further include a camera, a bluetooth module, and the like, which are not described in detail herein.
The electronic equipment provided in the embodiment of the application comprises a first camera, a second camera, an image processing chip and an application processing chip. The image processing chip is used for acquiring an image of an image acquisition scene, acquiring target data required by the application processing chip for calculating exposure parameters according to the image, and transmitting the target data to the application processing chip; the application processing chip is used for calculating exposure parameters according to the target data during image acquisition and acquiring images of the image acquisition scene according to the exposure parameters; the first camera acquires an image of the image acquisition scene, so that the image processing chip acquires the target data according to the image; and the second camera is used for carrying out image acquisition on the image acquisition scene according to the exposure parameters during image acquisition.
According to the electronic equipment provided by the embodiment of the application, before the second camera shoots, the low-resolution image of the image acquisition scene is acquired from the first camera in a low-power-consumption mode through the image processing chip, the target data required by the exposure parameter is calculated according to the image acquisition application processing chip, when the second camera is started to shoot, the exposure parameter is calculated according to the target data, and the second camera is controlled to acquire the image of the image acquisition scene in a high-resolution mode according to the exposure parameter, so that on one hand, the time required by exposure control can be effectively reduced, the speed of the exposure control in image acquisition is improved, on the other hand, the power consumption for calculating the exposure parameter can be effectively reduced, and the service time of the electronic equipment is prolonged.
In order to better implement the image processing chip provided by the embodiment of the present application, the embodiment of the present application further provides another electronic device.
In order to better implement the electronic device provided by the embodiment of the present application, an embodiment of the present application further provides an image processing method.
Referring to fig. 10, fig. 10 is a schematic flowchart of an image processing method according to an embodiment of the present disclosure, where the image processing method includes:
601. an image of an image capture scene is acquired.
When the image is obtained, one camera can be controlled to operate in a low power consumption mode at the background to obtain the image. For example, when the electronic device is in an operating state, the camera continuously operates in the background, and when the power of the electronic device is low, the first camera is turned off.
602. And acquiring exposure state information of the image acquisition scene according to the image.
The method for acquiring the exposure state information of the image acquisition scene according to the image can comprise the following steps:
performing image processing on the image to obtain a processed image; and carrying out state statistics on the processed image to obtain exposure state information.
The image processing may have a variety of representations, among others. For example, the image processing may be a pipeline processing flow (an integrated processing procedure, which refers to a flow of performing various image processing in order) in the image signal processing.
The Exposure state information may be automatic Exposure statistical information (AE Stats, Auto Exposure Stats) of the electronic device when shooting an image of a current image capture scene, and the automatic Exposure statistical information may include Bayer image Grid statistical information (BG Stats, Bayer Grid Stats), Histogram statistical information (Hist Stats, Histogram Stats), and the like.
Accordingly, according to the difference of the statistical information, there are various ways of performing state statistics on the processed image. For example, the bayer image grid statistical information may be obtained by dividing the raw image data according to the luminance value, the oversaturated pixel proportion, the red-green proportion, and the blue-green proportion. For another example, the histogram statistical information may be obtained by performing statistics according to the number of pixels of different color levels in the original image data of the image.
603. And when the camera is started to acquire images, calculating exposure parameters of the camera according to the exposure state information.
In some embodiments, the method for calculating the exposure parameter of the camera according to the exposure state information may include:
acquiring illumination information of an image acquisition scene; calculating according to the illumination information and the exposure state information to obtain the brightness information of the image acquisition scene; and obtaining the exposure parameters of the camera according to the brightness information.
For example, when a user needs to acquire an image, a main camera of the electronic device needs to be started to acquire a high-resolution image, before the main camera is started, illumination information can be acquired through a light sensor, then brightness information is obtained through calculation according to the illumination information and exposure state information, exposure parameters of the camera are obtained according to the brightness information, and the camera is controlled to shoot the image according to the exposure parameters.
After the illumination information and the exposure state information are obtained, the exposure state information can be calculated through a preset weight matrix to obtain the average brightness of the image. The brightness information of the image acquisition scene can be obtained by normalizing the average brightness information and the illumination information and then performing weighted calculation.
In some embodiments, the method of obtaining the brightness information of the image capturing scene by performing the calculation according to the illumination information and the exposure state information includes:
calculating exposure state information through a preset weight matrix to obtain average brightness information of the image;
respectively carrying out normalization processing on the average brightness information and the illumination information to obtain corresponding normalized brightness information and normalized illumination information; and performing weighted calculation on the normalized brightness information and the normalized illumination information to obtain the brightness information of the image acquisition scene.
The exposure state information may be calculated by a preset weight matrix in various ways. For example, referring to fig. 4, fig. 4 illustrates a Region Of Interest (ROI) based calculation method provided in an embodiment Of the present application. The region of interest is an image region selected from the image, which is a major point of interest in image processing, such as a person, an object, and the like in the image. After the region of interest is determined, when the average brightness of the image is calculated based on the weight matrix, weighting calculation is carried out on each block in the region of interest to obtain the sum of the brightness of the image. Then, the average luminance of the image can be obtained by averaging the sum of the luminances.
In some embodiments, the method for calculating the exposure state information through the preset weight matrix to obtain the average brightness information Of the image may include:
determining a region of interest and a region of non-interest of the image; calculating exposure state information of the region of interest through a preset weight matrix, and taking a calculation result as total brightness information of the image; and obtaining average brightness information of the image according to the total brightness information.
For example, the preset weight matrix is Score [64 × 48], each weight in the weight matrix is obtained in advance through a tuning process in the image processing chip, and the weight range is [0,1.0 ]. The calculation formula Of the image brightness based on the Region Of Interest (ROI) can be:
wherein, sumofroiliuma refers to the sum of the brightness of all blocks in the region of interest, i.e. the sum of the brightness of the image, ROICount refers to the number of blocks in the region of interest, roilium [ i ] refers to the ith block in the region of interest, and Score [ i ] refers to the ith weight in the preset weight matrix.
After the sum of the luminances of all the blocks in the region of interest is calculated, the average luminance of the image is obtained by taking an average value, and then the average luminance calculation formula of the image may be:
averageLuma＝SumOfROILuma/ROICount
wherein averageLuma refers to the average brightness of an image.
After the average brightness information and the illumination information of the acquired image are normalized, weighting calculation is performed again, and then the brightness information of the image acquisition scene can be obtained, and then the normalization processing formula of the average brightness information can be as follows:
SensorLuxIdx＝L 0 +log(averageLuma)
and the normalization processing formula of the illumination information can be as follows:
LightLuxIdx＝K 0 +log(Light_Lux)
where sensorlluxidx refers to normalized luminance information, LightLuxIdx refers to normalized illumination information, and Light _ Lux refers to illumination information obtained from the Light sensor.
After the normalized luminance information and the normalized illumination information are obtained, the luminance information of the image acquisition scene can be obtained by performing weighting calculation, and then the calculation formula of the luminance information may be as follows:
LightIntensityIdx＝SensorLuxIdx*(1-w)+LightLuxIdx*w
wherein, LightIntensityIdx refers to luminance information of an image capturing scene.
There are various ways to obtain the exposure parameter of the second camera according to the brightness information. For example, an exposure table may be obtained, and the exposure table includes the correspondence between the brightness, shutter, and gain of the scene. When the brightness of the current image acquisition scene is acquired, the shutter and the gain can be inquired according to the exposure table, and the second camera is controlled to acquire the image by the shutter and the gain, so that the output image reaches normal brightness.
604. And controlling the camera to acquire the image of the image acquisition scene according to the exposure parameters.
For example, after the exposure parameters (i.e. shutter and gain) are looked up through the exposure table, the second camera can be controlled to capture an image with the shutter and gain, so that the output image reaches normal brightness.
Therefore, the image processing method provided by the embodiment of the application obtains the image of the image acquisition scene according to the image, obtains the exposure state information of the image acquisition scene according to the image, calculates the exposure parameter of the camera according to the exposure state information when the camera is started to acquire the image, and controls the camera to acquire the image of the image acquisition scene according to the exposure parameter.
According to the embodiment of the application, before image acquisition, the image of an image acquisition scene can be acquired in advance, the exposure state information required by calculating the exposure parameters is acquired, when the camera is started to acquire the image, the exposure parameters are calculated according to the exposure state information, and the camera is controlled to acquire the image according to the exposure parameters. When the image processing method provided by the embodiment of the application is applied to the electronic equipment, the speed of exposure control in image acquisition can be effectively improved.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It should be noted that, for the image processing method of the embodiment of the present application, it can be understood by a person skilled in the art that all or part of the process of implementing the image processing method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, where the computer program can be stored in a computer-readable or computer-readable storage medium, such as a memory of an electronic device, and executed by at least one processor in the electronic device, and during the execution process, the process of the embodiment of the image processing method can be included. The computer readable storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.
For the image processing chip of the embodiment of the present application, each functional unit may be integrated into one processing chip, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium such as a read-only memory, a magnetic or optical disk, or the like.
The foregoing describes an image processing chip, an electronic device, and an image processing method provided in an embodiment of the present application in detail, and a specific example is applied to illustrate the principle and implementation of the present application, and the description of the foregoing embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (14)
1. An image processing chip, comprising:
the first image signal processor is used for acquiring an image of an image acquisition scene and acquiring exposure state information of the image acquisition scene according to the image;
and the first central processing unit is used for acquiring target data required by the application processing chip for calculating the exposure parameters according to the exposure state information.
2. The image processing chip of claim 1, wherein the first central processing unit is configured to:
acquiring illumination information of the image acquisition scene;
and calculating according to the illumination information and the exposure state information to obtain the brightness information of the image acquisition scene, and taking the brightness information as the target data.
3. The image processing chip of claim 2, wherein the first central processor is configured to:
calculating the exposure state information through a preset weight matrix to obtain average brightness information of the image;
respectively carrying out normalization processing on the average brightness information and the illumination information to obtain corresponding normalized brightness information and normalized illumination information;
and performing weighted calculation on the normalized brightness information and the normalized illumination information to obtain the brightness information of the image acquisition scene.
4. The image processing chip of claim 3, wherein the first central processor is configured to:
determining a region of interest of the image;
and calculating the exposure state information of the region of interest through a preset weight matrix to obtain the average brightness information of the image.
5. The image processing chip of claim 1, wherein the first central processor is further configured to:
acquiring illumination information of the image acquisition scene;
and taking the illumination information and the exposure state information as the target data.
6. The image processing chip according to any one of claims 1 to 5, wherein the image processing chip further comprises a memory, and the first central processor is further configured to:
and storing the target data into the memory according to the acquisition time of the target data, so that the application processing chip acquires the target data from the memory according to the acquisition time.
7. An application processing chip, comprising:
and the second image signal processor is used for acquiring target data required by calculating exposure parameters from the image processing chip when the camera is started to acquire images, acquiring the exposure parameters according to the target data and acquiring the images of the image acquisition scene according to the exposure parameters.
8. The application processing chip of claim 7, wherein when the target data is luminance information of the image capture scene, the second image signal processor is configured to:
and calculating the exposure parameters according to the brightness information.
9. The application processing chip of claim 7, further comprising a second central processor, wherein when the target data is exposure state information and illumination information of the image capture scene, the second central processor is configured to:
calculating according to the exposure state information and the illumination information to obtain brightness information of the image acquisition scene;
the second image signal processor is configured to: and obtaining the exposure parameters according to the brightness information.
10. An electronic device, comprising:
the image processing chip of any one of claims 1 to 6; and
an application processing chip as claimed in any one of claims 7 to 9.
11. The electronic device according to claim 10, further comprising a first camera and a second camera, wherein the first camera is configured to acquire an image of an image capturing scene, so that the image processing chip acquires target data required for calculating exposure parameters according to the image, and the second camera is configured to capture the image of the image capturing scene according to the exposure parameters when capturing the image.
12. The electronic device of claim 11, wherein the first camera operates with a first power consumption and the second camera operates with a second power consumption, and wherein the first power consumption is lower than the second power consumption.
13. An image processing method, comprising:
acquiring an image of an image acquisition scene;
acquiring exposure state information of the image acquisition scene according to the image;
when a camera is started to acquire images, calculating exposure parameters of the camera according to the exposure state information;
and controlling the camera to acquire the image of the image acquisition scene according to the exposure parameters.
14. The method according to claim 13, wherein the calculating the exposure parameter of the camera according to the exposure status information comprises:
acquiring illumination information of the image acquisition scene;
calculating according to the illumination information and the exposure state information to obtain brightness information of the image acquisition scene;
and obtaining the exposure parameters of the camera according to the brightness information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110304163.9A CN115118886A (en) | 2021-03-22 | 2021-03-22 | Image processing chip, application processing chip, electronic device and image processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110304163.9A CN115118886A (en) | 2021-03-22 | 2021-03-22 | Image processing chip, application processing chip, electronic device and image processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115118886A true CN115118886A (en) | 2022-09-27 |
Family
ID=83324421
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110304163.9A Pending CN115118886A (en) | 2021-03-22 | 2021-03-22 | Image processing chip, application processing chip, electronic device and image processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115118886A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070052839A1 (en) * | 2005-09-08 | 2007-03-08 | Hongzhi Kong | Method of exposure control for an imaging system |
| US20130208138A1 (en) * | 2012-02-09 | 2013-08-15 | Aptina Imaging Corporation | Imaging systems and methods for generating auto-exposed high-dynamic-range images |
| CN108681182A (en) * | 2018-04-24 | 2018-10-19 | 浙江大华技术股份有限公司 | A kind of diaphragm control method and terminal device |
| CN109688335A (en) * | 2018-12-04 | 2019-04-26 | 珠海格力电器股份有限公司 | Control method and device, the unlocking method and a device of terminal, mobile phone of camera |
| CN110225248A (en) * | 2019-05-29 | 2019-09-10 | Oppo广东移动通信有限公司 | Image-pickup method and device, electronic equipment, computer readable storage medium |
| CN110830728A (en) * | 2018-08-13 | 2020-02-21 | 浙江宇视科技有限公司 | Exposure adjusting method and device |
| CN111586312A (en) * | 2020-05-14 | 2020-08-25 | Oppo（重庆）智能科技有限公司 | Automatic exposure control method and device, terminal and storage medium |
-
2021
- 2021-03-22 CN CN202110304163.9A patent/CN115118886A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070052839A1 (en) * | 2005-09-08 | 2007-03-08 | Hongzhi Kong | Method of exposure control for an imaging system |
| US20130208138A1 (en) * | 2012-02-09 | 2013-08-15 | Aptina Imaging Corporation | Imaging systems and methods for generating auto-exposed high-dynamic-range images |
| CN108681182A (en) * | 2018-04-24 | 2018-10-19 | 浙江大华技术股份有限公司 | A kind of diaphragm control method and terminal device |
| CN110830728A (en) * | 2018-08-13 | 2020-02-21 | 浙江宇视科技有限公司 | Exposure adjusting method and device |
| CN109688335A (en) * | 2018-12-04 | 2019-04-26 | 珠海格力电器股份有限公司 | Control method and device, the unlocking method and a device of terminal, mobile phone of camera |
| CN110225248A (en) * | 2019-05-29 | 2019-09-10 | Oppo广东移动通信有限公司 | Image-pickup method and device, electronic equipment, computer readable storage medium |
| CN111586312A (en) * | 2020-05-14 | 2020-08-25 | Oppo（重庆）智能科技有限公司 | Automatic exposure control method and device, terminal and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111418201B (en) | Shooting method and equipment | |
| US11089207B2 (en) | Imaging processing method and apparatus for camera module in night scene, electronic device and storage medium | |
| CN108322646B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| US8994845B2 (en) | System and method of adjusting a camera based on image data | |
| US11532076B2 (en) | Image processing method, electronic device and storage medium | |
| US9571743B2 (en) | Dynamic exposure adjusting method and electronic apparatus using the same | |
| CN110445989B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN104883504B (en) | Open the method and device of high dynamic range HDR functions on intelligent terminal | |
| KR20150099302A (en) | Electronic device and control method of the same | |
| CN112073645B (en) | Exposure control method, device, terminal equipment and storage medium | |
| CN110266954B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN104917973B (en) | Dynamic exposure method of adjustment and its electronic device | |
| US9148582B2 (en) | Method and system for perfect shot imaging from multiple images | |
| CN116055890A (en) | Method and electronic device for generating high dynamic range video | |
| CN102469248B (en) | Device for filming image and image synthesis method thereof | |
| US8350930B2 (en) | Image capturing device and image synthesis method thereof | |
| CN110572585B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| WO2023226612A1 (en) | Exposure parameter determining method and apparatus | |
| CN110278386B (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN114222072B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN107454340B (en) | Image synthesis method and device based on high dynamic range principle and mobile terminal | |
| CN113810622B (en) | Image processing method and device | |
| CN114463191B (en) | Image processing method and electronic equipment | |
| CN115118886A (en) | Image processing chip, application processing chip, electronic device and image processing method | |
| EP2658245A1 (en) | System and method of adjusting camera image data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |