CN113906728A - Image processing method and device, camera module and movable equipment - Google Patents

Image processing method and device, camera module and movable equipment Download PDF

Info

Publication number
CN113906728A
CN113906728A CN202080041502.2A CN202080041502A CN113906728A CN 113906728 A CN113906728 A CN 113906728A CN 202080041502 A CN202080041502 A CN 202080041502A CN 113906728 A CN113906728 A CN 113906728A
Authority
CN
China
Prior art keywords
image data
processing
sending
target image
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202080041502.2A
Other languages
Chinese (zh)
Inventor
麻军平
余良
罗飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SZ DJI Technology Co Ltd
Original Assignee
SZ DJI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SZ DJI Technology Co Ltd filed Critical SZ DJI Technology Co Ltd
Publication of CN113906728A publication Critical patent/CN113906728A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules

Abstract

An image processing method, an image processing device, a camera module and a mobile device are provided, wherein the method comprises the following steps: acquiring image data (101) captured by each camera; transmitting target image data selected from the image data to an image processor for processing (102) in each transmission cycle; wherein the data amount of the target image data is less than or equal to the maximum data processing amount of the image processor. According to the method and the device, under the condition that image data are more, the instantaneous data processing pressure of the image processor is reduced by reducing the data transmission amount of each transmission period, and the design complexity and the cost of the multi-camera system are also reduced. In addition, because each sending period consumes short time and a plurality of sending periods are closely adjacent, the total time consumed by all the sending periods is also short, and the smoothness of the pictures shot by the camera is also ensured.

Description

Image processing method and device, camera module and movable equipment Technical Field
The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, a camera module, and a mobile device.
Background
Multi-camera systems are widely used in the fields of surveying and mapping, automatic navigation, etc. to achieve the progress of computer vision technology through multi-view shooting by multiple cameras.
In the related art, a plurality of cameras of a multi-camera system can be exposed at the same time, and simultaneously output images to an image processing system for processing through an image sensor of the cameras, so that the purpose of displaying the processed results on a display screen is finally achieved.
However, in the current scheme, the multiple cameras simultaneously expose and output image data, which results in a large amount of instantaneous data, resulting in a high requirement on the processing capability of the image processing system, and increasing the design complexity and cost of the multiple-camera system.
Disclosure of Invention
The application provides an image processing method, an image processing device, a camera module and a mobile device, which can solve the problem that in the prior art, a plurality of cameras simultaneously expose and output image data, so that the instantaneous data volume is very large, and the requirement on the processing capacity of an image processing system is high.
In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a camera module including multiple cameras, and includes:
acquiring image data shot by each camera;
in each sending period, sending target image data selected from the image data to an image processor for processing;
wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
In a second aspect, an embodiment of the present application provides an image processing apparatus applied to an image pickup module including a plurality of cameras, including: an acquisition module and a processor;
the acquisition module is configured to perform: acquiring image data shot by each camera;
the processor is configured to perform:
in each sending period, sending target image data selected from the image data to an image processor for processing;
wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
In a third aspect, an embodiment of the present application provides a camera module, including the image processing apparatus and a plurality of cameras, where the cameras include image sensors;
the camera collects image data through the image sensor;
and the image processing device receives and processes the image data shot by the camera.
In a fourth aspect, an embodiment of the present application provides a mobile device, which includes the camera module, a controller and a power system, where the controller controls a power output of the power system.
In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of the above aspect.
In a sixth aspect, the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of the above aspect.
In the embodiment of the application, the image data shot by each camera is acquired; in each sending period, sending target image data selected from the image data to an image processor for processing; wherein the data amount of the target image data is less than or equal to the maximum data processing amount of the image processor. According to the method and the device, the target image data with the data volume less than or equal to the maximum data processing volume can be sent to the image processor in each sending period, and all the image data are sent to the image processor through a plurality of sending periods under the condition that the image data are more, so that the instantaneous data processing pressure of the image processor is reduced, and the design complexity and the cost of the multi-camera system are also reduced. In addition, because each sending period consumes short time and a plurality of sending periods are closely adjacent, the total time consumed by all the sending periods is also short, and the smoothness of the pictures shot by the camera is also ensured.
Drawings
Fig. 1 is a system architecture diagram corresponding to an image processing method provided in an embodiment of the present application;
fig. 2 is a flowchart of an image processing method provided in an embodiment of the present application;
fig. 3 is a detailed flowchart of an image processing method according to an embodiment of the present application;
FIG. 4 is a detailed flowchart of another image processing method provided in the embodiments of the present application;
fig. 5 is a block diagram of an image processing apparatus according to an embodiment of the present application;
fig. 6 is a block diagram of a camera module according to an embodiment of the present disclosure;
fig. 7 is a block diagram of a removable device according to an embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
In this embodiment of the present application, referring to fig. 1, a system architecture diagram corresponding to an image processing method provided in this embodiment of the present application is shown, including: the camera group 10, the image processing device 20, the display device 30 and the storage device 40, the camera group 10 may include a plurality of cameras. The image processing apparatus 20 includes: a selection processor 21, an image processor 22 and a memory 25, wherein the memory 25 can be connected to the storage device 40 through a storage interface 24, and the memory 25 can also be connected to the display device 30 through a display interface 23.
The camera group 10 may collect image data through an image sensor (not shown in fig. 1) of the camera, and transmit the image data to the selection processor 21 through respective corresponding interface circuits for processing.
The image processor 22 is configured to process the received image data and store the image processing result in the memory 25, where the display device 30 may obtain the image processing result from the memory 25 through the display interface 23 for displaying, and the storage device 40 may obtain the image processing result from the memory 25 through the storage interface 24 for storing.
In the embodiment of the present application, in order to reduce the instantaneous data amount received by the image processor 22 and avoid the simultaneous exposure of the cameras to output image data to the image processor 22 simultaneously on the basis of ensuring that the display device 30 displays images normally, a selection strategy can be adopted by the selection processor 21, in each transmission cycle, only target image data with the data amount less than or equal to the maximum data processing amount of the image processor 22 is selected from all image data, the target image data is transmitted to the image processor 22 for processing, and other unselected image data is ignored in the current transmission cycle, so that the image processor 22 can receive the target image data within the processing capability range of itself for processing in each transmission cycle.
Under the condition of more image data, the instantaneous data processing pressure of the image processor 22 is reduced by a processing mode of a plurality of sending periods, wherein each sending period is short in time consumption, the sending periods are closely adjacent, so that the total time consumed by all the sending periods is short, the image processing result output by the image processor 22 can be displayed in the display device 30 in a near real-time effect, and the real-time display of the picture shot by each camera by the display device 30 is ensured. In addition, the data volume of the image processing result obtained by processing in each sending period is small, the storage speed in the memory 25 is high, and the memory bandwidth occupation of the memory 25 is reduced.
For example, in fig. 1, assuming that n is 5, wherein the instantaneous processing capability of the image processor 22 is to process image data captured by at most one camera, 5 image data captured by 5 cameras may be transmitted through 5 transmission cycles according to the round-robin selection strategy:
in the first sending period, only the image data 1 shot by the camera 1 is selected to be sent, and the image data shot by the cameras 2-5 is ignored;
in the second sending period, only the image data 2 shot by the camera 2 is selected to be sent, and the image data shot by the camera 1 and the cameras 3-5 are ignored;
in the third sending period, only the image data 3 shot by the camera 3 is selected to be sent, and the image data shot by the cameras 1-2 and 4-5 are ignored;
in the fourth sending period, only the image data 4 shot by the camera 4 is selected to be sent, and the image data shot by the cameras 1-3 and the camera 5 are ignored;
in the fifth sending period, only the image data 5 shot by the camera 5 is selected for sending, and the image data shot by the cameras 1-4 is ignored.
Fig. 2 is a flowchart of an image processing method provided in an embodiment of the present application, where the method is applied to a camera module including multiple cameras, and as shown in fig. 2, the method may include:
101. and acquiring image data shot by each camera.
In the fields of surveying and mapping, navigation, and the like, where computer vision calculation is applied, a camera may photograph a target point in real time and acquire image data through an image sensor, referring to fig. 1, the image sensor of the camera may transmit the image data acquired by the camera to a selection processor 21 through an interface circuit corresponding to the camera for processing.
102. And in each sending period, sending the target image data selected from the image data to an image processor for processing.
Wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
In practical applications, the image processor often has an index of the maximum data processing amount of the instantaneous data, for example, if the image processor is supposed to process only one path of data in real time, that is, the image processor can process only the picture shot by one camera in real time, the maximum data processing amount may be the maximum value of the data amount of the image data that can be shot by one camera in the instantaneous state.
In an implementation manner of the embodiment of the application, if the camera module includes a plurality of cameras, and the total amount of image data instantaneously captured by the plurality of cameras is far greater than the maximum data processing amount of the image processor for sequential data, a selection strategy may be adopted, in each sending period, only target image data with a data amount less than or equal to the maximum data processing amount of the image processor is selected from all image data, the target image data is sent to the image processor for processing, and other unselected image data is ignored in the current sending period, so that the image processor can receive the target image data within its own processing capability range in each sending period for processing. When the image data is more, the instantaneous data processing pressure of the image processor is reduced by a processing mode of a plurality of sending periods.
It should be noted that the selection strategy of each sending cycle may adopt the round-robin selection strategy illustrated in fig. 1, or may adopt other selection strategies, such as an important image data selection strategy, specifically including that, in each sending cycle, at least important image data captured by an important camera is selected for sending, so as to ensure smooth display of an important picture.
Optionally, the unit of the data amount of the target image data is a frame, and the unit of the maximum data processing amount of the image processor is a frame, or the unit of the data amount of the target image data is a slice, and the unit of the maximum data processing amount of the image processor is a slice.
In the embodiment of the present application, there may be multiple units of the data amount of the target image data and the maximum data processing amount of the image processor, where one image data captured by the camera at an instant may be taken as one frame image, if the unit of the data amount of the target image data is a frame, the unit of the maximum data processing amount of the image processor may also be a frame, and if the maximum data processing amount of the image processor is 2 frames, the image data captured by two cameras may be selected as the target image data every transmission period, where the target image data includes 2 frames of images.
Further, a slice (slice) is a unit of finer granularity contained in a frame, and a frame may contain one or more slices, a slice being composed of macroblocks, and a macroblock being a basic unit of an encoding process. An image is composed of 1-N slice groups, and each slice group is composed of one or several slices. The slice may be part of the content of an image data obtained by the camera at the instant of shooting, e.g. the slice is combined into a slice group, by which a certain area in the image can be represented.
In another implementation manner of the embodiment of the present application, if the camera module includes 1 camera, the unit of the data amount of the target image data may be a slice, and the unit of the maximum data processing amount of the image processor may also be a slice. Wherein, the selected slice groups in each transmission period are different, so that an image only shows different partial areas in different transmission periods.
For example, one picture is divided into four slice groups: the image processing device comprises an upper left area group, an upper right area group, a lower left area group and a lower right area group, the four groups are sequentially sent to an image processor for processing through four sending periods, and the effect that the upper left area, the upper right area, the lower left area and the lower right area of an image can be sequentially displayed on a display device through four periods is achieved.
Optionally, a preset time is spaced between adjacent sending periods, where the preset time is greater than or equal to a first threshold x and less than or equal to a second threshold y; the first threshold x is 1/(n × 24); the second threshold value y is 1/(n × 3), where n is the number of cameras, and the unit of the first threshold value x and the second threshold value y is second.
In the embodiment of the application, the application scenario of the image processing method may include aerial surveying and mapping, and if it is assumed that 5 cameras shoot in different directions during the process of flying an aerial device with 5 cameras, after a selection strategy with multiple sending periods is executed, each camera can reach a collection rate of 24 frames per second after frame dropping, and then 5 smooth videos can be correspondingly provided, so that the surveying and mapping requirements can be met. Therefore, the lower limit value of the time interval, i.e., the first threshold value x, is 1/(5 × 24) 1/120 seconds.
In order to guarantee the most basic mapping requirement, each camera preferably does not fall below 3 frames per second after frame loss. Therefore, the upper limit value of the time interval is set to 1/(5 × 3) 1/15 seconds.
In summary, in the image processing method provided by the embodiment of the present application, image data captured by each camera is obtained; in each sending period, sending target image data selected from the image data to an image processor for processing; wherein the data amount of the target image data is less than or equal to the maximum data processing amount of the image processor. According to the method and the device, the target image data with the data volume less than or equal to the maximum data processing volume can be sent to the image processor in each sending period, and all the image data are sent to the image processor through a plurality of sending periods under the condition that the image data are more, so that the instantaneous data processing pressure of the image processor is reduced, and the design complexity and the cost of the multi-camera system are also reduced. In addition, because each sending period consumes short time and a plurality of sending periods are closely adjacent, the total time consumed by all the sending periods is also short, and the smoothness of the pictures shot by the camera is also ensured.
Fig. 3 is a specific flowchart of an image processing method provided in an embodiment of the present application, where the method is applied to a camera module including multiple cameras, and as shown in fig. 3, the method may include:
201. and acquiring image data shot by each camera.
201 may specifically refer to the foregoing 101, and details are not repeated here.
202. And sequencing each image data according to a preset sequence to obtain a first image data sequence.
The embodiment of the application provides a polling selection strategy, namely, a camera module comprises a plurality of cameras, and the camera module can sequence each image data according to a preset sequence under the condition that the data volume of the image data obtained by instant shooting is far greater than the maximum data processing volume of an image processor, so as to obtain a first image data sequence.
For example, referring to fig. 1, assuming that n is 5, where the instantaneous maximum data processing amount of the image processor 22 is at most processing the image data captured by one camera, the 5 cameras may be arranged in an increasing order of arrangement to obtain a first image data sequence: { camera 1, camera 2, camera 3, camera 4, camera 5 }. It should be noted that the cameras may also be arranged in other arrangement orders, such as decreasing, which is not limited in the embodiments of the present application.
203. And in each sending period, sending the target image data selected from the first image data sequence to an image processor for processing, wherein the target image data sent in different sending periods are different.
Wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
In 203, referring to the example in 202 above, since the instantaneous processing power of the image processor 22 is at most processing image data taken by one camera, it is possible to, from the first image data sequence: starting with one end (the left end in this example) of { camera 1, camera 2, camera 3, camera 4, camera 5}, 5 image data captured by 5 cameras are transmitted through 5 transmission cycles according to a round robin selection strategy, which specifically includes: in the first sending period, only the image data 1 shot by the camera 1 is selected to be sent, and the image data shot by the cameras 2-5 is ignored; in the second sending period, only the image data 2 shot by the camera 2 is selected to be sent, and the image data shot by the camera 1 and the cameras 3-5 are ignored; in the third sending period, only the image data 3 shot by the camera 3 is selected to be sent, and the image data shot by the cameras 1-2 and 4-5 are ignored; in the fourth sending period, only the image data 4 shot by the camera 4 is selected to be sent, and the image data shot by the cameras 1-3 and the camera 5 are ignored; in a fifth sending period, only the image data 5 shot by the camera 5 is selected for sending, and the image data shot by the cameras 1-4 is ignored, wherein the target image data sent in different sending periods are different
The polling selection strategy can enable the image processor to process the pictures output by the image sensors of the 5 cameras in a balanced manner, so that the display effects of the pictures of the cameras are balanced and consistent, the pictures output by the image sensors of the 5 cameras can be displayed on the same display device, and the pictures output by the image sensors of the 5 cameras can be displayed on 5 different display devices respectively.
204. And when the image processor finishes processing the target image data received in one sending period and obtains a target image processing result corresponding to the target image data, storing the target image processing result in the memory.
Specifically, 204 may also be implemented by storing the target image processing result in the memory in a memory area corresponding to the camera that captures the target image data when the image processor finishes processing the target image data received in one sending cycle and obtains the target image processing result corresponding to the target image data.
In the embodiment of the present application, compared with a scheme in the prior art in which all image processing results are directly output to the memory for storage, when the image processor finishes processing target image data received in one sending cycle and obtains a target image processing result corresponding to the target image data, the embodiment of the present application can store the target image processing result in the memory, so that the amount of data stored each time is small, the instantaneous bandwidth pressure of the memory is reduced, and the utilization rate of the memory is improved.
Specifically, for the cameras included in the camera module, a memory area corresponding to each camera may be set in the memory in advance.
For example, image data captured by the camera 1 may be stored in the memory area 1 corresponding to the camera 1 in the memory after being processed as an image processing result.
205. And under the condition that a display instruction aiming at the target image processing result is received, extracting the target image processing result from the memory according to the display instruction, and displaying the target image processing result on a display device.
Referring to the example in 204, when the display device needs to display the image processing result corresponding to the camera 1, the display device may call the corresponding image processing result from the memory slice area 1 in the memory to display the image processing result according to the display instruction for the camera 1, thereby improving the memory management efficiency.
Optionally, a preset time is spaced between adjacent sending periods, where the preset time is greater than or equal to a first threshold x and less than or equal to a second threshold y; the first threshold x is 1/(n × 24); the second threshold value y is 1/(n × 3), where n is the number of cameras, and the unit of the first threshold value x and the second threshold value y is second. Specific contents may refer to the related description in the embodiment of fig. 2, and are not described herein again.
Optionally, the unit of the data amount of the target image data is a frame, and the unit of the maximum data processing amount of the image processor is a frame, or the unit of the data amount of the target image data is a slice, and the unit of the maximum data processing amount of the image processor is a slice. Specific contents may refer to the related description in the embodiment of fig. 2, and are not described herein again.
In summary, in the image processing method provided by the embodiment of the present application, image data captured by each camera is obtained; in each sending period, sending target image data selected from the image data to an image processor for processing; wherein the data amount of the target image data is less than or equal to the maximum data processing amount of the image processor. According to the method and the device, the target image data with the data volume less than or equal to the maximum data processing volume can be sent to the image processor in each sending period, and all the image data are sent to the image processor through a plurality of sending periods under the condition that the image data are more, so that the instantaneous data processing pressure of the image processor is reduced, and the design complexity and the cost of the multi-camera system are also reduced. In addition, because each sending period consumes short time and a plurality of sending periods are closely adjacent, the total time consumed by all the sending periods is also short, and the smoothness of the pictures shot by the camera is also ensured.
Fig. 4 is a specific flowchart of another image processing method provided in an embodiment of the present application, where the method is applied to a camera module including multiple cameras, and as shown in fig. 4, the method may include:
301. and acquiring image data shot by each camera.
301 may refer to 101, which is not described herein again.
302. Determining a processing priority for each of the cameras.
The embodiment of the application provides a selection strategy for preferentially sending important image data, in the fields of mapping, navigation and the like, under the condition that an image pickup module works, if the importance degree of a shooting object facing one camera is greater than the importance degrees of shooting objects facing other cameras, the processing priority of the camera can be set to be the highest, so that the processing priority indicates that the importance degree of the shooting object facing the camera is greater than the importance degrees of the shooting objects facing other cameras.
Therefore, the processing priority of each camera can be set by a policy not limited to the above-described policy, but for example, the processing priority of a camera with a higher shooting resolution can be set higher depending on the shooting resolution of the camera.
303. And selecting important image data shot by the target camera with the processing priority greater than or equal to a preset threshold from the image data.
For example, referring to fig. 1, assume that n is 5, where the instantaneous maximum data processing amount of the image processor 22 is to process the image data captured by at most one camera, and the processing priority relations of 5 cameras are: the processing priority of the camera 1 is greater than the processing priority of the camera 2, the processing priority of the camera 3 is approximately equal to the processing priority of the camera 4, the processing priority of the camera 5 is approximately equal to the processing priority of the camera 5, namely the processing priority of the camera 1 is the largest, and the processing priorities of the other four cameras are almost the same. If the average value of the processing priorities of the other four cameras is set to be the preset threshold, the important image data captured by the target camera may be the image data 1 captured by the camera 1.
304. And in each sending period, sending target image data selected from the image data to an image processor for processing, wherein in all sending periods, the number of sending periods of the target image data including the important image data is larger than the number of sending periods of the target image data not including the important image data.
In the embodiment of the present application, after determining important image data under some scene requirements, important attention needs to be paid to the important image data, that is, the processing priority of the important image data in the image processor is ensured, and the display smoothness and definition of the important image data are satisfied, so that, under the requirement, the number of sending cycles in which the important image data exists in the target image data can be set to be greater than the number of sending cycles in which the important image data does not exist in the target image data, so as to satisfy that the important image data is sent to the image processor preferentially and as much as possible for processing, and the display quality of the important image data is ensured, for example, referring to the example in the above 303, since the instantaneous maximum data processing amount of the image processor 22 is at most processing the image data shot by one camera, the image data 1 captured by the camera 1 can be sent to the image processor 22 as the target image data for processing in each sending period, so as to ensure the real-time processing and presentation of the pictures captured by the camera 1 to the maximum extent.
In addition, if the instantaneous maximum data processing amount of the image processor 22 is to process the image data shot by a plurality of cameras at most, in each sending period, on the basis of selecting the image data 1 shot by the camera 1, one or more image data shot by the remaining cameras with higher processing priority may be selected according to the processing priority of the remaining cameras until the total data amount of the selected image data reaches the instantaneous maximum data processing amount of the image processor 22.
Optionally, a preset time is spaced between adjacent sending periods, where the preset time is greater than or equal to a first threshold x, and is less than or equal to a second threshold y; the first threshold x is 1/(n × 24); the second threshold value y is 1/(n × 3), where n is the number of cameras, and the unit of the first threshold value x and the second threshold value y is second.
In the embodiment of the application, the application scenario of the image processing method may include aerial surveying and mapping, and if it is assumed that 5 cameras shoot in different directions during the process of flying an aerial device with 5 cameras, after a selection strategy with multiple sending periods is executed, each camera can reach an acquisition rate of 24 frames per second after frame loss, and then 5 smooth videos can be correspondingly provided, so that surveying and mapping requirements can be met. Therefore, the lower limit value of the time interval, i.e., the first threshold value x, is 1/(5 × 24) 1/120 seconds.
In order to guarantee the most basic mapping requirement, each camera preferably does not fall below 3 frames per second after frame loss. Therefore, the upper limit value of the time interval is set to 1/(5 × 3) 1/15 seconds.
Further, for the shooting of the target camera with the priority greater than or equal to the preset threshold, the frame loss is preferably not less than 10 frames per second, otherwise, the mapping picture is too unsmooth, and the watching of the important picture is affected. Therefore, in the case where there is a target camera shot with a priority greater than or equal to the preset threshold, the upper limit value of the time interval, second threshold y, is 1/(5 × 10) 1/50 seconds.
Optionally, 304 may specifically include:
3041. and sequencing the other image data except the important image data in all the image data according to a preset sequence to obtain a second image data sequence.
3042. And in a first sending period, sending the important image data to the image processor for processing.
3043. And in a second sending period, sending other image data selected from the second image data sequence to the image processor for processing.
The first transmission period is an odd transmission period, and the second transmission period is an even transmission period, or the first transmission period is an even transmission period and the second transmission period is an odd transmission period.
For 3041 to 3043, an example is provided for FIG. 1:
let n be 5, where the instantaneous maximum data processing amount of the image processor 22 is to process the image data captured by 1 camera at most, and the processing priority relations of the 5 cameras are as follows: the processing priority of the camera 1 is greater than the processing priority of the camera 2, the processing priority of the camera 3 is approximately equal to the processing priority of the camera 4, the processing priority of the camera 5 is approximately equal to the processing priority of the camera 5, namely the processing priority of the camera 1 is the largest, and the processing priorities of the other four cameras are almost the same. If the average value of the processing priorities of the other four cameras is set as the preset threshold, the important image data shot by the target camera can be the image data 1 shot by the camera 1, and the other image data except the important image data are sorted according to the ascending order of the camera numbers to obtain a second image data sequence: { camera 2, camera 3, camera 4, camera 5 }.
Then, in a first sending period, only the image data 1 shot by the camera 1 is selected to be sent, and the image data shot by the cameras 2 to 5 are ignored;
a second sending period, starting from one end (in this example, the left end) of the second image data sequence, first selecting the image data 2 captured by the camera 2 to send, and ignoring the image data captured by the camera 1 and the cameras 3-5;
in the third sending period, only the image data 1 shot by the camera 1 is selected to be sent, and the image data shot by the cameras 2-5 is ignored;
in the fourth sending period, in the second image data sequence, further selecting the image data 3 shot by the camera 3 to be sent, and ignoring the image data shot by the cameras 1-2 and 4-5;
in the fifth sending period, only the image data 1 shot by the camera 1 is selected to be sent, and the image data shot by the cameras 2-5 is ignored;
in the sixth sending period, in the second image data sequence, further selecting the image data 4 shot by the camera 4 to send, and ignoring the image data shot by the cameras 1-3 and 5;
in the seventh sending period, only the image data 1 shot by the camera 1 is selected for sending, and the image data shot by the cameras 2-5 is ignored;
and in the eighth sending period, in the second image data sequence, further selecting the image data 5 shot by the camera 5 to be sent, and ignoring the image data shot by the cameras 1-4. Of course, in the even transmission period, only the image data 1 captured by the camera 1 may be selected and transmitted, and in the odd transmission period, the image data captured by the remaining cameras may be sequentially selected and transmitted.
The selection strategy can further utilize even sending periods among odd sending periods for sending important images to carry out alternate sending processing on the images of other unimportant cameras on the basis of ensuring real-time processing and displaying of the images shot by the camera 1, and in practical application, the fluency of the image data 1 shot by the camera 1 in the final displayed image is highest, and the fluency of the image data shot by the other cameras can be accepted by a viewer.
305. And when the image processor finishes processing the target image data received in one sending period and obtains a target image processing result corresponding to the target image data, storing the target image processing result in the memory.
Specifically, 305 may also be implemented in a manner that when the image processor finishes processing the target image data received in one sending cycle and obtains a target image processing result corresponding to the target image data, the target image processing result is stored in the memory in a memory area corresponding to a camera that captures the target image data.
305 may refer to 204 above, and will not be described herein again.
306. And under the condition that a display instruction aiming at the target image processing result is received, extracting the target image processing result from the memory according to the display instruction, and displaying the target image processing result on a display device.
306 may specifically refer to 205, which is not described herein again.
Optionally, the unit of the data amount of the target image data is a frame, and the unit of the maximum data processing amount of the image processor is a frame, or the unit of the data amount of the target image data is a slice, and the unit of the maximum data processing amount of the image processor is a slice. Specific contents may refer to the related description in the embodiment of fig. 2, and are not described herein again.
In summary, in the image processing method provided by the embodiment of the present application, image data captured by each camera is obtained; in each sending period, sending target image data selected from the image data to an image processor for processing; wherein the data amount of the target image data is less than or equal to the maximum data processing amount of the image processor. According to the method and the device, the target image data with the data volume less than or equal to the maximum data processing volume can be sent to the image processor in each sending period, and all the image data are sent to the image processor through a plurality of sending periods under the condition that the image data are more, so that the instantaneous data processing pressure of the image processor is reduced, and the design complexity and the cost of the multi-camera system are also reduced. In addition, because each sending period consumes short time and a plurality of sending periods are closely adjacent, the total time consumed by all the sending periods is also short, and the smoothness of the pictures shot by the camera is also ensured.
Fig. 5 is a block diagram of an image processing apparatus provided in an embodiment of the present application, where the apparatus is applied to an image capture module including a plurality of cameras, and as shown in fig. 5, the image processing apparatus 400 may include: an acquisition module 401 and a processing module 402;
the obtaining module 401 is configured to perform: acquiring image data shot by each camera;
the processing module 402 is configured to perform:
in each sending period, sending target image data selected from the image data to an image processor for processing;
wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
Optionally, the processing module 402 is specifically configured to execute:
sequencing each image data according to a preset sequence to obtain a first image data sequence;
and in each sending period, sending the target image data selected from the first image data sequence to an image processor for processing, wherein the target image data sent in different sending periods are different.
Optionally, the processing module 402 is specifically configured to execute:
determining a processing priority for each of the cameras;
selecting important image data shot by the target camera with the processing priority greater than or equal to a preset threshold from the image data;
and in each sending period, sending target image data selected from the image data to an image processor for processing, wherein in all sending periods, the number of sending periods of the target image data including the important image data is larger than the number of sending periods of the target image data not including the important image data.
Optionally, the processing module 402 is specifically configured to execute:
sequencing other image data except the important image data in all the image data according to a preset sequence to obtain a second image data sequence;
in a first sending period, sending the important image data to the image processor for processing;
in a second sending period, sending other image data selected from the second image data sequence to the image processor for processing;
the first transmission period is an odd transmission period, and the second transmission period is an even transmission period, or the first transmission period is an even transmission period and the second transmission period is an odd transmission period.
Optionally, the processing module 402 is further configured to perform:
and when the image processor finishes processing the target image data received in one sending period and obtains a target image processing result corresponding to the target image data, storing the target image processing result in the memory.
Optionally, the processing module 402 is specifically configured to execute:
and storing the target image processing result in a memory area corresponding to a camera for shooting the target image data.
Optionally, the processing module 402 is further configured to perform:
and under the condition that a display instruction aiming at the target image processing result is received, extracting the target image processing result from the memory according to the display instruction, and displaying the target image processing result on a display device.
Optionally, a preset time is spaced between adjacent sending periods, where the preset time is greater than or equal to a first threshold x, and is less than or equal to a second threshold y; the first threshold x is 1/(n × 24); the second threshold value y is 1/(n × 3), where n is the number of cameras, and the unit of the first threshold value x and the second threshold value y is second.
Optionally, the unit of the data amount of the target image data is a frame, and the unit of the maximum data processing amount of the image processor is a frame, or the unit of the data amount of the target image data is a slice, and the unit of the maximum data processing amount of the image processor is a slice.
In summary, the image processing apparatus provided in the embodiment of the present application acquires image data captured by each camera; in each sending period, sending target image data selected from the image data to an image processor for processing; wherein the data amount of the target image data is less than or equal to the maximum data processing amount of the image processor. According to the method and the device, the target image data with the data volume less than or equal to the maximum data processing volume can be sent to the image processor in each sending period, and all the image data are sent to the image processor through a plurality of sending periods under the condition that the image data are more, so that the instantaneous data processing pressure of the image processor is reduced, and the design complexity and the cost of the multi-camera system are also reduced. In addition, because each sending period consumes short time and a plurality of sending periods are closely adjacent, the total time consumed by all the sending periods is also short, and the smoothness of the pictures shot by the camera is also ensured.
Referring to fig. 6, an embodiment of the present application further provides a camera module 500, where the camera module 500 includes the image processing apparatus 400 illustrated in fig. 5, a plurality of cameras 5011, and the cameras 5011 include image sensors 5012; the camera 5011 collects image data through the image sensor 5012; the image processing apparatus 400 receives and processes image data captured by the camera 5011.
Optionally, the camera module 500 further includes: and the pan/tilt head 502 controls the camera 5011 to adjust the shooting angle.
Optionally, the camera module 500 further includes: and a display device 503, wherein the display device 503 displays an image processing result obtained by processing the image data by the image processing device 400.
Optionally, the image sensor is a complementary metal oxide semiconductor sensor.
Referring to fig. 7, an embodiment of the present application further provides a mobile device 600, which includes a camera module 500, a controller 601, and a power system 602, where the controller 601 controls a power output of the power system 602.
Optionally, the mobile device 600 includes at least one of an unmanned aerial vehicle, an unmanned ship, and a handheld shooting device.
The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the processes of the embodiment of the image processing method, and can achieve the same technical effects, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
The acquisition module may be an interface through which the external control terminal is connected to the image processing apparatus. For example, the external control terminal may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a control terminal having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The acquisition module may be used to receive input (e.g., data information, power, etc.) from an external control terminal and transmit the received input to one or more elements within the image processing device or may be used to transmit data between the image processing device and the external control terminal.
Such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
The processor is a control center of the control terminal, is connected with each part of the whole control terminal by various interfaces and lines, and executes various functions and processing data of the control terminal by running or executing software programs and/or modules stored in the memory and calling data stored in the memory, thereby carrying out the overall monitoring on the control terminal. A processor may include one or more processing units; preferably, the processor may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor.
The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, control terminal, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create control terminals for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction control terminals which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.
The present application is described in detail above, and the principles and embodiments of the present application are described herein by using specific examples, which are only used to help understand the method and the core idea of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims (25)

  1. An image processing method applied to a camera module comprising a plurality of cameras, the method comprising:
    acquiring image data shot by each camera;
    in each sending period, sending target image data selected from the image data to an image processor for processing;
    wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
  2. The method of claim 1, wherein sending the target image data selected from the image data to an image processor for processing during each sending cycle comprises:
    sequencing each image data according to a preset sequence to obtain a first image data sequence;
    and in each sending period, sending the target image data selected from the first image data sequence to an image processor for processing, wherein the target image data sent in different sending periods are different.
  3. The method of claim 1, wherein transmitting target image data selected from the image data to an image processor for processing during each transmission cycle comprises:
    determining a processing priority for each of the cameras;
    selecting important image data shot by the target camera with the processing priority greater than or equal to a preset threshold from the image data;
    and in each sending period, sending target image data selected from the image data to an image processor for processing, wherein in all sending periods, the number of sending periods of the target image data including the important image data is larger than the number of sending periods of the target image data not including the important image data.
  4. The method according to claim 3, wherein the sending, to an image processor, in each sending cycle, target image data selected from the image data for processing, wherein the number of sending cycles of the target image data including the important image data is greater than the number of sending cycles of the target image data not including the important image data in all sending cycles, comprises:
    sequencing other image data except the important image data in all the image data according to a preset sequence to obtain a second image data sequence;
    in a first sending period, sending the important image data to the image processor for processing;
    in a second sending period, sending other image data selected from the second image data sequence to the image processor for processing;
    the first transmission period is an odd transmission period, the second transmission period is an even transmission period, or the first transmission period is an even transmission period, and the second transmission period is an odd transmission period.
  5. The method of claim 1, wherein after sending the target image data selected from the image data to an image processor for processing in each sending cycle, the method further comprises:
    and when the image processor finishes processing the target image data received in one sending period and obtains a target image processing result corresponding to the target image data, storing the target image processing result in the memory.
  6. The method of claim 5, wherein storing the target image processing result in a memory comprises:
    and storing the target image processing result in a memory area corresponding to a camera for shooting the target image data.
  7. The method of claim 5, wherein after storing the target image processing result in memory, the method further comprises:
    and under the condition that a display instruction aiming at the target image processing result is received, extracting the target image processing result from the memory according to the display instruction, and displaying the target image processing result on a display device.
  8. The method of claim 1, wherein a preset time is separated between adjacent transmission cycles, and the preset time is greater than or equal to a first threshold value x, and is less than or equal to a second threshold value y;
    the first threshold x is 1/(n × 24); the second threshold value y is 1/(n × 3), where n is the number of cameras, and the unit of the first threshold value x and the second threshold value y is second.
  9. The method according to any one of claims 1 to 8, wherein the unit of the data amount of the target image data is a frame and the unit of the maximum data processing amount of the image processor is a frame, or wherein the unit of the data amount of the target image data is a slice and the unit of the maximum data processing amount of the image processor is a slice.
  10. An image processing apparatus applied to a camera module including a plurality of cameras, comprising: the device comprises an acquisition module and a processing module;
    the acquisition module is used for acquiring image data shot by each camera;
    the processing module is used for sending target image data selected from the image data to an image processor for processing in each sending period;
    wherein a data amount of the target image data is less than or equal to a maximum data processing amount of the image processor.
  11. The apparatus of claim 10, wherein the processing module is specifically configured to:
    sequencing each image data according to a preset sequence to obtain a first image data sequence;
    and in each sending period, sending the target image data selected from the first image data sequence to an image processor for processing, wherein the target image data sent in different sending periods are different.
  12. The apparatus of claim 10, wherein the processing module is specifically configured to perform:
    determining a processing priority for each of the cameras;
    selecting important image data shot by the target camera with the processing priority greater than or equal to a preset threshold from the image data;
    and in each sending period, sending target image data selected from the image data to an image processor for processing, wherein in all sending periods, the number of sending periods of the target image data including the important image data is larger than the number of sending periods of the target image data not including the important image data.
  13. The apparatus of claim 12, wherein the processing module is specifically configured to perform:
    sequencing other image data except the important image data in all the image data according to a preset sequence to obtain a second image data sequence;
    in a first sending period, sending the important image data to the image processor for processing;
    in a second sending period, sending other image data selected from the second image data sequence to the image processor for processing;
    the first transmission period is an odd transmission period, and the second transmission period is an even transmission period, or the first transmission period is an even transmission period and the second transmission period is an odd transmission period.
  14. The apparatus of claim 10, wherein the processing module is further configured to perform:
    and when the image processor finishes processing the target image data received in one sending period and obtains a target image processing result corresponding to the target image data, storing the target image processing result in the memory.
  15. The apparatus of claim 14, wherein the processing module is specifically configured to perform:
    and storing the target image processing result in a memory area corresponding to a camera for shooting the target image data.
  16. The apparatus of claim 14, wherein the processing module is further configured to perform:
    and under the condition that a display instruction aiming at the target image processing result is received, extracting the target image processing result from the memory according to the display instruction, and displaying the target image processing result on a display device.
  17. The apparatus of claim 14, wherein a preset time is separated between adjacent transmission cycles, and the preset time is greater than or equal to a first threshold value x, and is less than or equal to a second threshold value y;
    the first threshold x is 1/(n × 24); the second threshold value y is 1/(n × 3), where n is the number of cameras, and the unit of the first threshold value x and the second threshold value y is second.
  18. The apparatus according to any one of claims 10 to 17, wherein the unit of the data amount of the target image data is a frame and the unit of the maximum data processing amount of the image processor is a frame, or wherein the unit of the data amount of the target image data is a slice and the unit of the maximum data processing amount of the image processor is a slice.
  19. A camera module, comprising the image processing apparatus of any one of claims 10 to 18, a plurality of cameras, the cameras comprising image sensors;
    the camera collects image data through the image sensor;
    and the image processing device receives and processes the image data shot by the camera.
  20. The camera module of claim 19, further comprising: and the cloud platform controls the camera to adjust the shooting angle.
  21. The camera module of claim 19, further comprising: and the display device displays an image processing result obtained after the image processing device processes the image data.
  22. The camera module of claim 19, wherein the image sensor is a cmos sensor.
  23. A mobile device comprising the camera module of any of claims 19-22, a controller, and a power system, the controller controlling a power output of the power system.
  24. The device of claim 23, wherein the movable device comprises at least one of a drone, a drone vehicle, a drone, a handheld camera device.
  25. A computer-readable storage medium characterized by comprising instructions that, when executed on a computer, cause the computer to perform the image processing method of any one of claims 1 to 9.
CN202080041502.2A 2020-04-17 2020-04-17 Image processing method and device, camera module and movable equipment Pending CN113906728A (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/085264 WO2021208053A1 (en) 2020-04-17 2020-04-17 Image processing method and apparatus, camera module, and movable device

Publications (1)

Publication Number Publication Date
CN113906728A true CN113906728A (en) 2022-01-07

Family

ID=78083698

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202080041502.2A Pending CN113906728A (en) 2020-04-17 2020-04-17 Image processing method and device, camera module and movable equipment

Country Status (2)

Country Link
CN (1) CN113906728A (en)
WO (1) WO2021208053A1 (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS62265879A (en) * 1986-05-13 1987-11-18 Sony Corp Monitor device
US6026097A (en) * 1997-03-13 2000-02-15 8 X 8, Inc. Data processor having controlled scalable input data source and method thereof
JP2006067140A (en) * 2004-08-25 2006-03-09 Matsushita Electric Ind Co Ltd Receiving frame processor, communication processor, and method of processing receiving frame
US20070046817A1 (en) * 2005-08-24 2007-03-01 Yoriko Yagi Processor, processing method and processing program
CN110191324A (en) * 2019-06-28 2019-08-30 Oppo广东移动通信有限公司 Image processing method, device, server and storage medium
CN210072612U (en) * 2018-11-13 2020-02-14 张俊峰 Internet of things chip system
JP2020047972A (en) * 2018-09-14 2020-03-26 サクサ株式会社 Image processing system
US20210212026A1 (en) * 2018-09-28 2021-07-08 Vivo Mobile Communication Co., Ltd. Resource selection method and terminal

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009118367A (en) * 2007-11-09 2009-05-28 Sony Corp Recording apparatus, recording method, program and recording system
JP4893649B2 (en) * 2008-02-08 2012-03-07 富士通株式会社 Bandwidth control server, bandwidth control program, and monitoring system
JP2010041535A (en) * 2008-08-07 2010-02-18 Hitachi Ltd Data distribution apparatus
JP6098824B2 (en) * 2013-03-08 2017-03-22 パナソニックIpマネジメント株式会社 Camera system and switching device
JP6459331B2 (en) * 2014-09-17 2019-01-30 富士ゼロックス株式会社 Image forming apparatus and program
CN106302458A (en) * 2016-08-16 2017-01-04 广东中星电子有限公司 Video information transmission method and video information transmission system

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS62265879A (en) * 1986-05-13 1987-11-18 Sony Corp Monitor device
US6026097A (en) * 1997-03-13 2000-02-15 8 X 8, Inc. Data processor having controlled scalable input data source and method thereof
JP2006067140A (en) * 2004-08-25 2006-03-09 Matsushita Electric Ind Co Ltd Receiving frame processor, communication processor, and method of processing receiving frame
US20070046817A1 (en) * 2005-08-24 2007-03-01 Yoriko Yagi Processor, processing method and processing program
JP2020047972A (en) * 2018-09-14 2020-03-26 サクサ株式会社 Image processing system
US20210212026A1 (en) * 2018-09-28 2021-07-08 Vivo Mobile Communication Co., Ltd. Resource selection method and terminal
CN210072612U (en) * 2018-11-13 2020-02-14 张俊峰 Internet of things chip system
CN110191324A (en) * 2019-06-28 2019-08-30 Oppo广东移动通信有限公司 Image processing method, device, server and storage medium

Also Published As

Publication number Publication date
WO2021208053A1 (en) 2021-10-21

Similar Documents

Publication Publication Date Title
CN111182268B (en) Video data transmission method, system, equipment and shooting device
US11503205B2 (en) Photographing method and device, and related electronic apparatus
US10582211B2 (en) Neural network to optimize video stabilization parameters
US20130021483A1 (en) Using motion information to assist in image processing
KR100776805B1 (en) Efficient image transmission method and apparatus using stereo vision processing for intelligent service robot system
CN104349039B (en) Video anti-fluttering method and device
US20150244756A1 (en) Method, Apparatus and System for Determining Terminal That is to Share Real-Time Video
CN108520493A (en) Processing method, device, storage medium and the electronic equipment that image is replaced
WO2012175783A1 (en) Video remixing system
US9251423B2 (en) Estimating motion of an event captured using a digital video camera
CN110868547A (en) Photographing control method, photographing control device, electronic equipment and storage medium
US9256789B2 (en) Estimating motion of an event captured using a digital video camera
CN108574803A (en) Choosing method, device, storage medium and the electronic equipment of image
WO2016192467A1 (en) Method and device for playing videos
CN110809797B (en) Micro video system, format and generation method
CN112449117B (en) Focusing step length determining method and device, storage medium and electronic device
CN112640419B (en) Following method, movable platform, device and storage medium
CN106357982B (en) Image service device and method
CN111263118A (en) Image acquisition method and device, storage medium and electronic device
CN113906728A (en) Image processing method and device, camera module and movable equipment
CN109121194B (en) Method and apparatus for state transition of electronic device
EP3300349A1 (en) Picture processing method, sending method, processing apparatus and sending apparatus
EP2613530A1 (en) A method for video surveillance, a related system, a related surveillance server, and a related surveillance camera
CN107534736B (en) Image registration method and device of terminal and terminal
CN115484382A (en) Parameter control method, electronic device, computer storage medium, and program product

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20220107