CN112101316B - Target detection method and system - Google Patents
Target detection method and system Download PDFInfo
- Publication number
- CN112101316B CN112101316B CN202011282010.0A CN202011282010A CN112101316B CN 112101316 B CN112101316 B CN 112101316B CN 202011282010 A CN202011282010 A CN 202011282010A CN 112101316 B CN112101316 B CN 112101316B
- Authority
- CN
- China
- Prior art keywords
- confidence coefficient
- concentration
- image data
- target
- haze
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01W—METEOROLOGY
- G01W1/00—Meteorology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/143—Sensing or illuminating at different wavelengths
Abstract
The application provides a target detection method and a target detection system, and relates to the technical field of automatic driving. The target detection method comprises the following steps: acquiring image data and radar data of a target in front of a vehicle; acquiring environmental information of an environment where a vehicle is located; and detecting the target according to the image data, the radar data and the confidence coefficient of the image data, wherein the confidence coefficient is in negative correlation with the environment severity indicated by the environment information, and the product of the confidence coefficient and the confidence coefficient of the target obtained based on the image data detection is the final confidence coefficient of the target in the image data. The confidence coefficient of the image data is determined according to the environment information of the vehicle, and then whether the image data is adopted to detect the target is determined based on the confidence coefficient of the image data, so that the situation that whether the image data is adopted to detect the target is determined based on the threshold values of image brightness, color cast or definition and the like can be avoided, the false detection rate of the target is effectively reduced, and the accuracy of target detection is improved.
Description
Technical Field
The application relates to the technical field of automatic driving, in particular to a target detection method and a target detection system.
Background
With the widespread use of automated driving techniques, it is becoming increasingly important to detect targets during driving. At present, various sensors are mainly used in vehicles, and targets are detected through data output by the various sensors. In practical application, most of the optical sensors such as a camera, infrared sensors, thermal sensors and laser sensors are adopted to obtain image data of a target, a millimeter wave radar sensor is adopted to obtain radar data such as the distance between the target and a vehicle and the speed of the target, the obtained image data and the radar data are fused, and the target is detected based on the fused data.
However, in a severe environment, the effectiveness of image data acquired by using optical sensors such as a camera, an infrared sensor, a thermal sensor, and a laser may be reduced, but the confidence of a target in the image data may still be high, which obviously cannot be achieved if the image data acquired by the optical sensors in the severe environment is filtered out through the confidence. If the image data collected by the optical sensor is still fused with the radar data collected by the millimeter wave radar sensor, false detection is generated, and then the vehicle is forced to stop, so that the running efficiency of the vehicle is influenced. Therefore, there is a need to find a method that can efficiently determine whether to use image data acquired by an optical sensor for object detection. The determination method mainly adopted at present is as follows: the gradient, the gray variance and the like of the image data output by the optical sensor are calculated, the brightness, the color cast, the definition and the like of the image data are obtained, and whether the image data acquired by the optical sensor is adopted for target detection is determined according to whether the brightness, the color cast, the definition and the like of the image data reach set thresholds or not. When the brightness, color cast or definition of the image data reach a set threshold value, fusing the image data collected by the optical sensor and the radar data collected by the millimeter wave radar sensor, and further carrying out target detection according to the fused data; and when the brightness, color cast or definition of the image data and the like do not reach the set threshold value, only performing target detection according to the radar data acquired by the millimeter wave radar sensor.
However, since the environment is not always constant due to the variability of the environment, whether to perform target detection using image data acquired by an optical sensor cannot be accurately determined based on whether the brightness, color shift, sharpness, or the like of the image data reaches a set threshold.
Disclosure of Invention
The embodiment of the application aims to provide a target detection method and a target detection system, which can accurately determine whether image data acquired by an optical sensor is adopted or not when a target is detected.
In order to solve the above technical problem, an embodiment of the present application provides the following technical solutions:
a first aspect of the present application provides a target detection method, including: acquiring image data and radar data of a target in front of the vehicle; acquiring environmental information of an environment where a vehicle is located; detecting the target according to the image data, the radar data and a confidence coefficient of the image data, wherein the confidence coefficient is in negative correlation with the environment severity indicated by the environment information, and the product of the confidence coefficient and the confidence coefficient of the target detected based on the image data is the final confidence coefficient of the target in the image data.
In some variations of the first aspect of the present application, the environmental information comprises: haze concentration; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: when the haze concentration is less than or equal to a first concentration, the confidence coefficient is 1; or when the haze concentration is greater than a first concentration and less than a second concentration, the confidence coefficient is inversely related to the haze concentration, and the confidence coefficient is a value greater than 0 and less than 1; or when the haze concentration is greater than or equal to the second concentration, the confidence coefficient is 0.
In some variations of the first aspect of the present application, the confidence coefficient is based on a value of the haze concentration when the haze concentration is greater than a first concentration and less than a second concentration
(ii) a Wherein the content of the first and second substances,
for the purpose of the confidence coefficient, it is,
in order to obtain the haze concentration as described above,
in order to be said first concentration, the first concentration,
is the second concentration.
In some variations of the first aspect of the present application, the environment information further comprises: weather information; the weather indicated by the weather information does not include rain, snow, and hail.
In some variations of the first aspect of the present application, the environmental information comprises: weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: the confidence coefficient is 0 when the weather indicated by the weather information includes at least one of rain, snow, and hail; alternatively, when the weather indicated by the weather information does not include rain, snow, and hail, the confidence coefficient is 1.
In some variations of the first aspect of the present application, the environment information further comprises: haze concentration; the haze concentration is less than or equal to the first concentration.
In some variations of the first aspect of the present application, the environmental information comprises: haze concentration and weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration; or judging whether the haze concentration is greater than or equal to a second concentration; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration and the weather information; or determining a first confidence coefficient according to the haze concentration and determining a second confidence coefficient according to the weather information; selecting a confidence coefficient having a small coefficient value from the first confidence coefficient and the second confidence coefficient as the confidence coefficient.
A second aspect of the present application provides a target detection system, comprising: the image module is used for acquiring image data of the object in front of the vehicle; the radar module is used for acquiring radar data of a target in front of the vehicle; the environment sensor is used for acquiring environment information of the environment where the vehicle is located; and the judgment processing module is used for detecting the target according to the image data, the radar data and a confidence coefficient of the image data, the confidence coefficient is in negative correlation with the environment severity indicated by the environment information, and the product of the confidence coefficient and the confidence coefficient of the target detected based on the image data is the final confidence coefficient of the target in the image data.
In some variations of the second aspect of the present application, the environmental information comprises: haze concentration; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: when the haze concentration is less than or equal to a first concentration, the confidence coefficient is 1; or when the haze concentration is greater than a first concentration and less than a second concentration, the confidence coefficient is inversely related to the haze concentration, and the confidence coefficient is a value greater than 0 and less than 1; or when the haze concentration is greater than or equal to the second concentration, the confidence coefficient is 0.
In some modified embodiments of the second aspect of the present application, the confidence coefficient is determined when the haze concentration is greater than a first concentration and less than a second concentration
(ii) a Wherein the content of the first and second substances,
for the purpose of the confidence coefficient, it is,
in order to obtain the haze concentration as described above,
in order to be said first concentration, the first concentration,
is the second concentration.
In some variations of the second aspect of the present application, the environmental information further comprises: weather information; the weather indicated by the weather information does not include rain, snow, and hail.
In some variations of the second aspect of the present application, the environmental information comprises: weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: the confidence coefficient is 0 when the weather indicated by the weather information includes at least one of rain, snow, and hail; alternatively, when the weather indicated by the weather information does not include rain, snow, and hail, the confidence coefficient is 1.
In some variations of the second aspect of the present application, the environmental information further comprises: haze concentration; the haze concentration is less than or equal to the first concentration.
In some variations of the second aspect of the present application, the environmental information comprises: haze concentration and weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration; or judging whether the haze concentration is greater than or equal to a second concentration; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration and the weather information; or determining a first confidence coefficient according to the haze concentration and determining a second confidence coefficient according to the weather information; selecting a confidence coefficient having a small coefficient value from the first confidence coefficient and the second confidence coefficient as the confidence coefficient.
Compared with the prior art, according to the target detection method provided by the first aspect of the application, after the environmental information of the environment where the vehicle is located, the image data and the radar data of the target in front of the vehicle are acquired, the confidence coefficient of the image data is determined according to the environmental information, and then whether the image data is adopted to detect the target is determined according to the confidence coefficient of the image data. That is, when the confidence coefficient is 0, the target is detected by using the image data and the radar data; when the confidence coefficient is between 0 and 1, detecting the target by using the radar data and selectively using the image data; when the confidence coefficient is 1, only radar data is used to detect a target. Compared with the prior art that whether the image data detection target is adopted or not is determined by judging whether the brightness, the color cast, the definition and the like of the image data meet the threshold value or not, and whether the image data detection target is adopted or not cannot be accurately determined by a certain threshold value due to the variability of the environment.
The object detection system provided by the second aspect of the present application has the same advantageous effects as the object detection method provided by the first aspect.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present application will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present application are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings and in which like reference numerals refer to similar or corresponding parts and in which:
FIG. 1 schematically shows a flow chart of a target detection method;
FIG. 2 schematically illustrates a right side view of the vehicle mounting components;
FIG. 3 schematically illustrates a top view of the vehicle mounting components;
FIG. 4 schematically illustrates a graph of confidence in a target versus haze concentration of an environment;
FIG. 5 schematically shows a flowchart for determining whether to use image data according to haze concentration;
FIG. 6 schematically shows a flow chart for determining whether to employ image data based on weather information;
FIG. 7 schematically illustrates a flow chart for determining confidence coefficients based on haze concentration and then adjusting the confidence coefficients based on weather information;
FIG. 8 schematically illustrates a flow chart for determining confidence coefficients based on weather information and then adjusting the confidence coefficients based on haze concentration;
FIG. 9 schematically illustrates a flow chart for determining confidence coefficients based on both haze concentration and weather information;
FIG. 10 schematically illustrates a block diagram of an object detection system I;
FIG. 11 schematically shows a second block diagram of the object detection system.
Detailed Description
Exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which this application belongs.
The embodiment of the application provides an object detection method, and an execution main body of the object detection method can be a vehicle control system installed on a vehicle or a remote control system installed outside the vehicle. After detecting the target, the remote control system may send the detected target to the vehicle control system, so that the vehicle control system controls the vehicle to run according to the detected target. Fig. 1 schematically shows a flow chart of an object detection method, which, referring to fig. 1, may comprise:
s101: image data and radar data of a target in front of the vehicle are acquired.
In the process of automatic driving of a vehicle, in order to detect a target in front of the vehicle, image data and radar data of the target in front of the vehicle need to be acquired, and then the target is detected according to the image data and the radar data. The vehicle can be an intelligent agricultural machine, an intelligent mine car, an intelligent truck and other vehicles with automatic driving functions. The target here may refer to an obstacle, a vehicle ahead of the own vehicle, a destination, or the like, which has a function of obstructing or guiding the travel of the vehicle in the embodiment of the present application.
The image data is data with semantics obtained by a camera module mounted on a vehicle. For example: color images such as RGB acquired by a monocular camera or a binocular camera, grayscale images acquired by an infrared camera, color images acquired by a thermal camera, three-dimensional stereoscopic images acquired by a laser camera, and the like. The camera module can acquire image data in real time, and then carry out target detection in real time.
The radar data is data that is not easily disturbed by a severe environment and is obtained by a radar mounted on a vehicle. For example: radar data acquired by a millimeter wave radar, a centimeter wave radar, or the like. The millimeter wave radar here mainly refers to a 77GHz or 24GHz vehicle-mounted millimeter wave radar. The radar can acquire radar data such as the position (horizontal and vertical coordinates), the speed, the approaching state (gradually approaching, gradually departing and relatively static), the reflection intensity and the probability of the parameters of the target in real time, and further detect the target in real time.
Taking the use of the camera and the millimeter wave radar on the intelligent agricultural machine as an example, in the specific implementation process, the camera and the millimeter wave radar can be installed on the central axis of the head of the intelligent agricultural machine. The yaw angle of the camera is preferably 0 degrees, and the pitch angle is determined according to the installation height of the camera and the size of a blind area of the intelligent agricultural machine. In practical application, the installation height of the camera is 1.5m away from the ground, and the pitch angle is 15 degrees downwards. Millimeter wave radars generally have a recommended mounting height, which may range approximately from 0.2m to 1.2m, depending on the vehicle type. In practical application, because the working environment of the intelligent agricultural machine is usually an unstructured road, if the installation height of the millimeter wave radar is too low, false detection of low soil piles and vegetation can be caused, and therefore the installation height of the millimeter wave radar is 0.8m away from the ground. And, the normal direction of the front plane of the millimeter wave radar needs to be consistent with the driving direction of the intelligent agricultural machine, that is, the direction of the millimeter wave radar transmitting radar waves faces the right front of the intelligent agricultural machine.
It should be noted here that in order to enable better fusion of the image data and the radar data, i.e. to simultaneously measure the targets in part or all of the driving scene, the field angles of the camera and the millimeter wave radar need to be at least partially overlapped, preferably completely overlapped.
It should be further noted that the acquisition of the image data and the acquisition of the radar data are not in sequence in time, and the image data and the radar data may be acquired synchronously or asynchronously.
S102: environmental information of an environment in which the vehicle is located is acquired.
Since image data is highly susceptible to the influence of a severe environment, it is necessary to determine whether to use image data when detecting an object. In the process of determining whether to adopt the image data to detect the target, the environment information of the environment where the vehicle is located needs to be acquired, and then whether to adopt the image data to detect the target is determined according to the environment information. The environmental information here may include haze concentration and weather information. The haze is a combined word of fog and haze. Therefore, the haze concentration refers to the concentration of mist and the concentration of solid particles such as smoke dust in the air. And the weather information refers to the condition of weather, such as: sunny, cloudy, rainy, snowy, hail, etc.
In concrete implementation process, in order to obtain haze concentration, can install air quality sensor on the vehicle, if: a PM2.5 air mass sensor, a PM10 air mass sensor, or the like. Because whether need confirm to adopt image data to detect the target according to haze concentration, the event needs to make the distance of air quality sensor and camera module be less than preset distance, is about to install near the camera module air quality sensor. And a rain and snow sensor may be mounted on the vehicle in order to acquire weather information. Because whether image data are adopted to detect the target needs to be determined according to weather information, in order to avoid the rain and snow sensor being shielded, the rain and snow sensor can accurately determine the weather, the rain and snow sensor needs to be installed on the roof. The rain and snow sensor herein can detect not only whether it is raining or snowing, but also whether it is hail. The rain and snow sensor also has an automatic heating function, when rain and snow fall on the rain and snow sensor, the rain and snow sensor can automatically heat, so that the rain and snow falling on the rain and snow sensor are melted, and the accuracy of detecting the rain and snow by the rain and snow sensor is improved. Haze concentration and weather information can be obtained in real time through the air quality sensor and the rain and snow sensor, and then whether image data are adopted to detect a target or not is determined in real time.
Because the air quality sensor and the rain and snow sensor are low in price, the detection cost of the target cannot be greatly increased when the air quality sensor and the rain and snow sensor are integrated in the vehicle, even the air quality sensor and the rain and snow sensor exist in some vehicles, and the air quality sensor and the rain and snow sensor are not used for selecting image data in the prior art, so that the accuracy of target detection can be improved and the target detection cost can be saved by judging whether to detect the target by adopting the image data according to the data acquired by the air quality sensor and the rain and snow sensor. In addition, the air quality sensor and the rain and snow sensor are small in size and easy to install, and therefore the original target detection system of the vehicle does not need to be changed greatly. Moreover, the data acquired by the air quality sensor and the rain and snow sensor is used for judging whether the image data is adopted to detect the target, the calculation speed is faster than that of the image data obtained by calculation, such as brightness, color cast, definition and the like, compared with the threshold value to judge whether the image data is adopted to detect the target, extra calculation force guarantee is not needed, the target detection accuracy is improved, and meanwhile, the target detection efficiency can also be improved.
It should be noted that S101 and S102 are not consecutive in execution order. S101 and then S102 may be executed, S102 and then S101 may be executed, or S101 and S102 may be executed simultaneously.
Fig. 2 schematically shows a right side view of various parts mounted on the vehicle, and referring to fig. 2, a camera 202 and a millimeter wave radar 203 are mounted right in front of a vehicle 201, an air quality sensor 204 is mounted in the vicinity of the camera 202 and the millimeter wave radar 203, and a rain and snow sensor 205 is mounted right above the vehicle 201. Fig. 3 schematically shows a top view of various parts mounted on the vehicle, and referring to fig. 3, it can also be seen that a camera 202 and a millimeter wave radar 203 are mounted right in front of the vehicle 201, an air quality sensor 204 is mounted in the vicinity of the camera 202 and the millimeter wave radar 203, and a rain and snow sensor 205 is mounted right above the vehicle 201.
S103: and detecting the target according to the image data, the radar data and the confidence coefficient of the image data.
Wherein the confidence coefficient is inversely related to the environmental severity indicated by the environmental information. The product of the confidence coefficient and the confidence of the target detected based on the image data is the final confidence of the target in the image data. That is, the higher the environmental severity, the lower the confidence coefficient, and the lower the confidence of the target finally output based on the image data.
In the specific implementation process, if the environmental severity is higher than a preset level, for example: the concentration of rain, snow, hail and haze is higher than the preset concentration or strong light, so the confidence coefficient is set to be 0, namely, image data is not adopted, and the target is detected only according to the radar data. If the environmental severity is lower than a preset level, such as: and if the rain, snow, hail and haze concentration are lower than the preset concentration, the confidence coefficient is set to be 1, namely, the image data is adopted, and the target is detected according to the image data and the radar data.
There is also an intermediate case where the environmental severity is general, i.e. the environmental severity is higher than a first level and lower than a second level, such as: the rain, snow and hail are not produced, but the haze concentration is higher. Then the confidence coefficient is set between 0-1, i.e., the image data is selectively trusted, and the target is detected based on the radar data and the selectively trusted image data.
For detecting a target only according to radar data, specifically, firstly, whether image data and a confidence coefficient of the image data are received or not can be judged, if not, the current environment is not suitable for detecting the target by adopting the image data, namely only the radar data are processed; then, carrying out projection transformation from a radar coordinate system to a vehicle body coordinate system; then, selecting a region of interest (ROI); and finally, judging the foreground and the background. Since the specific implementation of detecting a target only according to radar data is the prior art, it is not described herein again.
In a scene of using an intelligent agricultural machine to detect obstacles, in the process of judging the foreground and the background based on radar data, the foreground generally refers to people, livestock, agricultural vehicles, buckets, trees, telegraph poles and the like, and the background refers to soil blocks, high-reflectivity metals and the like which easily interfere with the obstacle detection. In a specific implementation process, irrelevant noise can be filtered through counting the accumulated occurrence times of the obstacles, the confidence degree of the detected obstacles and the like, so that whether the detected obstacles are a foreground or a background is judged. For example: when the number of the detected accumulative occurrences of the obstacle is large, determining the obstacle as a foreground; when the accumulated occurrence number of the detected obstacles is only one time or effective times, the obstacles are indicated as the objects which are flashed in front of the intelligent agricultural machine, such as: flying birds, etc., the obstacle is determined to be background.
For detecting a target according to image data and radar data, specifically, firstly, whether the image data and the confidence coefficient of the image data are received or not can be judged, if only the image data are received and the confidence coefficient of the image data are not received, the current environment is suitable for detecting the target by adopting the image data, namely, the image data and the radar data are normally fused; then, detecting an obstacle bounding box based on the image data, and judging the foreground and the background based on the radar data; then, carrying out projection transformation of the radar data to the coordinate system of the image data; then, screening and filtering out the obstacles according to the position and the confidence of the bounding box; and finally, fusing the image data after screening and filtering with radar data. The fusion of the image data and the radar data and the specific implementation manner of detecting the target according to the fused data of the image data and the radar data are the prior art, so the details are not repeated herein.
In addition, for detecting a target according to image data and radar data, specifically, firstly, whether the confidence coefficient of the image data and the image data is received or not can be judged, and if the confidence coefficient of the image data and the image data is received at the same time, it is indicated that the current environment is more suitable for detecting the target by using the image data, but certain influence exists, namely, the confidence coefficient of the image data is multiplied by the confidence coefficient and then the image data is normally fused with the radar data. The method comprises the following steps: judging the foreground and the background based on the radar data; transforming the radar data into projection under an image data coordinate system; detecting an obstacle bounding box based on the image data, and screening and filtering the obstacle based on the confidence coefficient multiplied by the confidence coefficient; and fusing the image data after screening and filtering with radar data.
As can be seen from the above, according to the target detection method provided in the embodiment of the present application, after the environmental information of the environment where the vehicle is located, the image data of the target in front of the vehicle, and the radar data are acquired, the confidence coefficient of the image data is determined according to the environmental information, and then whether to detect the target by using the image data is determined according to the confidence coefficient of the image data. That is, when the confidence coefficient is 0, the target is detected by using the image data and the radar data; when the confidence coefficient is between 0 and 1, detecting the target by using the radar data and selectively using the image data; when the confidence coefficient is 1, only radar data is used to detect a target. Compared with the prior art that whether the image data detection target is adopted or not is determined by judging whether the brightness, the color cast, the definition and the like of the image data meet the threshold value or not, and whether the image data detection target is adopted or not cannot be accurately determined by a certain threshold value due to the variability of the environment.
Further, as a refinement and an extension of the object detection method shown in fig. 1, a specific method of determining the confidence coefficient is explained next from three aspects.
In a first aspect: the environmental information only comprises haze concentration, namely, the confidence coefficient is determined only according to the haze concentration.
Specifically, when the haze concentration is less than or equal to the first concentration, the haze concentration is low, the image data is less affected by the environment, and therefore the confidence coefficient is set to 1; when the haze concentration is greater than the first concentration and less than the second concentration, it is indicated that the haze concentration is general, and the image data is affected by the environment, but the influence is not serious, so the confidence coefficient is set to a value greater than 0 and less than 1, and the greater the haze concentration is, the lower the confidence coefficient is set; when the haze concentration is greater than or equal to the second concentration, the haze concentration is higher, the image data is greatly influenced by the environment, and therefore the confidence coefficient is set to be 0.
When the haze concentration is greater than the first concentration and less than the second concentration, the confidence coefficient may be specifically determined according to the following formula:
wherein the content of the first and second substances,
in order to be a confidence coefficient of the image,
is the haze concentration of the current environment of the vehicle,
is a first concentration of the first component in the first concentration,
at the second concentration.
Because different camera modules are different to the false retrieval rate of same target under different haze concentrations, the false retrieval rate of same target also can be different even if different camera modules are under same haze concentration, therefore, need test the false retrieval rate of same target under different haze concentrations to different camera modules, and then carry out statistical analysis to the test result, reach the confidence coefficient of camera module to same target testing result under different haze concentrations, and then confirm the concrete numerical value of first concentration and second concentration according to statistical analysis's confidence coefficient. Specifically, in the same time period (keeping similar conditions such as the brightness of the environment), the haze concentration output by the air quality sensor and the image data output by the camera module are collected at the same time, the image data in the same haze concentration interval are classified, and then the average detection confidence of the same target type (people, vehicles, farm implements and the like) in different haze concentration intervals is calculated through a target detection algorithm. Generally, when the haze concentration is low, the confidence of detection is high. For example, the confidence of detection is usually higher than 95%, and the confidence of detection decreases with the increase of the haze concentration until the haze concentration is too high to detect an effective target.
Fig. 4 schematically shows a relationship diagram of the confidence of the target and the haze concentration of the environment, and referring to fig. 4, after testing and statistics, the relationship between the confidence of the target and the haze concentration of the environment is obtained. Selecting haze concentration of environment corresponding to confidence degree of target of 75%
As the first concentration, the haze concentration of the environment corresponding to the confidence of 35% of the target is selected
As the second concentration.
Fig. 5 schematically shows a flowchart for determining whether to adopt image data according to haze concentration, and referring to fig. 5, may include:
s501: acquiring image data and radar data of a target in front of a vehicle;
here, S501 and S101 have the same implementation, and the specific implementation process of S501 may refer to the specific description of S101, which is not described herein again.
S502: obtaining the haze concentration of the environment where the vehicle is located;
here, S502 and S102 have similar implementation manners, and a specific implementation process of S502 may refer to a specific description of S102, which is not described herein again.
S503: judging whether the haze concentration is greater than a second concentration; if yes, executing S504: if not, executing S505;
s504: outputting radar data;
s505: judging whether the haze concentration is less than a first concentration; if yes, go to S506; if not, executing S507;
s506: outputting the image data and the radar data;
s507: and outputting the image data, the radar data and the confidence coefficient of the image data.
In a second aspect: the environment information includes only weather information, i.e., the confidence coefficient is determined only from the weather information.
Specifically, when the weather indicated by the weather information includes at least one of rain, snow, and hail, it is described that the weather is bad, the image data is greatly affected by the environment, and therefore the confidence coefficient is set to 0; when the weather indicated by the weather information does not include rain, snow, and hail, it is described that the weather is good and the image data is less affected by the environment, so the confidence coefficient is set to 1.
Fig. 6 schematically shows a flowchart for determining whether to adopt image data according to weather information, and referring to fig. 6, may include:
s601: acquiring image data and radar data of a target in front of a vehicle;
here, S601 and S101 have the same implementation, and the specific implementation process of S601 may refer to the specific description of S101, which is not described herein again.
S602: acquiring weather information of the environment where the vehicle is located;
here, S602 and S102 have similar implementation manners, and a specific implementation process of S602 may refer to a specific description of S102, which is not described herein again.
S603: judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, go to S604: if not, executing S605;
s604: outputting radar data;
s605: the image data and the radar data are output.
In a third aspect: the environment information comprises haze concentration and weather information, and the confidence coefficient is determined according to the haze concentration and the weather information.
Specifically, determining the confidence coefficient according to the haze concentration and the weather information includes three implementation manners.
The first method is as follows: and determining a confidence coefficient according to the haze concentration, and adjusting the confidence coefficient according to the weather information.
Fig. 7 schematically shows a flowchart of determining a confidence coefficient according to the haze concentration, and then adjusting the confidence coefficient according to the weather information, and as shown in fig. 7, the method may include:
s701: acquiring image data and radar data of a target in front of a vehicle;
here, S701 and S101 have the same implementation, and the specific implementation process of S701 may refer to the specific description of S101, which is not described herein again.
S702: obtaining haze concentration and weather information of the environment where the vehicle is located;
here, S702 and S102 have similar implementation manners, and a specific implementation process of S702 may refer to a specific description of S102, which is not described herein again.
S703: judging whether the haze concentration is greater than a second concentration; if yes, go to S704: if not, executing S705;
s704: outputting radar data;
s705: judging whether the haze concentration is less than a first concentration; if yes, go to S706; if not, executing S706 and S707;
s706: judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, go to S704; if not, executing S708 or S709;
s707: determining a confidence coefficient of the image data;
s708: outputting image data, radar data and confidence coefficient of the image data;
s709: the image data and the radar data are output.
It should be noted that in the process of executing S705, if the determination result is yes, S706 is executed, and if the determination result is no, S709 is executed, that is, if the haze concentration is less than the first concentration and there is no rain, snow, or hail, the image data and the radar data are output. In the process of S705, if the determination result is no, S706 and S707 are executed, and if the determination result is no, S708 is executed, that is, if the haze concentration is greater than the first concentration and less than the second concentration, and there is no rain, snow, or hail, the image data, the radar data, and the confidence coefficient of the image data are output.
Or after S703, if not, determining a first confidence coefficient (1 or a value between 0 and 1) according to the haze concentration, and determining a second confidence coefficient (0 or 1) according to the weather information; the confidence coefficient having a small coefficient value is selected from the first confidence coefficient and the second confidence coefficient as the confidence coefficient of the image data. In the first mode, when the haze concentration is smaller than the second concentration, the specific mode of determining the confidence coefficient of the image data according to the haze concentration and the weather information is not limited herein.
The second method comprises the following steps: and determining a confidence coefficient according to the weather information, and adjusting the confidence coefficient according to the haze concentration.
Fig. 8 schematically shows a flowchart of determining a confidence coefficient according to weather information, and then adjusting the confidence coefficient according to haze concentration, and as shown in fig. 8, the method may include:
s801: acquiring image data and radar data of a target in front of a vehicle;
here, S801 and S101 have the same implementation manner, and the specific implementation process of S801 may refer to the specific description of S101, which is not described herein again.
S802: obtaining haze concentration and weather information of the environment where the vehicle is located;
here, S802 and S102 have similar implementation manners, and a specific implementation process of S802 may refer to a specific description of S102, which is not described herein again.
S803: judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, go to S804: if not, executing S805;
s804: outputting radar data;
s805: judging whether the haze concentration is less than a first concentration; if yes, executing S806; if not, executing S807;
s806: outputting the image data and the radar data;
s807: and outputting the image data, the radar data and the confidence coefficient of the image data.
Here, when the haze concentration is greater than the first concentration and less than the second concentration, the greater the haze concentration, the smaller the confidence coefficient of the image data, and the smaller the haze concentration, the greater the confidence coefficient of the image data. The confidence coefficient of the image data here is a numerical value between 0 and 1.
It should be noted that, whether the image data is output or not is determined by the first or second method, as long as the image data is determined to be greatly influenced by the environment for the first time, the radar data is directly output without performing the second determination. Thus, the speed of detecting whether the target is detected by using the image data can be increased, and the target detection speed can be increased. For example, with regard to the first mode, as long as the haze concentration is determined to be greater than the second concentration, the radar data is directly output without determining whether the weather indicated by the weather information includes at least one of rain, snow and hail. For the second mode, as long as it is determined that the weather indicated by the weather information includes at least one of rain, snow and hail, the radar data is directly output, and it is not necessary to determine whether the haze concentration is greater than the second concentration or less than the first concentration.
The third method comprises the following steps: and finally, selecting the coefficient with a small coefficient value from the first confidence coefficient and the second confidence coefficient as a final confidence coefficient.
Fig. 9 schematically shows a flowchart for determining the confidence coefficient according to the haze concentration and the weather information, and referring to fig. 9, the determining may include:
s901: acquiring image data and radar data of a target in front of a vehicle;
here, S901 and S101 have the same implementation manner, and the specific implementation process of S901 may refer to the specific description of S101, which is not described herein again.
S902: obtaining haze concentration and weather information of the environment where the vehicle is located;
here, S902 and S102 have similar implementation manners, and a specific implementation process of S902 may refer to a specific description of S102, which is not described herein again.
S903: determining a first confidence coefficient according to the haze concentration;
s904: determining a second confidence coefficient according to the weather information;
s905: selecting a confidence coefficient with a small coefficient value from the first confidence coefficient and the second confidence coefficient as a final confidence coefficient;
s906: and outputting the image data, the radar data and the final confidence coefficient of the image data.
It should be noted that S901 and S902 are not sequentially divided in the execution order, and may be executed simultaneously or not. Similarly, S903 and S904 are not sequentially executed, and may be executed at the same time or at different times.
Based on the same inventive concept, as an implementation of the target detection method, the embodiment of the application also provides a target detection system. FIG. 10 schematically illustrates a block diagram of an object detection system, which, referring to FIG. 10, may include: an image module 1001 for acquiring image data of an object ahead of the vehicle; the radar module 1002 is used for acquiring radar data of a target in front of the vehicle; an environment sensor 1003 for acquiring environment information of an environment in which the vehicle is located; a determining module 1004, configured to detect the target according to the image data, the radar data, and a confidence coefficient of the image data, where the confidence coefficient is negatively correlated with the environmental severity indicated by the environmental information, and a product of the confidence coefficient and a confidence of the target detected based on the image data is a final confidence of the target in the image data.
Specifically, fig. 11 schematically illustrates a second structure diagram of the target detection system, and referring to fig. 11, the image module 1001 may specifically be a camera module 1101 and/or a laser radar 1102, the radar module 1002 may specifically be a millimeter wave radar 203, the environmental sensor 1003 may specifically be an air quality sensor 204 and/or a rain and snow sensor 205, and the determination processing module 1004 may specifically be divided into a data validity determination module 1103 and a target information processing unit 1104. The data validity judging module 1103 is configured to determine a confidence coefficient of image data acquired by the camera module 1101 and/or the laser radar 1102, that is, determine whether to detect a target using the image data. The target information processing unit 1104 is configured to detect a target according to image data collected by the camera module 1101 and/or the laser radar 1102, radar data collected by the millimeter wave radar 203, and the confidence coefficient determined by the data validity judgment module 1103.
In other embodiments of the present application, the environment information includes: haze concentration; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: when the haze concentration is less than or equal to a first concentration, the confidence coefficient is 1; or when the haze concentration is greater than a first concentration and less than a second concentration, the confidence coefficient is inversely related to the haze concentration, and the confidence coefficient is a value greater than 0 and less than 1; or when the haze concentration is greater than or equal to the second concentration, the confidence coefficient is 0.
In other embodiments of the present application, the confidence coefficient is determined when the haze concentration is greater than the first concentration and less than the second concentration
(ii) a Wherein the content of the first and second substances,
for the purpose of the confidence coefficient, it is,
in order to obtain the haze concentration as described above,
in order to be said first concentration, the first concentration,
is the second concentration.
In other embodiments of the present application, the environment information further includes: weather information; the weather indicated by the weather information does not include rain, snow, and hail.
In other embodiments of the present application, the environment information includes: weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: the confidence coefficient is 0 when the weather indicated by the weather information includes at least one of rain, snow, and hail; alternatively, when the weather indicated by the weather information does not include rain, snow, and hail, the confidence coefficient is 1.
In other embodiments of the present application, the environment information further includes: haze concentration; the haze concentration is less than or equal to the first concentration.
In other embodiments of the present application, the environment information includes: haze concentration and weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including: judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration; or judging whether the haze concentration is greater than or equal to a second concentration; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration and the weather information; or determining a first confidence coefficient according to the haze concentration and determining a second confidence coefficient according to the weather information; selecting a confidence coefficient having a small coefficient value from the first confidence coefficient and the second confidence coefficient as the confidence coefficient.
It is to be noted here that the above description of the embodiment of the object detection system, similar to the above description of the embodiment of the object detection method, has similar advantageous effects as the embodiment of the object detection method. For technical details that are not disclosed in the embodiments of the object detection system of the present application, please refer to the description of the embodiments of the object detection method of the present application.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of object detection, comprising:
acquiring image data and radar data of a target in front of a vehicle, wherein the radar data is acquired by a millimeter wave radar;
the method comprises the steps of obtaining environmental information of an environment where a vehicle is located to determine whether an image data detection target is adopted or not, wherein the environmental information comprises haze concentration and weather information, the haze concentration is obtained through an air quality sensor, the weather information is obtained through a rain and snow sensor, the distance between the air quality sensor and a camera module for obtaining the image data is smaller than a preset distance, and the rain and snow sensor has an automatic heating function;
detecting the target according to the image data, the radar data, and a confidence coefficient of the image data, wherein the confidence coefficient is negatively correlated with the environmental severity indicated by the environmental information, the confidence coefficient is 0 when the haze concentration is greater than or equal to a second concentration, or the weather indicated by the weather information includes at least one of rain, snow, and hail, and the confidence coefficient is 1 when the haze concentration is less than or equal to a first concentration, and a product of the confidence coefficient and the confidence coefficient of the target detected based on the image data is a final confidence of the target in the image data;
the false detection rate of the same target is tested under different haze concentrations by different camera modules, statistical analysis is carried out on the test result, confidence coefficients of the camera modules to the detection result of the same target under different haze concentrations are obtained, and specific numerical values of the first concentration and the second concentration are determined according to the confidence coefficients of the statistical analysis.
2. The method of claim 1, wherein the context information comprises: haze concentration; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including:
when the haze concentration is greater than the first concentration and less than the second concentration, the confidence coefficient is inversely related to the haze concentration, and the confidence coefficient is a value greater than 0 and less than 1.
3. The method of claim 2, wherein the confidence coefficient is based on a haze concentration greater than a first concentration and less than a second concentration
；
Wherein the content of the first and second substances,
for the purpose of the confidence coefficient, it is,
in order to obtain the haze concentration as described above,
in order to be said first concentration, the first concentration,
is the second concentration.
4. The method of claim 2 or 3, wherein the context information further comprises: weather information; the weather indicated by the weather information does not include rain, snow, and hail.
5. The method of claim 1, wherein the context information comprises: weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including:
when the weather indicated by the weather information does not include rain, snow, and hail, the confidence coefficient is 1.
6. The method of claim 1, wherein the context information comprises: haze concentration and weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including:
judging whether the weather indicated by the weather information comprises at least one of rain, snow and hail; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration; alternatively, the first and second electrodes may be,
judging whether the haze concentration is greater than or equal to a second concentration; if yes, setting the confidence coefficient to be 0; if not, determining the confidence coefficient according to the haze concentration and the weather information; alternatively, the first and second electrodes may be,
determining a first confidence coefficient according to the haze concentration, and determining a second confidence coefficient according to the weather information; selecting a confidence coefficient having a small coefficient value from the first confidence coefficient and the second confidence coefficient as the confidence coefficient.
7. An object detection system, comprising:
the image module is used for acquiring image data of a target in front of the vehicle;
the radar module is used for acquiring radar data of a target in front of the vehicle, and the radar module is a millimeter wave radar;
the environment sensor is used for acquiring environment information of an environment where a vehicle is located so as to determine whether an image data detection target is adopted or not, the environment information comprises haze concentration and weather information, the environment sensor comprises an air quality sensor and a rain and snow sensor, the haze concentration is acquired through the air quality sensor, the weather information is acquired through the rain and snow sensor, the distance between the air quality sensor and a camera module for acquiring the image data is smaller than a preset distance, and the rain and snow sensor has an automatic heating function;
a determination processing module, configured to detect the target according to the image data, the radar data, and a confidence coefficient of the image data, where the confidence coefficient is negatively correlated with an environmental severity indicated by the environmental information, the confidence coefficient is 0 when the haze concentration is greater than or equal to a second concentration, or weather indicated by the weather information includes at least one of rain, snow, and hail, and the confidence coefficient is 1 when the haze concentration is less than or equal to a first concentration, and a product of the confidence coefficient and a confidence coefficient of the target detected based on the image data is a final confidence coefficient of the target in the image data;
the false detection rate of the same target is tested under different haze concentrations by different camera modules, statistical analysis is carried out on the test result, confidence coefficients of the camera modules to the detection result of the same target under different haze concentrations are obtained, and specific numerical values of the first concentration and the second concentration are determined according to the confidence coefficients of the statistical analysis.
8. The system of claim 7, wherein the environmental information comprises: haze concentration; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including:
when the haze concentration is greater than the first concentration and less than the second concentration, the confidence coefficient is inversely related to the haze concentration, and the confidence coefficient is a value greater than 0 and less than 1.
9. The system of claim 8, wherein the confidence coefficient is based on a haze concentration greater than a first concentration and less than a second concentration
；
Wherein the content of the first and second substances,
for the purpose of the confidence coefficient, it is,
in order to obtain the haze concentration as described above,
in order to be said first concentration, the first concentration,
is the second concentration.
10. The system of claim 7, wherein the environmental information comprises: weather information; the confidence coefficient is inversely related to the environmental severity indicated by the environmental information, including:
when the weather indicated by the weather information does not include rain, snow, and hail, the confidence coefficient is 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011282010.0A CN112101316B (en) | 2020-11-17 | 2020-11-17 | Target detection method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011282010.0A CN112101316B (en) | 2020-11-17 | 2020-11-17 | Target detection method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112101316A CN112101316A (en) | 2020-12-18 |
| CN112101316B true CN112101316B (en) | 2022-03-25 |
Family
ID=73785638
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011282010.0A Active CN112101316B (en) | 2020-11-17 | 2020-11-17 | Target detection method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112101316B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113378628B (en) * | 2021-04-27 | 2023-04-14 | 阿里云计算有限公司 | Road obstacle area detection method |
| CN113553937A (en) * | 2021-07-19 | 2021-10-26 | 北京百度网讯科技有限公司 | Target detection method, target detection device, electronic equipment and storage medium |
| CN116824362A (en) * | 2022-04-06 | 2023-09-29 | 布瑞克(苏州)农业互联网股份有限公司 | Agricultural product monitoring method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542843A (en) * | 2010-12-07 | 2012-07-04 | 比亚迪股份有限公司 | Early warning method for preventing vehicle collision and device |
| DE102016212716A1 (en) * | 2016-07-13 | 2018-01-18 | Conti Temic Microelectronic Gmbh | CONTROL DEVICE AND METHOD |
| CN108313088A (en) * | 2018-02-22 | 2018-07-24 | 中车长春轨道客车股份有限公司 | A kind of contactless rail vehicle obstacle detection system |
| CN110837800A (en) * | 2019-11-05 | 2020-02-25 | 畅加风行(苏州)智能科技有限公司 | Port severe weather-oriented target detection and identification method |
| CN111142528A (en) * | 2019-12-31 | 2020-05-12 | 天津职业技术师范大学(中国职业培训指导教师进修中心) | Vehicle dangerous scene sensing method, device and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110406544A (en) * | 2019-08-06 | 2019-11-05 | 阿尔法巴人工智能(深圳)有限公司 | Vehicle sensory perceptual system and method under misty rain scene |
-
2020
- 2020-11-17 CN CN202011282010.0A patent/CN112101316B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102542843A (en) * | 2010-12-07 | 2012-07-04 | 比亚迪股份有限公司 | Early warning method for preventing vehicle collision and device |
| DE102016212716A1 (en) * | 2016-07-13 | 2018-01-18 | Conti Temic Microelectronic Gmbh | CONTROL DEVICE AND METHOD |
| CN108313088A (en) * | 2018-02-22 | 2018-07-24 | 中车长春轨道客车股份有限公司 | A kind of contactless rail vehicle obstacle detection system |
| CN110837800A (en) * | 2019-11-05 | 2020-02-25 | 畅加风行(苏州)智能科技有限公司 | Port severe weather-oriented target detection and identification method |
| CN111142528A (en) * | 2019-12-31 | 2020-05-12 | 天津职业技术师范大学(中国职业培训指导教师进修中心) | Vehicle dangerous scene sensing method, device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112101316A (en) | 2020-12-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112101316B (en) | Target detection method and system | |
| CN109634282B (en) | Autonomous vehicle, method and apparatus | |
| CN112215306B (en) | Target detection method based on fusion of monocular vision and millimeter wave radar | |
| US10933798B2 (en) | Vehicle lighting control system with fog detection | |
| EP0586857A1 (en) | Vehicle lane position detection system | |
| EP1271179A2 (en) | Device for detecting the presence of objects | |
| JP2006184276A (en) | All-weather obstacle collision preventing device by visual detection, and method therefor | |
| CN111582130B (en) | Traffic behavior perception fusion system and method based on multi-source heterogeneous information | |
| Liu et al. | Development of a vision-based driver assistance system with lane departure warning and forward collision warning functions | |
| CN113820714B (en) | Dust fog weather road environment sensing system based on multi-sensor fusion | |
| JP6557923B2 (en) | On-vehicle radar device and area detection method | |
| CN110659552B (en) | Tramcar obstacle detection and alarm method | |
| CN114415171A (en) | Automobile travelable area detection method based on 4D millimeter wave radar | |
| CN114578344A (en) | Target sensing method, device and system suitable for rainy environment | |
| CN115034324A (en) | Multi-sensor fusion perception efficiency enhancement method | |
| CN106405539B (en) | Vehicle radar system and method for removing a non-interesting object | |
| CN108256418B (en) | Pedestrian early warning method and system based on infrared imaging | |
| JP2008056163A (en) | Obstacle detecting device for vehicle | |
| CN111414857A (en) | Front vehicle detection method based on vision multi-feature fusion | |
| Hautière et al. | Detection of visibility conditions through use of onboard cameras | |
| CN114084129A (en) | Fusion-based vehicle automatic driving control method and system | |
| JP4033106B2 (en) | Ranging performance degradation detection device for vehicles | |
| CN114675295A (en) | Method, device and equipment for judging obstacle and storage medium | |
| Lu et al. | A vision-based system for the prevention of car collisions at night | |
| CN112241004A (en) | Object recognition device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |