CN111829435A - Multi-binocular camera and line laser cooperative detection method - Google Patents
Multi-binocular camera and line laser cooperative detection method Download PDFInfo
- Publication number
- CN111829435A CN111829435A CN201910793994.XA CN201910793994A CN111829435A CN 111829435 A CN111829435 A CN 111829435A CN 201910793994 A CN201910793994 A CN 201910793994A CN 111829435 A CN111829435 A CN 111829435A
- Authority
- CN
- China
- Prior art keywords
- binocular camera
- camera
- binocular
- line laser
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
A multi-binocular camera and line laser cooperative detection method comprises the following steps: erecting a binocular camera module; carrying out three-dimensional calibration on left and right cameras of each binocular camera; respectively acquiring a left image and a right image of a measured object, performing three-dimensional correction, matching point pairs, and calculating to obtain three-dimensional information of the measured object according to left and right view parallax; carrying out coordinate conversion of the pose relation; and fusing the real-time depth image data and the real-time attitude data at the same moment, and respectively calculating the three-dimensional position of each coordinate system to obtain complete three-dimensional data. The binocular camera modules are arranged on the same plane of the platform and are arranged in a straight line. The invention adopts a plurality of groups of line lasers and binocular camera modules which are arranged on a plane in parallel to carry out object detection, three-dimensional reconstruction and data fusion are carried out after three-dimensional coordinates are obtained, the measurement speed is greatly accelerated by adopting a multi-module collaborative scanning mode, the measurement efficiency is improved, and the measurement precision is higher than that of the traditional single-binocular camera.
Description
Technical Field
The invention relates to the technical field of object three-dimensional information detection, in particular to a multi-binocular camera and line laser cooperative detection method.
Background
Binocular vision is an important branch of computers, can simulate human eyes and the process of human stereoscopic vision perception, and is one of the core subjects of computer vision research. In recent years, binocular vision technology has been widely used in the fields of obstacle detection, target object detection, and the like.
The binocular stereo vision uses one or two CCD or CMOS digital cameras to shoot the same surface of the measured object from different angles, and obtains the three-dimensional coordinates of the point by calculating the parallax of the space point in the two images. This measurement method requires determining the corresponding position of the same point in space on two or more images taken at different angles. The information of the surface of an object is collected when the object is scanned, laser rays are projected to the surface of the object by using a laser, the laser rays are collected by a camera after being reflected by the surface of the object, and the laser stripes on the collected image can provide the information of dense object surface points.
In the common binocular vision shooting range, three-dimensional coordinate data of all points in a space are calculated through different visual angle image matching and triangulation principles, the precision of three-dimensional data obtained by a stereo camera completely depends on the precision of image matching, the precision of image matching is greatly influenced by factors such as illumination, a measured space and the like, simultaneously, because line laser is emitted from the top or the side, only object three-dimensional information in a certain visual plane can be obtained through one-time detection, and certain blind areas exist due to the fact that omnibearing shooting cannot be carried out, and the precision of three-dimensional detection is seriously influenced. The laser scanner can only obtain the three-dimensional coordinate data of points on the laser line, the laser scanner or a scanned object needs to be moved, the three-dimensional data of all the points in the scanned space can be obtained through splicing, and the cost of the whole system is high; in summary, no three-dimensional data acquisition product with high precision, low price, large shooting range and spatial resolution exists in the market at present.
Disclosure of Invention
The invention aims to solve the technical problem of providing a multi-binocular camera and line laser cooperative detection device and method aiming at the defects involved in the background technology.
The purpose of the invention and the technical problem to be solved are realized by adopting the following technical scheme:
a multi-binocular camera and line laser cooperative detection method comprises the following steps:
s1: the method is characterized in that a platform where a measured object is located is used as a reference surface, a binocular camera module consisting of a binocular camera and a line laser emitter is erected above the reference surface, the binocular camera can shoot laser lines, and the positions of the binocular camera and the line laser emitter are kept relatively fixed.
S2: and carrying out three-dimensional calibration on the left camera and the right camera of each binocular camera.
S3: each binocular camera module respectively acquires a left image and a right image of a measured object through a left camera and a right camera of a binocular camera, and performs stereo correction on the left image and the right image;
matching the corrected left view and the corrected right view to obtain a linear laser imaging matching point pair;
and obtaining left and right view parallax according to the line laser matching point pairs, and calculating to obtain three-dimensional information of the object to be measured according to the left and right view parallax.
S4: and (4) carrying out coordinate conversion of the pose relationship on the binocular cameras, namely converting the pose relationship into the same coordinate system.
S5: and fusing the real-time depth image data and the real-time attitude data of the measured object acquired by each binocular camera module at the same moment, and respectively calculating the three-dimensional positions of the corresponding calibration objects in the coordinate systems of the binocular stereo cameras to obtain complete three-dimensional data.
Preferably, in the step S1, the binocular camera modules are installed on the same plane of the platform of the object to be measured and are installed in a linear arrangement.
Preferably, in step S1, the number of the binocular camera modules is two or more.
Preferably, in the step S1, the line laser emitter is built in the binocular camera, or the line laser emitter is externally hung on the binocular camera and forms a common structure with the binocular camera, or the line laser emitter and the binocular camera are installed in a split manner.
Preferably, in the step S2, the left camera and the right camera of each binocular camera are calibrated stereoscopically; including the calculation of the internal reference matrix K and the distortion matrix D:
for each image and the corresponding calibration checkerboard angular point, the least square solution of B can be obtained by a matrix method, and the camera internal parameter K is further obtained. Further solving an optimal solution according to the checkerboard angular points of the plurality of images, wherein the optimal solution is a solution which enables the following equation to be minimum:
considering the radial distortion, the above optimization problem translates into a minimization problem of the following formula:
and (3) by using a Levenberg-Marquardt method, and using the solution obtained from the first image as an initial value, thereby obtaining an optimal solution by iteration.
Preferably, in the step S2, the binocular camera modules work cooperatively when beginning to work, and the timing synchronization is maintained, that is, the binocular camera modules are set to be triggered simultaneously by an external trigger signal at the same time, or an upper computer triggers an incoming flow mode of each binocular camera, and the timers of each binocular stereo camera are set to be cleared at the same time, and a master camera sends a trigger signal and works simultaneously after receiving the trigger signal from the slave camera.
Preferably, in the step S4, the distance in the x-axis direction between the coordinate systems to which the binocular camera modules belong is calculated according to the distance between the binocular camera modules and the moving speed of the object to be measured, and the three-dimensional coordinates of the object to be measured in the coordinate systems of the binocular camera modules are obtained from the distance in the x-axis direction.
Preferably, in the step S4, the binocular stereo camera module respectively acquires three-dimensional data of the object to be measured, and stereo-matches the same characteristics of each group of views with the multiple groups of three-dimensional data using the unified calibration coordinate system as an origin to obtain the entire three-dimensional data of the object to be measured.
Advantageous effects
The invention adopts a plurality of groups of line lasers and binocular camera modules which are arranged in parallel on a plane for object detection, effectively utilizes the high brightness and high linearity of the lasers to improve the anti-interference capability of the measuring method, and carries out three-dimensional reconstruction and data fusion after acquiring three-dimensional coordinates.
Drawings
Fig. 1 is a schematic flow chart of a multi-binocular camera and line laser cooperative detection method.
Detailed Description
The technical solution of the present invention is further described in detail below with reference to the accompanying drawings. It is obvious that the described embodiments are only a part of the embodiments of the present invention, not all embodiments, and all other embodiments obtained by those skilled in the art without inventive efforts belong to the protection scope of the present invention.
It should be understood that in the description of the present invention, it should be noted that the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings or orientations or positional relationships conventionally used in the product of the present invention, which are only used for convenience in describing the present invention and simplifying the description, but do not indicate or imply that the device or element referred to must have a specific orientation, be constructed in a specific orientation, and be operated, and thus should not be construed as limiting the present invention. Furthermore, although the terms first, second, third, etc. may be used herein to describe various elements, components and/or sections, these elements, components and/or sections should not be limited by these terms.
In this embodiment, a method for detecting cooperation of multiple binocular cameras and line laser includes the following steps:
s1: the method is characterized in that a platform where a measured object is located is used as a reference surface, a binocular camera module consisting of a binocular camera and a line laser emitter is erected above the reference surface, the binocular camera can shoot laser lines, and the positions of the binocular camera and the line laser emitter are kept relatively fixed.
S2: and carrying out three-dimensional calibration on the left camera and the right camera of each binocular camera.
S3: each binocular camera module respectively acquires a left image and a right image of a measured object through a left camera and a right camera of a binocular camera, and performs stereo correction on the left image and the right image;
matching the corrected left view and the corrected right view to obtain a linear laser imaging matching point pair;
and obtaining left and right view parallax according to the line laser matching point pairs, and calculating to obtain three-dimensional information of the object to be measured according to the left and right view parallax.
S4: and (4) carrying out coordinate conversion of the pose relationship on the binocular cameras, namely converting the pose relationship into the same coordinate system.
S5: and fusing the real-time depth image data and the real-time attitude data of the measured object acquired by each binocular camera module at the same moment, and respectively calculating the three-dimensional positions of the corresponding calibration objects in the coordinate systems of the binocular stereo cameras to obtain complete three-dimensional data.
The method comprises the steps of adopting a module consisting of a plurality of groups of binocular cameras and line lasers to collect images of a measured object and the line lasers in real time, sending collected image information to an image processor for processing, completing the stereo matching of left and right images, performing three-dimensional reconstruction and data fusion of coordinates of each binocular camera module, and fitting complete three-dimensional pose information of the measured object.
Preferably, in the step S1, the binocular camera modules are installed on the same plane of the platform of the object to be measured and are installed in a linear arrangement.
Preferably, in step S1, the number of the binocular camera modules is two or more.
Preferably, in the step S1, the line laser emitter is built in the binocular camera, or the line laser emitter is externally hung on the binocular camera and forms a common structure with the binocular camera, or the line laser emitter and the binocular camera are installed in a split manner.
In step S2, performing stereo calibration on the left camera and the right camera of each binocular camera; including the calculation of the internal reference matrix K and the distortion matrix D:
for each image and the corresponding calibration checkerboard angular point, the least square solution of B can be obtained by a matrix method, and the camera internal parameter K is further obtained. Further solving an optimal solution according to the checkerboard angular points of the plurality of images, wherein the optimal solution is a solution which enables the following equation to be minimum:
considering the radial distortion, the above optimization problem translates into a minimization problem of the following formula:
and (3) by using a Levenberg-Marquardt method, and using the solution obtained from the first image as an initial value, thereby obtaining an optimal solution by iteration.
Preferably, in step S2, the left camera and the right camera of the binocular camera are subjected to stereo calibration to obtain an internal reference matrix a of the binocular camera, a rotation matrix R between the left camera and the right camera, and a translation vector T:
respectively calibrating a left camera and a right camera of the binocular camera to obtain an internal reference matrix A of the binocular camera, a rotation matrix R1 of the left camera and a rotation matrix Rr of the right camera, and a translation vector T1 of the left camera and a translation vector Tr of the right camera;
and calculating a rotation matrix R and a translation vector T between the left camera and the right camera by the following formula:
preferably, in the step S2, the binocular camera modules work cooperatively when beginning to work, and the timing synchronization is maintained, that is, the binocular camera modules are set to be triggered simultaneously by an external trigger signal at the same time, or an upper computer triggers an incoming flow mode of each binocular camera, and the timers of each binocular stereo camera are set to be cleared at the same time, and a master camera sends a trigger signal and works simultaneously after receiving the trigger signal from the slave camera.
In this embodiment, performing stereo correction on the left steel bar end face contour image and the right steel bar end face contour image includes:
decomposing the rotation matrix R into two rotation matrices R1 and rr, wherein R1 and rr are obtained by assuming that the optical axes of the left camera and the right camera are parallel by rotating each of the left camera and the right camera by half;
aligning the left steel bar end face outline image and the right steel bar end face outline image by the following formula:
where Rrect is a rotation matrix that aligns the rows:
the rotation matrix Rrect starts from the direction of the pole e1, the origin of the profile image of the end face of the left steel bar is taken as the main point, and the direction of the translation vector of the left camera to the right camera is taken as the direction of the main point:
e1and e2Is orthogonal to e1Normalized to unit vector:
wherein Tx is a component of the translation vector T in the horizontal direction in the plane where the binocular camera is located, and Ty is a component of the translation vector T in the vertical direction in the plane where the binocular camera is located;
e3 is orthogonal to e1 and e2, and e3 is calculated by the following formula:
e3＝e2×e1
according to the physical significance of the rotation matrix, the method comprises the following steps:
wherein alpha is the angle of the left camera and the right camera which need to rotate in the plane where the left camera and the right camera are located, and alpha is more than or equal to 0 and less than or equal to 180 degrees; the left camera is rotated by α' about the e3 direction, and the right camera is rotated by α "about the e3 direction.
Preferably, in the step S4, the distance in the x-axis direction between the coordinate systems to which the binocular camera modules belong is calculated according to the distance between the binocular camera modules and the moving speed of the object to be measured, and the three-dimensional coordinates of the object to be measured in the coordinate systems of the binocular camera modules are obtained from the distance in the x-axis direction.
Preferably, in the step S4, the binocular stereo camera module respectively acquires three-dimensional data of the object to be measured, and stereo-matches the same characteristics of each group of views with the multiple groups of three-dimensional data using the unified calibration coordinate system as an origin to obtain the entire three-dimensional data of the object to be measured.
Advantageous effects
The invention adopts a plurality of groups of line lasers and binocular camera modules which are arranged in parallel on a plane for object detection, effectively utilizes the high brightness and high linearity of the lasers to improve the anti-interference capability of the measuring method, and carries out three-dimensional reconstruction and data fusion after acquiring three-dimensional coordinates.
The preferred embodiments of the present disclosure have been disclosed to assist in describing the disclosure, and alternative embodiments have not been set forth in detail to avoid obscuring the invention in the particular embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the specification and its practical application, to thereby enable others skilled in the art to best understand the specification and its practical application. The specification is limited only by the claims and their full scope and equivalents.
Claims (8)
1. A multi-binocular camera and line laser cooperative detection method is characterized by comprising the following steps:
s1: the method is characterized in that a platform where a measured object is located is used as a reference surface, a binocular camera module consisting of a binocular camera and a line laser emitter is erected above the reference surface, the binocular camera can shoot laser lines, and the positions of the binocular camera and the line laser emitter are kept relatively fixed.
S2: and carrying out three-dimensional calibration on the left camera and the right camera of each binocular camera.
S3: each binocular camera module respectively acquires a left image and a right image of a measured object through a left camera and a right camera of a binocular camera, and performs stereo correction on the left image and the right image;
matching the corrected left view and the corrected right view to obtain a linear laser imaging matching point pair;
and obtaining left and right view parallax according to the line laser matching point pairs, and calculating to obtain three-dimensional information of the object to be measured according to the left and right view parallax.
S4: and (4) carrying out coordinate conversion of the pose relationship on the binocular cameras, namely converting the pose relationship into the same coordinate system.
S5: and fusing the real-time depth image data and the real-time attitude data of the measured object acquired by each binocular camera module at the same moment, and respectively calculating the three-dimensional positions of the corresponding calibration objects in the coordinate systems of the binocular stereo cameras to obtain complete three-dimensional data.
2. The multi-binocular camera and line laser cooperative detection method according to claim 1, wherein in the step S1, the binocular camera modules are installed on the same plane of the platform of the object to be detected and arranged in a straight line.
3. The multi-binocular camera and line laser cooperative detection method according to claim 1, wherein in the step S1, the number of the binocular camera modules is two or more.
4. The multi-binocular camera and line laser cooperative detection method according to claim 1, wherein in the step S1, the line laser emitter is built in the binocular camera, or the line laser emitter is externally hung on the binocular camera and forms a common structure with the binocular camera, or the line laser emitter and the binocular camera are installed in a split manner.
5. The multi-binocular camera and line laser cooperative detection method according to claim 1, wherein in the step S2, a left camera and a right camera of each binocular camera are calibrated stereoscopically; including the calculation of the internal reference matrix K and the distortion matrix D:
for each image and the corresponding calibration checkerboard angular point, the least square solution of B can be obtained by a matrix method, and the camera internal parameter K is further obtained. Further solving an optimal solution according to the checkerboard angular points of the plurality of images, wherein the optimal solution is a solution which enables the following equation to be minimum:
considering the radial distortion, the above optimization problem translates into a minimization problem of the following formula:
and (3) by using a Levenberg-Marquardt method, and using the solution obtained from the first image as an initial value, thereby obtaining an optimal solution by iteration.
6. The method for detecting the coordination of the multiple binocular cameras and the line laser according to claim 1, wherein in the step S2, the binocular camera modules work cooperatively when starting to work, and the timing synchronization is maintained, that is, the binocular camera modules are set to be simultaneously triggered by an external trigger signal at the same time, or an upper computer triggers the binocular cameras to enter the stream mode, and the binocular stereo camera timers are simultaneously set to be zero at the same time, and one master camera sends a trigger signal and works simultaneously after receiving the trigger signal from the cameras.
7. The method for detecting the coordination of the multi-binocular camera and the line laser according to claim 1, wherein in the step S4, the distance between the coordinate systems of the binocular camera modules in the x-axis direction is calculated according to the distance between the binocular camera modules and the moving speed of the object to be detected, and the three-dimensional coordinates of the object to be detected in the coordinate systems of the binocular camera modules are obtained according to the distance in the x-axis direction.
8. The multi-binocular camera and line laser cooperative detection method according to claim 1, wherein in the step S4, the binocular stereo camera module respectively acquires three-dimensional data of the object to be detected, and stereo-matches the same features of each group of views with a unified calibration coordinate system as an origin to acquire the overall three-dimensional data of the object to be detected.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910793994.XA CN111829435A (en) | 2019-08-27 | 2019-08-27 | Multi-binocular camera and line laser cooperative detection method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910793994.XA CN111829435A (en) | 2019-08-27 | 2019-08-27 | Multi-binocular camera and line laser cooperative detection method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111829435A true CN111829435A (en) | 2020-10-27 |
Family
ID=72911655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910793994.XA Pending CN111829435A (en) | 2019-08-27 | 2019-08-27 | Multi-binocular camera and line laser cooperative detection method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111829435A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112461127A (en) * | 2020-11-05 | 2021-03-09 | 渤海造船厂集团有限公司 | Hyperbolic outer plate forming detection device and method |
| CN113269823A (en) * | 2021-05-18 | 2021-08-17 | Oppo广东移动通信有限公司 | Depth data acquisition method and device, storage medium and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103971353A (en) * | 2014-05-14 | 2014-08-06 | 大连理工大学 | Splicing method for measuring image data with large forgings assisted by lasers |
| CN107063129A (en) * | 2017-05-25 | 2017-08-18 | 西安知象光电科技有限公司 | A kind of array parallel laser projection three-dimensional scan method |
| CN107621226A (en) * | 2017-07-18 | 2018-01-23 | 深圳大学 | The 3-D scanning method and system of multi-view stereo vision |
| CN107907048A (en) * | 2017-06-30 | 2018-04-13 | 长沙湘计海盾科技有限公司 | A kind of binocular stereo vision method for three-dimensional measurement based on line-structured light scanning |
| CN109974623A (en) * | 2018-09-13 | 2019-07-05 | 北京伟景智能科技有限公司 | 3 D information obtaining method and device based on line laser and binocular vision |
-
2019
- 2019-08-27 CN CN201910793994.XA patent/CN111829435A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103971353A (en) * | 2014-05-14 | 2014-08-06 | 大连理工大学 | Splicing method for measuring image data with large forgings assisted by lasers |
| CN107063129A (en) * | 2017-05-25 | 2017-08-18 | 西安知象光电科技有限公司 | A kind of array parallel laser projection three-dimensional scan method |
| CN107907048A (en) * | 2017-06-30 | 2018-04-13 | 长沙湘计海盾科技有限公司 | A kind of binocular stereo vision method for three-dimensional measurement based on line-structured light scanning |
| CN107621226A (en) * | 2017-07-18 | 2018-01-23 | 深圳大学 | The 3-D scanning method and system of multi-view stereo vision |
| CN109974623A (en) * | 2018-09-13 | 2019-07-05 | 北京伟景智能科技有限公司 | 3 D information obtaining method and device based on line laser and binocular vision |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112461127A (en) * | 2020-11-05 | 2021-03-09 | 渤海造船厂集团有限公司 | Hyperbolic outer plate forming detection device and method |
| CN113269823A (en) * | 2021-05-18 | 2021-08-17 | Oppo广东移动通信有限公司 | Depth data acquisition method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110044300B (en) | Amphibious three-dimensional vision detection device and detection method based on laser | |
| CN111750806B (en) | Multi-view three-dimensional measurement system and method | |
| CN102364299B (en) | Calibration technology for multiple structured light projected three-dimensional profile measuring heads | |
| CN111243002A (en) | Monocular laser speckle projection system calibration and depth estimation method applied to high-precision three-dimensional measurement | |
| TWI555379B (en) | An image calibrating, composing and depth rebuilding method of a panoramic fish-eye camera and a system thereof | |
| US7196719B2 (en) | Angled axis machine vision system and method | |
| CN111369630A (en) | Method for calibrating multi-line laser radar and camera | |
| CN111854636B (en) | Multi-camera array three-dimensional detection system and method | |
| CN111815716A (en) | Parameter calibration method and related device | |
| CN109827521B (en) | Calibration method for rapid multi-line structured optical vision measurement system | |
| CN109544628B (en) | Accurate reading identification system and method for pointer instrument | |
| CN107941153B (en) | Visual system for optimizing calibration of laser ranging | |
| CN114998499A (en) | Binocular three-dimensional reconstruction method and system based on line laser galvanometer scanning | |
| CN110763140B (en) | Non-parallel optical axis high-precision binocular ranging method | |
| CN111854622B (en) | Large-field-of-view optical dynamic deformation measurement method | |
| CN108180888A (en) | A kind of distance detection method based on rotating pick-up head | |
| CN111829435A (en) | Multi-binocular camera and line laser cooperative detection method | |
| CN111721194A (en) | Multi-laser-line rapid detection method | |
| Su et al. | Obtaining obstacle information by an omnidirectional stereo vision system | |
| CN112686960B (en) | Method for calibrating entrance pupil center and sight direction of camera based on ray tracing | |
| CN209978890U (en) | Multi-laser-line rapid detection system | |
| CN114359365A (en) | Convergent binocular vision measuring method with high resolution | |
| CN112785647A (en) | Three-eye stereo image detection method and system | |
| Zeng et al. | A 3D passive optical localization system based on binocular infrared cameras | |
| CN111462171A (en) | Mark point detection tracking method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201027 |