CN115082559B - Multi-target intelligent sorting method and system for flexible parts and storage medium - Google Patents
Multi-target intelligent sorting method and system for flexible parts and storage medium Download PDFInfo
- Publication number
- CN115082559B CN115082559B CN202210851072.1A CN202210851072A CN115082559B CN 115082559 B CN115082559 B CN 115082559B CN 202210851072 A CN202210851072 A CN 202210851072A CN 115082559 B CN115082559 B CN 115082559B
- Authority
- CN
- China
- Prior art keywords
- nickel sheet
- target
- nickel
- point cloud
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
- B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
- B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
- B07C5/36—Sorting apparatus characterised by the means used for distribution
- B07C5/361—Processing or control devices therefor, e.g. escort memory
- B07C5/362—Separating or distributor mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention discloses a multi-target intelligent sorting method, a system and a storage medium for flexible parts, wherein the method comprises the following steps: acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame, inputting the RGB images into a trained example segmentation model, and outputting a mask image of a target nickel sheet after passing through the example segmentation model; aligning the output target mask image with the depth image, dividing a target nickel sheet in the depth image, and generating a target nickel sheet point cloud through camera calibration parameters; registering the target nickel sheet point cloud and the model nickel sheet point cloud to obtain the spatial pose information of the target nickel sheet in the material frame, and sending the spatial pose information of the target nickel sheet in the material frame to a controller to guide the manipulator to a specified position to absorb the nickel sheet; and after the nickel sheet is sucked, the manipulator moves to an industrial camera for deformation detection and secondary fine positioning, and sorting and assembling processes are completed according to the deformation detection and secondary fine positioning results. The invention improves the efficiency and the accuracy of nickel sheet sorting and further ensures the quality of finished lithium battery products.
Description
Technical Field
The invention relates to the technical field of intelligent sorting, in particular to a multi-target intelligent sorting method, a multi-target intelligent sorting system and a storage medium for flexible parts.
Background
With the continuous development of industrial technology, the assembly of lithium batteries is also towards automation and intellectualization. Nickel sheet laminates are an integral part of a modular lithium battery and are often used as conductive and welding parts to connect multiple cells so that the cells can be assembled into a battery. The nickel sheet assembling process of the existing lithium battery assembling line is as follows: the machine vision system discerns the positional information who obtains the nickel piece in the material frame in the location, then sends positional information for control system, and the guiding manipulator goes the assigned position and picks up the nickel piece, then assembles the assigned position to combination formula lithium cell group.
However, nickel sheet assembly has several problems: the nickel sheets are randomly and randomly stacked in the material frame, and the parts are overlapped and shielded. The existing machine vision recognition positioning system can detect a single separated nickel sheet, but the recognition positioning under the condition that a plurality of nickel sheets are overlapped and shielded is difficult to realize. On the other hand, the height information of the nickel sheets in the material frame is unknown due to the overlapping condition of the nickel sheets, and the common 2D plane grabbing mode can cause the nickel sheets to deform or damage the manipulator end effector, so that the production requirement of the scene cannot be met.
In addition, the nickel sheet has the characteristics of complex structure, thin thickness, low strength and easy deformation, and the deformation of the nickel sheet is inevitable in the manufacturing and transportation process, and the lithium battery pack can be scrapped once the deformed nickel sheet does not meet the assembly requirements. In summary, there is a need for an intelligent sorting method for thin nickel pieces to solve the above problems.
Disclosure of Invention
In order to solve at least one technical problem, the invention provides a multi-target intelligent sorting method, a system and a storage medium for flexible parts.
The invention provides a multi-target intelligent sorting method for flexible parts, which comprises the following steps:
acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame through a three-dimensional camera, inputting the RGB images into a trained example segmentation model, and outputting a mask image of a target nickel sheet through the example segmentation model;
aligning the output target mask image with the depth image, dividing a target nickel sheet in the depth image, and generating a target nickel sheet point cloud through camera calibration parameters;
registering the target nickel sheet point cloud and the model nickel sheet point cloud to obtain the spatial pose information of the target nickel sheet in the material frame, and sending the spatial pose information of the target nickel sheet in the material frame to a controller to guide the manipulator to a specified position to absorb the nickel sheet;
and after the nickel sheet is sucked, the mechanical arm moves to an industrial camera for deformation detection and secondary fine positioning, and sorting and assembling processes are completed according to the deformation detection and secondary fine positioning results.
In the scheme, the mask image of the target nickel sheet is output through the example segmentation model, and the method specifically comprises the following steps:
performing labeling definition on the multi-target nickel sheet part image according to a preset nickel sheet label definition strategy, wherein the label is divided into a front side and a back side;
in addition, dividing a positive sample and a negative sample according to the condition that the nickel sheet is shielded, and dividing the nickel sheet into 4 regions according to a preset division standard, wherein the first region is a suction region of a sucker at the tail end of a manipulator, and the second, third and fourth regions are regions for judging the positive and negative sides of the nickel sheet;
when the first area is shielded, the first area is divided into negative samples which are not used as labels; when the first area is not shielded, the second area and the third area are partially shielded, the first area is divided into positive samples, and when the first area is not shielded, the second area and the third area are divided into negative samples;
after nickel sheet labels are determined, generating a training data set according to labeled data, constructing an example segmentation model based on Mask R-CNN, and performing initialization training on the example segmentation model by using the training data set;
and carrying out positive and negative recognition on the nickel sheet in the input image through the trained example segmentation model, calculating a confidence score for the nickel sheet which accords with the label definition strategy, and carrying out mask output on the nickel sheet with the highest confidence score.
In the scheme, the method for acquiring the pose information of the target nickel sheet in the three-dimensional space specifically comprises the following steps:
acquiring a mask image of a target nickel sheet, aligning the mask image with the depth image, mapping pixel position information in the mask image to the depth image, segmenting pixel information of the target nickel sheet in the depth image, and generating a point cloud of the target nickel sheet from the segmented depth image according to calibration parameters of a three-dimensional camera;
generating point cloud by using a three-dimensional model of a nickel sheet, performing point cloud registration through point cloud processing and a registration algorithm, estimating the pose of a target nickel sheet, and acquiring a rigid transformation matrix between the target nickel sheet point cloud and the model nickel sheet point cloud;
and converting the rigid transformation matrix into actual control information according to the hand-eye calibration parameters, and controlling the manipulator to suck the nickel sheets to the corresponding position.
In this scheme, absorb behind the nickel piece manipulator remove industrial camera department and warp detection and secondary fine positioning, accomplish letter sorting and assembling process according to warping the detection and secondary fine positioning result, specifically do:
acquiring a target nickel sheet deformation detection image through an industrial two-dimensional camera, and performing image preprocessing on the target nickel sheet deformation detection image, wherein the preprocessing comprises image graying, threshold segmentation and region-of-interest interception;
determining key characteristics of nickel sheet deformation detection by analyzing the assembly requirements, wherein the key characteristics of nickel sheet deformation detection comprise a reference positioning hole, a positioning edge and a nickel sheet outline circular arc;
edge detection is carried out through a Canny algorithm, a sub-pixel edge positioning technology is introduced, fine detection and fine positioning of the outline edge are achieved, then a least square method is used for fitting the characteristic outline edge, and size information among key characteristics of nickel sheet deformation is calculated;
and judging whether the size information is in a preset size threshold range, if so, judging that the target nickel sheet does not meet the assembly requirement, and if not, assembling to a specified position according to the position information of the fine positioning.
In the scheme, after the manipulator absorbs the target nickel sheet, if the nickel sheet is the reverse side, the nickel sheet needs to be moved to the reversing device for surface changing operation, and then the nickel sheet is moved to the industrial two-dimensional camera for deformation detection and fine positioning. And if the nickel sheet is the front surface, directly moving to a two-dimensional camera for deformation detection and fine positioning.
The second aspect of the invention also provides a multi-target intelligent sorting system for flexible parts, which mainly comprises: the device comprises a memory, a processor, an example segmentation multi-target nickel piece part identification and positioning module, a nickel piece pose estimation module and a nickel piece deformation detection module, wherein a multi-target intelligent sorting method program of a flexible piece is stored and executed in the memory and the processor;
acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame through a three-dimensional camera, segmenting an RGB image input example into a multi-target nickel sheet part identification positioning module to perform front and back identification positioning on the nickel sheets, and outputting mask images of the target nickel sheets;
generating a target nickel sheet point cloud through a nickel sheet pose estimation module, and registering the target nickel sheet point cloud and the model nickel sheet point cloud to acquire the spatial pose information of the target nickel sheet in the material frame;
and carrying out deformation detection and secondary fine positioning through the nickel sheet deformation detection module, and finishing the sorting and assembling process according to the deformation detection and secondary fine positioning results.
In the scheme, the multi-target nickel sheet part identification and positioning module for example segmentation specifically comprises:
the example segmentation multi-target nickel piece part identification positioning module mainly comprises an example segmentation model part and a nickel piece label definition strategy part;
the example segmentation model part is constructed on the basis of Mask R-CNN, positive and negative recognition is carried out on nickel sheets in an input image, confidence coefficient scores are calculated for the nickel sheets meeting a label definition strategy, and the nickel sheets with the highest confidence coefficient scores are subjected to Mask output;
the nickel sheet label definition strategy part carries out labeling definition on the multi-target nickel sheet part images, and the labels are divided into a front surface and a back surface; dividing a positive sample and a negative sample according to the condition that the nickel sheet is shielded, and dividing the nickel sheet into 4 regions according to a preset division standard, wherein the first region is a suction region of a sucker at the tail end of a manipulator, and the second, third and fourth regions are regions for judging the positive and negative sides of the nickel sheet;
when the first area is blocked, the first area is divided into negative samples which are not used as labels; when the first area is not shielded, and the second area and the third area are partially shielded, the first area is divided into positive samples, and when the first area is not shielded, the second area and the third area are divided into negative samples;
after the nickel sheet label is determined, generating a training data set according to the labeling data, and training the example segmentation model through the training data set.
In this scheme, the nickel sheet pose estimation module specifically includes:
the nickel sheet pose estimation module comprises a target nickel sheet point cloud generation part, a model nickel sheet point cloud generation part and a point cloud registration pose estimation part;
the target nickel sheet point cloud generating part acquires a mask image of a target nickel sheet, aligns the mask image with the depth image, maps information in the mask image to the depth image, segments the target nickel sheet in the depth image, and generates a target nickel sheet point cloud from the segmented depth image according to calibration parameters of the three-dimensional camera;
the model nickel sheet point cloud generating part generates point cloud by using a three-dimensional model of a nickel sheet;
the point cloud registration pose estimation part carries out point cloud registration through point cloud processing and a registration algorithm, estimates the pose of the target nickel sheet and obtains a rigid transformation matrix between the target nickel sheet point cloud and the model nickel sheet point cloud.
In this scheme, nickel piece deformation detection module specifically is:
the nickel sheet deformation detection module consists of a nickel sheet deformation evaluation standard part, an image preprocessing part and a key characteristic extraction deformation detection part;
acquiring a target nickel sheet deformation detection image through an industrial two-dimensional camera, and performing image preprocessing on the target nickel sheet deformation detection image through an image preprocessing part, wherein the preprocessing comprises image graying, threshold segmentation and region-of-interest interception;
the nickel sheet deformation evaluation standard part determines key characteristics of nickel sheet deformation detection by analyzing assembly requirements, wherein the key characteristics of nickel sheet deformation detection comprise a reference positioning hole, a positioning edge and a nickel sheet outline circular arc;
the key feature extraction deformation detection part carries out edge detection through a Canny algorithm, introduces a sub-pixel edge positioning technology, realizes precise detection and precise positioning of the contour edge, fits the feature contour edge by using a least square method, and calculates size information between key features of nickel sheet deformation to detect whether the part is deformed.
The third aspect of the invention also provides a computer readable storage medium, which includes a program for the multi-target intelligent sorting method of flexible parts, and when the program is executed by a processor, the steps of the method for the multi-target intelligent sorting of flexible parts are realized.
The invention discloses a multi-target intelligent sorting method, a system and a storage medium for flexible parts, wherein the method comprises the following steps: acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame, inputting the RGB images into a trained example segmentation model, and outputting a mask image of a target nickel sheet after passing through the example segmentation model; aligning the output target mask image with the depth image, dividing a target nickel sheet in the depth image, and generating a target nickel sheet point cloud through camera calibration parameters; registering the target nickel piece point cloud and the model nickel piece point cloud to obtain the spatial position and attitude information of the target nickel piece in the material frame, and sending the spatial position and attitude information of the target nickel piece in the material frame to a controller to guide the manipulator to a specified position to absorb the nickel piece; and after the nickel sheet is sucked, the manipulator moves to an industrial camera for deformation detection and secondary fine positioning, and sorting and assembling processes are completed according to the deformation detection and secondary fine positioning results. The method can accurately identify and position the target nickel sheet from a scene shielded by overlapping of a plurality of nickel sheets, and acquire the space pose information of the target nickel sheet, thereby facilitating the absorption of a mechanical arm and improving the efficiency and accuracy of nickel sheet sorting. In order to further ensure the quality of the finished lithium battery, the nickel sheet is added before assembly for deformation detection, so that the qualification rate of the finished lithium battery can be improved, and the rejection rate can be reduced.
Drawings
FIG. 1 is a flow chart of a multi-target intelligent sorting method for flexible parts according to the invention;
FIG. 2 is a schematic diagram showing the division of a nickel plate into 4 regions according to the present invention;
FIG. 3 is a block diagram of a multi-objective intelligent sorting system for flexible articles according to the present invention;
FIG. 4 is a block diagram of an exemplary segmented multi-target nickel sheet part identification and location module of the present invention;
FIG. 5 shows a block diagram of a nickel plate pose estimation module in the present invention;
FIG. 6 shows a block diagram of a nickel plate deformation detection module in the present invention.
Detailed Description
In order that the above objects, features and advantages of the present invention can be more clearly understood, a more particular description of the invention will be rendered by reference to the appended drawings. It should be noted that the embodiments and features of the embodiments of the present application may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, however, the present invention may be practiced in other ways than those specifically described herein, and therefore the scope of the present invention is not limited by the specific embodiments disclosed below.
FIG. 1 is a flow chart of a multi-target intelligent sorting method for flexible parts.
As shown in fig. 1, a first aspect of the present invention provides a multi-target intelligent sorting method for flexible parts, including:
starting a system, initializing the system, judging whether the system is in a standby state, if the system is in a non-standby state, waiting, if the system is in the standby state, photographing by a three-dimensional camera, and photographing by the three-dimensional camera to obtain an RGB (red, green and blue) image and a depth image of a nickel sheet in a material frame;
inputting the RGB image into a trained example segmentation model, judging whether nickel sheets exist in a material frame by the example segmentation model, if not, informing a controller to discharge materials, if so, sequencing according to confidence scores of the nickel sheets, and outputting a nickel sheet mask image with the highest confidence score;
mapping pixel position information in the nickel sheet mask image into a depth map, segmenting the pixel information of the nickel sheet in the depth map, and converting the depth map into point cloud through three-dimensional camera calibration parameters;
registering the target nickel sheet point cloud and the model nickel sheet point cloud, estimating the pose of the target nickel sheet, and acquiring a rigid transformation matrix between the target nickel sheet point cloud and the model nickel sheet point cloud;
and converting the rigidity transformation matrix into manipulator control parameters, and controlling the manipulator to absorb the target nickel sheet. In order to shorten the running time of the whole process, the embodiment segmentation, identification and positioning module and the pose estimation module also carry out the identification and positioning of the next target nickel sheet while the manipulator absorbs the nickel sheet to carry out deformation detection and fine positioning and assembly;
after a target nickel sheet is sucked, if the nickel sheet is the back side, the nickel sheet needs to be moved to a reversing device for surface changing operation, then the nickel sheet is moved to a two-dimensional camera for deformation detection and fine positioning, and if the nickel sheet is the front side, the nickel sheet is directly moved to the two-dimensional camera for deformation detection and fine positioning;
the nickel sheet deformation detection module firstly judges whether the nickel sheet meets the assembly requirement, if the nickel sheet does not meet the assembly requirement, the nickel sheet is sorted to a scrapped frame, and if the nickel sheet meets the assembly requirement, the nickel sheet is assembled to the designated position of the lithium battery pack according to the position information of fine positioning.
The method comprises the following steps that after a three-dimensional camera is used for obtaining images of multi-target nickel sheet parts in a material frame, the images are input into a trained example segmentation model, the example segmentation model can classify each nickel sheet in the images, the position of the nickel sheet in the images is framed by a boundary frame, and a pixel-level mask is generated; labeling definition is carried out on the multi-target nickel piece part images according to a preset nickel piece label definition strategy, wherein the nickel piece has a front surface and a back surface in a material frame, and the labels are divided into the front surface and the back surface; in addition, the positive sample and the negative sample are divided according to the condition that the nickel sheet is shielded, and the nickel sheet is divided into 4 areas according to a preset division standard, as shown in fig. 2, wherein the first area is a suction area of a sucker at the tail end of a manipulator, and the second area, the third area and the fourth area are areas for judging the positive and negative sides of the nickel sheet; when the first area is blocked, the first area is divided into negative samples which are not used as labels; when the first area is not shielded, and the second area and the third area are partially shielded, the first area is divided into positive samples, and when the first area is not shielded, the second area and the third area are divided into negative samples;
after nickel sheet labels are determined, generating a training data set according to labeled data, constructing an example segmentation model based on Mask R-CNN, and performing initialization training on the example segmentation model by using the training data set; and carrying out positive and negative recognition on the nickel sheet in the input image through the trained example segmentation model, calculating a confidence score for the nickel sheet which accords with the label definition strategy, and carrying out mask output on the nickel sheet with the highest confidence score.
It should be noted that the obtaining of pose information of the target nickel sheet in the three-dimensional space specifically includes: acquiring a two-dimensional image of a mask of a target nickel sheet, aligning the mask image with the depth image, mapping pixel position information in the mask image to the depth image, segmenting pixel information of the target nickel sheet in the depth image, and generating a point cloud of the target nickel sheet from the segmented depth image according to calibration parameters of a three-dimensional camera;
generating Point clouds by using a three-dimensional model of a nickel sheet, performing Point Cloud registration by using a Point Cloud processing and registration algorithm, estimating the pose of a target nickel sheet, and acquiring a rigid transformation matrix between the Point clouds of the target nickel sheet and the Point clouds of model nickel sheets, wherein the nickel sheet is produced by a mould, and the size of each nickel sheet has no large error, so that the Point clouds can be generated by using the three-dimensional model of the nickel sheet, preferably, the three-dimensional model of the nickel sheet is exported to a Standard Triangular Language (STL) by using SolidWorks three-dimensional design software, and then, the STL file is converted into a Point Cloud Library (PCL); in the registration process, the point cloud voxelization grid downsampling filtering algorithm is used for reducing the number of the target nickel sheet point cloud and the model nickel sheet point cloud, and the registration speed and the registration accuracy can be improved. And then calculating surface normal characteristics of the target nickel sheet point cloud and the model nickel sheet point cloud, performing registration by adopting a fast point characteristic histogram (FPFH) local characteristic description operator, using a sampling consistency (SAC-IA) algorithm for coarse registration, and using an Iterative Closest Point (ICP) algorithm for fine registration. And converting the rigid transformation matrix into actual control information according to the hand-eye calibration parameters, and controlling the manipulator to suck the nickel sheets to the corresponding position.
It should be noted that after the nickel sheet is sucked, the mechanical arm moves to an industrial camera for deformation detection and secondary fine positioning, and sorting and assembling processes are completed according to the deformation detection and secondary fine positioning results, which specifically comprises the following steps:
acquiring a target nickel sheet deformation detection image through an industrial two-dimensional camera, and carrying out image preprocessing on the target nickel sheet deformation detection image, wherein the preprocessing comprises image graying, threshold segmentation and region-of-interest interception, the optimal threshold segmentation adopts a self-adaptive global threshold segmentation method, and the region-of-interest interception is acquired based on a connected domain area method; determining nickel sheet deformation detection key characteristics by analyzing the assembly requirements, wherein the nickel sheet deformation detection key characteristics comprise a reference positioning hole, a positioning edge and a nickel sheet outline circular arc; edge detection is carried out through a Canny algorithm, a sub-pixel edge positioning technology is introduced, fine detection and fine positioning of the outline edge are achieved, then a least square method is used for fitting the characteristic outline edge, and size information among key characteristics of nickel sheet deformation is calculated; and judging whether the size information is in a preset size threshold range, if so, judging that the target nickel sheet does not meet the assembly requirement, and if not, assembling to a specified position according to the position information of the fine positioning.
According to the embodiment of the invention, whether secondary processing can be carried out on the nickel sheet which does not meet the assembly requirement is judged according to the characteristics, and the method specifically comprises the following steps:
judging the position information of deformation according to the deformation key characteristics of the nickel sheet which does not meet the assembly requirements;
obtaining size information between deformation key features of the nickel sheet which does not meet the assembly requirement, and comparing and judging the size information with size information corresponding to a standard nickel sheet to generate size deviation;
establishing an evaluation system according to a grading standard by matching 4 areas divided by the nickel sheets with a preset deviation interval, and evaluating through the evaluation system according to the position information and the size deviation to generate an evaluation score of the nickel sheets which do not meet the assembly requirement;
and judging whether the nickel sheets which do not meet the assembly requirement can be subjected to secondary processing or not according to the evaluation score, if the evaluation score is greater than or equal to a preset threshold value, proving that the secondary processing can be performed, sorting again after the secondary processing is completed, and if the evaluation score is less than the preset threshold value, scrapping the nickel sheets which do not meet the assembly requirement.
According to the embodiment of the invention, in order to avoid damage caused by falling of redundant nickel sheets after a plurality of nickel sheets are grabbed at one time due to the connection of the nickel sheets in the grabbing process of the manipulator caused by the stacking of the nickel sheets, the grabbing sequence is set according to the point cloud of the nickel sheets in the material frame, and the method specifically comprises the following steps:
acquiring point cloud under the state that the nickel sheets are tiled and not stacked according to the model nickel sheet point cloud, extracting variance in the Z-axis direction in a three-dimensional space through the point cloud, and acquiring the maximum value of the variance as a nickel sheet point cloud threshold;
obtaining a nickel sheet mask image in the material frame through an example segmentation model, obtaining a nickel sheet point cloud, and calculating the variance and mean value of the nickel sheet point cloud in the Z-axis direction;
judging whether the number of nickel sheets is greater than 1 according to the nickel sheet mask image, if so, judging whether the variance of the nickel sheet point cloud in the Z-axis direction is greater than the threshold of the nickel sheet point cloud;
if the average value is larger than the first threshold value, performing reverse sorting according to the average value to generate a first sorting, and if the average value is smaller than the second threshold value, performing reverse sorting according to the average value to generate a second sorting;
and adding the second sequence to the first sequence to generate a final grabbing sequence.
FIG. 3 is a block diagram of a multi-target intelligent sorting system for flexible articles according to the present invention.
The second aspect of the invention also provides a multi-target intelligent sorting system for flexible parts, which mainly comprises: the device comprises a memory, a processor, an example segmentation multi-target nickel piece part identification and positioning module, a nickel piece pose estimation module and a nickel piece deformation detection module, wherein a multi-target intelligent sorting method program of a flexible piece is stored and executed in the memory and the processor;
acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame through a three-dimensional camera, segmenting an RGB image input example into a multi-target nickel sheet part identification positioning module to perform front and back identification positioning on the nickel sheets, and outputting mask images of the target nickel sheets;
generating a target nickel piece point cloud through a nickel piece pose estimation module, and registering the target nickel piece point cloud and the model nickel piece point cloud to acquire space pose information of a target nickel piece in a material frame;
and carrying out deformation detection and secondary fine positioning through the nickel sheet deformation detection module, and finishing the sorting and assembling process according to the deformation detection and secondary fine positioning results.
FIG. 4 shows a block diagram of an example segmented multi-target nickel sheet part identification and location module in accordance with the present invention.
The embodiment segmentation multi-target nickel sheet part identification and positioning module mainly solves the problems of overlapping shielding between nickel sheets and identification and positioning of the front and back surfaces of the nickel sheets, and mainly comprises an embodiment segmentation model part and a nickel sheet label definition strategy part; the example segmentation model part is constructed based on Mask R-CNN, positive and negative identification is carried out on nickel sheets in an input image, confidence coefficient scores are calculated for the nickel sheets meeting a label definition strategy, and the nickel sheets with the highest confidence coefficient scores are subjected to Mask output;
the nickel sheet label definition strategy part is used for performing labeling definition on multi-target nickel sheet part images, and the labels are divided into a front side and a back side; dividing a positive sample and a negative sample according to the shielded condition of the nickel sheet, and dividing the nickel sheet into 4 areas according to a preset division standard, wherein the first area is a suction area of a sucker at the tail end of a manipulator, and the second area, the third area and the fourth area are areas for judging the positive and negative sides of the nickel sheet; when the first area is blocked, the first area is divided into negative samples which are not used as labels; when the first area is not shielded, the second area and the third area are partially shielded, the first area is divided into positive samples, and when the first area is not shielded, the second area and the third area are divided into negative samples; after the nickel sheet label is determined, a training data set is generated according to the labeling data, and the example segmentation model is trained through the training data set. The trained example segmentation model can be used for carrying out positive and negative recognition on nickel sheets in an input image, a confidence score is provided for the nickel sheets meeting the label definition strategy, the higher the confidence score is, the more the accuracy of a prediction result can be represented, and the model finally outputs the nickel sheet mask with the highest confidence score.
FIG. 5 shows a block diagram of the nickel plate pose estimation module of the present invention.
The nickel sheet pose estimation module mainly solves the problem that the height information of a nickel sheet is unknown, and the 3D point cloud-based nickel sheet pose estimation module mainly acquires pose information of a target nickel sheet in a three-dimensional space, so that the manipulator can conveniently absorb the pose information. The nickel sheet pose estimation module consists of a target nickel sheet point cloud generation part, a model nickel sheet point cloud generation part and a point cloud registration pose estimation part;
the target nickel sheet point cloud generating part acquires a two-dimensional image of a target nickel sheet mask, aligns the mask image with the depth image, maps information in the mask image to the depth image, segments a target nickel sheet in the depth image, and generates a target nickel sheet point cloud from the segmented depth image according to calibration parameters of the three-dimensional camera;
the model nickel sheet point cloud generating part generates point cloud by using a three-dimensional model of a nickel sheet; because the nickel sheets are produced through the die, the size of each nickel sheet has no large error, and therefore, the point cloud is generated by using the three-dimensional model of the nickel sheets. Preferably, a SolidWorks three-dimensional design software is used for exporting a three-dimensional model of the nickel sheet into a Standard Triangular Language (STL), and then a Point Cloud Library (PCL) is used for converting the STL file into a Point Cloud file;
the point cloud registration pose estimation part carries out point cloud registration through point cloud processing and a registration algorithm, estimates the pose of a target nickel sheet and obtains a rigid transformation matrix between the target nickel sheet point cloud and a model nickel sheet point cloud. And then calculating surface normal characteristics of the target nickel sheet point cloud and the model nickel sheet point cloud, performing registration by adopting a fast point characteristic histogram (FPFH) local characteristic description operator, using a sampling consistency (SAC-IA) algorithm for coarse registration, and using an Iterative Closest Point (ICP) algorithm for fine registration.
FIG. 6 shows a block diagram of a nickel plate deformation detection module according to the present invention.
It should be noted that, the nickel sheet deformation detection module mainly solves the problem that the nickel sheet deformation affects the quality of the finished lithium battery pack, and after the manipulator sucks the nickel sheet, the nickel sheet deformation detection is required. The nickel sheet deformation detection module consists of a nickel sheet deformation evaluation standard part, an image preprocessing part and a key characteristic extraction deformation detection part;
acquiring a target nickel sheet deformation detection image through an industrial two-dimensional camera, and performing image preprocessing on the target nickel sheet deformation detection image through an image preprocessing part, wherein the preprocessing comprises image graying, threshold segmentation and region-of-interest interception; because the original image has a large amount of information and certain noise, the direct subsequent processing can affect the processing speed and the detection precision of the image, and therefore a series of preprocessing is required. The preprocessing can reduce the data volume of the image, simultaneously ensure that the image characteristics are not lost, improve the image processing time and meet the requirement of the real-time performance of the system. Preferably, the threshold segmentation adopts a self-adaptive global threshold segmentation method, and ROI interception adopts a method based on a connected domain area;
the nickel sheet deformation evaluation standard part determines nickel sheet deformation detection key characteristics by analyzing assembly requirements, wherein the nickel sheet deformation detection key characteristics comprise a reference positioning hole, a positioning edge and a nickel sheet outline circular arc;
the key feature extraction deformation detection part carries out edge detection through a Canny algorithm, introduces a sub-pixel edge positioning technology, realizes precise detection and precise positioning of the contour edge, fits the feature contour edge by using a least square method, and calculates size information between key features of nickel sheet deformation to detect whether the part is deformed.
The third aspect of the present invention further provides a computer readable storage medium, which includes a program for a method for multi-target intelligent sorting of flexible parts, and when the program is executed by a processor, the method for multi-target intelligent sorting of flexible parts realizes the steps of the method for multi-target intelligent sorting of flexible parts as described in any one of the above.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described device embodiments are merely illustrative, for example, the division of the unit is only one logical function division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as one unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in the form of hardware, or in the form of hardware plus a software functional unit.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: a mobile storage device, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
Alternatively, the integrated unit of the present invention may be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. Based on such understanding, the technical solutions of the embodiments of the present invention may be essentially implemented or a part contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present invention. And the aforementioned storage medium includes: a removable storage device, a ROM, a RAM, a magnetic or optical disk, or various other media that can store program code.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present invention, and shall cover the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (5)
1. A multi-target intelligent sorting method for flexible parts is characterized by comprising the following steps:
acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame through a three-dimensional camera, inputting the RGB images into a trained example segmentation model, and outputting a mask image of a target nickel sheet after the RGB images pass through the example segmentation model;
after the output target mask image is aligned with the depth image, a target nickel sheet in the depth image is segmented, and target nickel sheet point cloud is generated according to the obtained target nickel sheet depth image through camera calibration parameters;
registering the target nickel piece point cloud and the model nickel piece point cloud to obtain the spatial position and attitude information of the target nickel piece in the material frame, and sending the spatial position and attitude information of the target nickel piece in the material frame to a controller to guide the manipulator to a specified position to absorb the nickel piece;
after nickel sheets are sucked, the manipulator moves to an industrial camera for deformation detection and secondary fine positioning, and sorting and assembling processes are completed according to the results of the deformation detection and the secondary fine positioning;
outputting a mask image of the target nickel sheet through an example segmentation model, specifically:
performing labeling definition on the multi-target nickel sheet part image according to a preset nickel sheet label definition strategy, wherein the label is divided into a front surface and a back surface;
in addition, dividing a positive sample and a negative sample according to the condition that the nickel sheet is shielded, generating a training data set according to tagged data after the nickel sheet label is determined, constructing an example segmentation model based on Mask R-CNN, and performing initialization training on the example segmentation model by using the training data set;
carrying out positive and negative recognition on nickel sheets in the input image through the trained example segmentation model, calculating confidence score for the nickel sheets meeting the label definition strategy, and carrying out mask output on the nickel sheet with the highest confidence score;
absorb behind the nickel sheet manipulator and remove to industry camera department and carry out deformation detection and secondary fine positioning, accomplish letter sorting and assembly process according to deformation detection and secondary fine positioning result, specifically do:
acquiring a target nickel sheet deformation detection image through an industrial two-dimensional camera, and performing image preprocessing on the target nickel sheet deformation detection image, wherein the preprocessing comprises image graying, threshold segmentation and region-of-interest interception;
determining nickel sheet deformation detection key characteristics by analyzing assembly requirements, wherein the nickel sheet deformation detection key characteristics comprise a reference positioning hole, a positioning edge and a nickel sheet outline circular arc;
edge detection is carried out through a Canny algorithm, a sub-pixel edge positioning technology is introduced, fine detection and fine positioning of the contour edge are achieved, then a least square method is used for fitting the characteristic contour edge, and size information among key characteristics of nickel sheet deformation is calculated;
judging whether the size information is within a preset size threshold range, if so, judging that the target nickel sheet does not meet the assembly requirement, and if not, assembling to a specified position according to the precisely positioned position information;
after the manipulator absorbs a target nickel sheet, if the nickel sheet is the reverse side, the nickel sheet needs to be moved to a reversing device for surface changing operation, and then the nickel sheet is moved to an industrial two-dimensional camera for deformation detection and fine positioning; and if the nickel sheet is the front surface, directly moving to a two-dimensional camera for deformation detection and fine positioning.
2. The multi-target intelligent sorting method for the flexible pieces according to claim 1, wherein pose information of target nickel pieces in a three-dimensional space is acquired, and the method specifically comprises the following steps:
acquiring a mask image of a target nickel sheet, aligning the mask image with the depth image, mapping pixel position information in the mask image to the depth image, segmenting pixel information of the target nickel sheet in the depth image, and generating a point cloud of the target nickel sheet from the segmented depth image according to calibration parameters of a three-dimensional camera;
generating point cloud by using a three-dimensional model of a nickel sheet, performing point cloud registration through point cloud processing and a registration algorithm, estimating the pose of a target nickel sheet, and acquiring a rigid transformation matrix between the target nickel sheet point cloud and the model nickel sheet point cloud;
and converting the rigid transformation matrix into actual control information according to the hand-eye calibration parameters, and controlling the manipulator to suck the nickel sheets to the corresponding position.
3. A multi-target intelligent sorting system for flexible parts is characterized by mainly comprising: the device comprises a memory, a processor, an example segmentation multi-target nickel piece part identification and positioning module, a nickel piece pose estimation module and a nickel piece deformation detection module, wherein a multi-target intelligent sorting method program of a flexible piece is stored and executed in the memory and the processor;
acquiring RGB images and depth images of a plurality of nickel sheets distributed in a material frame through a three-dimensional camera, inputting the RGB images into an example segmentation multi-target nickel sheet part identification positioning module to perform nickel sheet front and back identification positioning, and outputting mask images of the target nickel sheets;
generating a target nickel sheet point cloud through a nickel sheet pose estimation module, and registering the target nickel sheet point cloud and the model nickel sheet point cloud to acquire the spatial pose information of the target nickel sheet in the material frame;
deformation detection and secondary fine positioning are carried out through a nickel sheet deformation detection module, and sorting and assembling processes are completed according to deformation detection and secondary fine positioning results;
the example segmentation multi-target nickel piece part identification positioning module specifically comprises:
the example segmentation multi-target nickel piece part identification positioning module mainly comprises an example segmentation model part and a nickel piece label definition strategy part;
the example segmentation model part is constructed on the basis of Mask R-CNN, positive and negative recognition is carried out on nickel sheets in an input image, confidence coefficient scores are calculated for the nickel sheets meeting a label definition strategy, and the nickel sheets with the highest confidence coefficient scores are subjected to Mask output;
the nickel sheet label definition strategy part carries out labeling definition on the multi-target nickel sheet part images, and the labels are divided into a front surface and a back surface; dividing a positive sample and a negative sample according to the condition that the nickel sheet is shielded, generating a training data set according to tagging data after the nickel sheet tag is determined, and training an example segmentation model through the training data set;
the nickel sheet deformation detection module specifically comprises:
the nickel sheet deformation detection module consists of a nickel sheet deformation evaluation standard part, an image preprocessing part and a key characteristic extraction deformation detection part;
acquiring a target nickel sheet deformation detection image through an industrial two-dimensional camera, and performing image preprocessing on the target nickel sheet deformation detection image through an image preprocessing part, wherein the preprocessing comprises image graying, threshold segmentation and region-of-interest interception;
the nickel sheet deformation evaluation standard part determines key characteristics of nickel sheet deformation detection by analyzing assembly requirements, wherein the key characteristics of nickel sheet deformation detection comprise a reference positioning hole, a positioning edge and a nickel sheet outline circular arc;
and the key feature extraction deformation detection part performs edge detection through a Canny algorithm, introduces a sub-pixel edge positioning technology, realizes the precise detection and the precise positioning of the contour edge, fits the feature contour edge by using a least square method, and calculates the size information among the key features of nickel sheet deformation to detect whether the part is deformed.
4. The multi-target intelligent sorting system for the flexible pieces according to claim 3, wherein the nickel piece pose estimation module is specifically:
the nickel sheet pose estimation module comprises a target nickel sheet point cloud generation part, a model nickel sheet point cloud generation part and a point cloud registration pose estimation part;
the target nickel sheet point cloud generating part acquires a mask image of a target nickel sheet, aligns the mask image with the depth image, maps information in the mask image to the depth image, segments the target nickel sheet in the depth image, and generates a target nickel sheet point cloud from the segmented depth image according to calibration parameters of the three-dimensional camera;
the model nickel sheet point cloud generating part generates point cloud by using a three-dimensional model of a nickel sheet;
the point cloud registration pose estimation part carries out point cloud registration through point cloud processing and a registration algorithm, estimates the pose of the target nickel sheet and obtains a rigid transformation matrix between the target nickel sheet point cloud and the model nickel sheet point cloud.
5. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a program for a method for multi-target intelligent sorting of flexible items, and when the program for a method for multi-target intelligent sorting of flexible items is executed by a processor, the steps of a method for multi-target intelligent sorting of flexible items according to any one of claims 1 to 2 are implemented.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210851072.1A CN115082559B (en) | 2022-07-20 | 2022-07-20 | Multi-target intelligent sorting method and system for flexible parts and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210851072.1A CN115082559B (en) | 2022-07-20 | 2022-07-20 | Multi-target intelligent sorting method and system for flexible parts and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115082559A CN115082559A (en) | 2022-09-20 |
| CN115082559B true CN115082559B (en) | 2022-11-01 |
Family
ID=83260369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210851072.1A Active CN115082559B (en) | 2022-07-20 | 2022-07-20 | Multi-target intelligent sorting method and system for flexible parts and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115082559B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661104B (en) * | 2022-11-04 | 2023-08-11 | 广东杰成新能源材料科技有限公司 | Method, device, equipment and medium for evaluating overall integrity of power battery |
| CN117190866B (en) * | 2023-11-08 | 2024-01-26 | 广东工业大学 | Polarity discrimination detection method, device and equipment for multiple stacked electronic components |
| CN117686939A (en) * | 2024-02-02 | 2024-03-12 | 宁德时代新能源科技股份有限公司 | Battery nickel piece welding quality detection system and battery nickel piece welding quality detection method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113450408A (en) * | 2021-06-23 | 2021-09-28 | 中国人民解放军63653部队 | Irregular object pose estimation method and device based on depth camera |
| US11200429B1 (en) * | 2018-12-28 | 2021-12-14 | Zoox, Inc. | Tracking objects using sensor data segmentations and/or representations |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103500290B (en) * | 2013-10-21 | 2016-08-24 | 广东工业大学 | A kind of flexible material machining deformation influence factor extracting method |
| CN110580725A (en) * | 2019-09-12 | 2019-12-17 | 浙江大学滨海产业技术研究院 | Box sorting method and system based on RGB-D camera |
| CN110910451B (en) * | 2019-10-23 | 2023-06-02 | 同济大学 | Object pose estimation method and system based on deformation convolution network |
| CN112270249B (en) * | 2020-10-26 | 2024-01-23 | 湖南大学 | Target pose estimation method integrating RGB-D visual characteristics |
| CN112837371A (en) * | 2021-02-26 | 2021-05-25 | 梅卡曼德（北京）机器人科技有限公司 | Object grabbing method and device based on 3D matching and computing equipment |
| CN113034593B (en) * | 2021-03-09 | 2023-12-12 | 深圳市广宁股份有限公司 | 6D pose labeling method, system and storage medium |
| CN113112504B (en) * | 2021-04-08 | 2023-11-03 | 浙江大学 | Plant point cloud data segmentation method and system |
-
2022
- 2022-07-20 CN CN202210851072.1A patent/CN115082559B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11200429B1 (en) * | 2018-12-28 | 2021-12-14 | Zoox, Inc. | Tracking objects using sensor data segmentations and/or representations |
| CN113450408A (en) * | 2021-06-23 | 2021-09-28 | 中国人民解放军63653部队 | Irregular object pose estimation method and device based on depth camera |
Non-Patent Citations (1)
| Title |
|---|
| "智能视觉数控切割系统";刘夏丽等;《自动化与信息工程》;20200824;第41卷(第4期);第30-33，54页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115082559A (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115082559B (en) | Multi-target intelligent sorting method and system for flexible parts and storage medium | |
| CN111652085B (en) | Object identification method based on combination of 2D and 3D features | |
| US9327406B1 (en) | Object segmentation based on detected object-specific visual cues | |
| CN111062915A (en) | Real-time steel pipe defect detection method based on improved YOLOv3 model | |
| CN111461107A (en) | Material handling method, apparatus and system for identifying regions of interest | |
| CN107704867A (en) | Based on the image characteristic point error hiding elimination method for weighing the factor in a kind of vision positioning | |
| CN112734844B (en) | Monocular 6D pose estimation method based on octahedron | |
| CN114972489A (en) | Tray identification method and device, storage medium and intelligent forklift equipment | |
| Holz et al. | Fast edge-based detection and localization of transport boxes and pallets in rgb-d images for mobile robot bin picking | |
| CN114863250A (en) | Container lockhole identification and positioning method, system and storage medium | |
| CN113947576A (en) | Container positioning method and device, container access equipment and storage medium | |
| CN113724322A (en) | Cargo pallet positioning method and system for unmanned forklift | |
| CN111563871B (en) | Image processing method, device and equipment, visual guide unstacking method and system | |
| Cociaş et al. | Multiple-superquadrics based object surface estimation for grasping in service robotics | |
| CN114897974B (en) | Target object space positioning method, system, storage medium and computer equipment | |
| CN116309882A (en) | Tray detection and positioning method and system for unmanned forklift application | |
| CN114800533B (en) | Sorting control method and system for industrial robot | |
| Nakano | Stereo vision based single-shot 6d object pose estimation for bin-picking by a robot manipulator | |
| CN116188540A (en) | Target identification and pose estimation method based on point cloud information | |
| CN115272465A (en) | Object positioning method, device, autonomous mobile device and storage medium | |
| CN115100416A (en) | Irregular steel plate pose identification method and related equipment | |
| CN115049842A (en) | Aircraft skin image damage detection and 2D-3D positioning method | |
| CN114332073A (en) | Target workpiece detection method and device, intelligent sorting system and storage medium | |
| CN113822869A (en) | Transparent soft packaging bag static detection method and system based on machine vision | |
| CN110738268A (en) | intelligent stereoscopic warehouse goods automatic identification method based on SIFT and DDIS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |