CN114905515A - Robot control method and system based on flexible perception neural network - Google Patents
Robot control method and system based on flexible perception neural network Download PDFInfo
- Publication number
- CN114905515A CN114905515A CN202210588502.5A CN202210588502A CN114905515A CN 114905515 A CN114905515 A CN 114905515A CN 202210588502 A CN202210588502 A CN 202210588502A CN 114905515 A CN114905515 A CN 114905515A
- Authority
- CN
- China
- Prior art keywords
- workpiece
- polishing
- information
- processed
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
- B25J11/005—Manipulators for mechanical processing tasks
- B25J11/0065—Polishing or grinding
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/1633—Programme controls characterised by the control loop compliant, force, torque control, e.g. combined with position control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention discloses a robot control method and system based on a flexible perception neural network, relating to the technical field of artificial intelligence, wherein the method comprises the following steps: acquiring image information of a workpiece to be processed through the visual information acquisition module; inputting the target polishing standard information and the image information of the workpiece to be processed into an applicable workpiece polishing analysis model to obtain a polishing scheme of the workpiece to be processed; the flexible polishing robot is controlled by the motion trail control module to polish the workpiece according to the polishing scheme of the workpiece to be processed; acquiring action force compensation information according to the workpiece polishing force information; and correcting the polishing scheme of the workpiece to be processed by analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network. Reach through carrying out real-time interaction with flexible polishing robot, realize polishing the power compensation to the robot of arbitrary gesture, ensure the work piece dynamics of polishing, improve the work piece precision of polishing, and then guarantee the technological effect of polishing work piece quality.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a robot control method and system based on a flexible perception neural network.
Background
Workpiece polishing is one of surface modification technologies, and a processing method for changing the physical properties of the surface of a material through friction is widely applied to the fields of machine manufacturing industry, processing industry, die industry, wood industry, leather industry and the like. The grinding aims to obtain the specific surface roughness of a workpiece, remove burrs on the surface of the workpiece of a product, make the workpiece smooth and easy to continue processing or reach a finished product.
However, the polishing control precision of the polishing robot in the prior art is not high in the operation process, so that the quality of the polished workpiece does not reach the standard.
Disclosure of Invention
The application provides a robot control method and system based on a flexible perception neural network, and solves the technical problems that polishing control precision of a polishing robot in the operation process is not high, and the quality of a polished workpiece is not up to standard in the prior art, achieves real-time interaction with the flexible polishing robot through machine vision acquisition information and polishing force acquisition information, achieves polishing force compensation of the robot in any posture, ensures workpiece polishing force, improves workpiece polishing precision, and further ensures the technical effect of workpiece polishing quality.
In view of the above problems, the present invention provides a robot control method and system based on a flexible perception neural network.
In a first aspect, the present application provides a robot control method based on a flexible perception neural network, the method including: acquiring images of a workpiece to be processed through a visual information acquisition module to obtain image information of the workpiece to be processed; uploading the image information of the workpiece to be processed to a polishing robot control system for interactive analysis based on a data interaction module; acquiring target polishing standard information, inputting the target polishing standard information and the to-be-processed workpiece image information into an applicable workpiece polishing analysis model by the polishing robot control system, and acquiring a model output result, wherein the model output result comprises a to-be-processed workpiece polishing scheme; sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module, and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through a motion trail control module; acquiring workpiece polishing force information in a workpiece polishing process through a force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to obtain action force compensation information; and capturing the gesture according to the visual information acquisition module to obtain polishing pose spatial information, and analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network to correct the polishing track of the polishing scheme of the workpiece to be processed.
In another aspect, the present application further provides a robot control system based on a flexible perception neural network, the system including: the robot forming module is used for the flexible polishing robot and comprises a visual information acquisition module, a force axis sensor module, a data interaction module and a motion trail control module; the visual information acquisition module is used for acquiring images of the workpiece to be processed through the visual information acquisition module to obtain image information of the workpiece to be processed; the data interaction module is used for uploading the image information of the workpiece to be processed to a polishing robot control system for interactive analysis based on the data interaction module; the polishing robot control system inputs the target polishing standard information and the image information of the workpiece to be processed into an applicable workpiece polishing analysis model to obtain a model output result, wherein the model output result comprises a polishing scheme of the workpiece to be processed; the motion trail control module is used for sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through the motion trail control module; the force axis sensor module is used for acquiring workpiece polishing force information in a workpiece polishing process through the force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to acquire action force compensation information; and the polishing track correction module is used for capturing postures according to the visual information acquisition module to obtain polishing pose spatial information, and performing polishing track correction on the polishing scheme of the workpiece to be processed by analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network.
One or more technical solutions provided in the present application have at least the following technical effects or advantages:
the image acquisition of the workpiece to be processed is carried out through the visual information acquisition module, the image information of the workpiece to be processed is uploaded to the polishing robot control system for interactive analysis based on the data interaction module, the target polishing standard information and the image information of the workpiece to be processed are input into an applicable workpiece polishing analysis model, and a model output result, namely a polishing scheme of the workpiece to be processed, is obtained; the flexible grinding robot is controlled by a motion trail control module to grind workpieces according to the to-be-processed workpiece grinding scheme, workpiece grinding force information in the workpiece grinding process is acquired by a force axis sensor module, force compensation analysis is carried out to acquire action force compensation information, meanwhile gesture capture is carried out by a visual information acquisition module to acquire grinding pose spatial information, and the action force compensation information and the grinding pose spatial information are analyzed by a flexible sensing network to carry out grinding trail correction on the to-be-processed workpiece grinding scheme. And then reach and carry out real-time interaction through machine vision collection information and dynamics of polishing collection information and flexible polishing robot, realize carrying out the power compensation of polishing to the robot of arbitrary gesture, ensure the work piece dynamics of polishing, improve the work piece precision of polishing, and then guarantee the technological effect of the work piece quality of polishing.
Drawings
Fig. 1 is a schematic flow chart of a robot control method based on a flexible perception neural network according to the present application;
FIG. 2 is a schematic flow chart of a robot control method based on a flexible perception neural network according to the present application for obtaining image information of a standard workpiece to be processed;
FIG. 3 is a schematic flow chart of a polishing scheme of a workpiece to be processed output in the robot control method based on the flexible perception neural network according to the present application;
FIG. 4 is a schematic structural diagram of a robot control system based on a flexible perception neural network according to the present application;
description of reference numerals: the robot comprises a robot composition module 11, a visual information acquisition module 12, a data interaction module 13, a model output module 14, a motion track control module 15, a force axis sensor module 16 and a polishing track correction module 17.
The present application is described below with reference to the drawings attached hereto.
Example one
As shown in fig. 1, the present application provides a robot control method based on flexible perception neural network, which is applied to a grinding robot control system, the system is communicatively connected with a flexible grinding robot, the method includes:
step S100: the flexible polishing robot comprises a visual information acquisition module, a force axis sensor module, a data interaction module and a motion trail control module;
specifically, workpiece polishing is one of surface modification techniques, and a processing method for changing the physical properties of the surface of a material through friction is widely applied in the fields of machinery manufacturing industry, processing industry, die industry, wood industry, leather industry and the like. The grinding aims to obtain the specific surface roughness of a workpiece, remove burrs on the surface of the workpiece of a product, make the workpiece smooth and easy to continue processing or reach a finished product.
The workpiece is polished through the flexible polishing robot, a tail end shaft of the flexible polishing robot body is fused with the flexible sensing neural network, a wrist part has a unique shrinkage design, the rigidity is improved, the shaking in the dynamic polishing process is effectively avoided, the polishing robot has the polishing advantage, and the polishing robot comprises a visual information acquisition module, a force axis sensor module, a data interaction module and a motion track control module, and the workpieces are finely polished through the mutual matching control among the modules.
Step S200: acquiring images of a workpiece to be processed through the visual information acquisition module to obtain image information of the workpiece to be processed;
specifically, the visual information acquisition module is used for acquiring images of the workpiece to be processed, and the visual information acquisition module, namely the optical imaging module, can adopt a camera or a camera to identify and acquire images of the workpiece to be processed, and comprises information such as the size of the workpiece, the structure of the workpiece, the color of the workpiece, the burr of the workpiece and the like.
Step S300: uploading the image information of the workpiece to be processed to the polishing robot control system for interactive analysis based on the data interaction module;
specifically, based on the data interaction module will pending work piece image information upload to polishing robot control system carries out the interactive analysis, the data interaction module is used for flexible polishing robot and polishing robot control system to carry out the data transmission interaction, polishing robot control system is used for carrying out the scheme analysis of polishing and formulating based on work piece visual information.
Step S400: acquiring target polishing standard information, inputting the target polishing standard information and the to-be-processed workpiece image information into an applicable workpiece polishing analysis model by the polishing robot control system, and acquiring a model output result, wherein the model output result comprises a to-be-processed workpiece polishing scheme;
as shown in fig. 2, further, before inputting the target polishing standard information and the workpiece image information to be processed into the applicable workpiece polishing analysis model, step S400 of the present application further includes:
step S410: denoising and filtering the image information of the workpiece to be processed based on an image filtering algorithm to obtain the image information of the workpiece to be processed;
step S420: carrying out gray level equalization processing on the image information of the de-noised workpiece to be processed to obtain the image information of the workpiece to be processed with the mean value;
step S430: and performing image enhancement on the image information of the workpiece to be processed with the mean value based on the generated countermeasure network, and then performing image grid segmentation on the enhanced image to obtain the image information of the standard workpiece to be processed.
Specifically, target polishing standard information of the workpiece to be processed is acquired, wherein the target polishing standard information is the requirements of surface smoothness and surface curvature for polishing the workpiece. And the grinding robot control system inputs the target grinding standard information and the to-be-processed workpiece image information into an applicable workpiece grinding analysis model, and the applicable workpiece grinding analysis model is used for formulating a workpiece grinding scheme. Before the target polishing standard information and the to-be-processed workpiece image information are input into an applicable workpiece polishing analysis model, denoising and filtering are carried out on the to-be-processed workpiece image information based on an image filtering algorithm, namely, the noise of a target image is suppressed under the condition that image detail characteristics are kept as much as possible, the operation is indispensable in image preprocessing, and the effectiveness and the reliability of subsequent image processing and analysis are directly influenced by the quality of the processing effect. The commonly used image filtering algorithm comprises nonlinear filtering, median filtering, wavelet filtering, bilateral filtering and the like, and the image information of the workpiece to be processed after denoising is obtained.
And then, carrying out gray level equalization processing on the image information of the to-be-processed de-noised workpiece, namely carrying out image gray level histogram equalization, which is a simple and effective image enhancement technology, changing the gray level of each pixel in the image by changing the histogram of the image, and mainly used for enhancing the contrast of the image with a small dynamic range and obtaining the equalized image information of the to-be-processed workpiece. And performing image enhancement on the image information of the mean workpiece to be processed based on the generated countermeasure network, namely enhancing useful information in the image, improving the visual effect of the image, and purposefully emphasizing the overall or local characteristics of the image for the application occasion of the given image.
And then, carrying out image grid segmentation on the enhanced image, wherein the finer the grid segmentation is, the more detailed the image detail characteristics are, the better the effect is, and obtaining the image information of the standard workpiece to be processed after preprocessing. And inputting standard workpiece image information to be processed and the target polishing standard information into an applicable workpiece polishing analysis model to obtain a model output result, wherein the model output result comprises a workpiece polishing scheme to be processed, and the workpiece polishing scheme to be processed comprises information such as a workpiece polishing motion track, a polishing speed, a polishing force and the like. The image application effect is improved by denoising, balancing and image enhancement preprocessing on the workpiece collected image, so that the image is standardized, and the accuracy of the polishing scheme of the workpiece to be processed is improved.
Step S500: sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module, and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through the motion trail control module;
step S600: acquiring workpiece polishing force information in a workpiece polishing process through the force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to obtain action force compensation information;
specifically, the polishing scheme of the workpiece to be processed is sent to the flexible polishing robot through the data interaction module, and the flexible polishing robot is controlled by the motion trail control module to polish the workpiece according to the polishing scheme of the workpiece to be processed. In the polishing process, polishing force information of the workpiece in each force axis direction in the workpiece polishing process is obtained through the force axis sensor module. Due to the self weight of the robot and the posture influence in the polishing process, the polishing force is possibly insufficient, and force compensation analysis is carried out according to the workpiece polishing force information to obtain the action force compensation information needing to be subjected to operation force compensation.
Step S700: and capturing the gesture according to the visual information acquisition module to obtain polishing pose spatial information, and analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network to correct the polishing track of the polishing scheme of the workpiece to be processed.
Specifically, attitude capture is performed according to the visual information acquisition module, and grinding pose spatial information, namely current position and attitude information of the flexible grinding robot, is obtained. And analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network, wherein the flexible sensing network is interacted with a tail end shaft of the flexible robot. The polishing track of the polishing scheme of the workpiece to be processed is corrected through an analysis result, the force feedback response speed is improved, the polishing force compensation of the robot in any posture is realized, the polishing force of the workpiece is ensured, the polishing precision of the workpiece is improved, and the quality of the polished workpiece is further ensured.
As shown in fig. 3, further, step S430 of the present application further includes:
step S431: classifying the basic information of the workpiece to be processed through a workpiece polishing feature decision tree to obtain classification feature information of the workpiece to be polished;
step S432: calibrating the workpiece to be processed according to the classification characteristic information of the polished workpiece to determine a calibration parameter of the polished workpiece;
step S433: calling the applicable workpiece polishing analysis model from a workpiece polishing analysis model library based on the polishing workpiece calibration parameters;
step S434: and inputting the target polishing standard information and the standard workpiece image information to be processed into the applicable workpiece polishing analysis model, and outputting the workpiece polishing scheme to be processed.
Specifically, the basic information of the workpiece to be processed is classified through a grinding workpiece feature decision tree, the grinding workpiece feature decision tree is constructed through workpiece feature information to obtain grinding workpiece classification feature information, and the grinding workpiece classification feature information is a workpiece feature classification result and comprises a workpiece size specification, a workpiece type, a workpiece material and the like. And workpieces with different characteristic categories and different grinding scheme analysis modes are also different, so that grinding workpiece calibration parameters are determined according to the grinding workpiece classification characteristic information, and the grinding workpiece calibration parameters are used for dividing model selection during grinding analysis of the workpieces.
And calling the applicable workpiece grinding analysis model from a workpiece grinding analysis model library based on the grinding workpiece calibration parameters, wherein the applicable workpiece grinding analysis model is a grinding scheme analysis model applicable to the workpiece. And inputting the target polishing standard information and the standard workpiece image information to be processed into the applicable workpiece polishing analysis model, and outputting the workpiece polishing scheme to be processed. Through carrying out classification and identification on the grinding workpieces, the grinding analysis model suitable for the workpieces is called in an individualized mode, and therefore accuracy and reasonability of model output results are improved.
Further, step S434 of the present application further includes:
step S4341: the applicable workpiece polishing analysis model comprises an input layer, a convolution image network layer, a characteristic analysis logic layer and an output layer;
step S4342: inputting the target polishing standard information and the standard workpiece image information to be processed into the convolution image network layer as input layers to obtain the workpiece characteristic information to be processed;
step S4343: inputting the target polishing standard information and the workpiece feature information to be processed into the feature analysis logic layer to obtain the workpiece polishing scheme to be processed;
step S4344: and outputting the polishing scheme of the workpiece to be processed as an output result through the output layer.
Specifically, the applicable workpiece polishing analysis model comprises an input layer, a convolutional image network layer, a feature analysis logic layer and an output layer, and the target polishing standard information and the standard workpiece image information to be processed are used as the input layer and input into the convolutional image network layer. The convolution image network layer is used for extracting the features of the workpiece image and extracting the feature information of the workpiece to be processed, and the feature information of the workpiece to be processed comprises the flatness feature, the curvature feature, the hole defect feature and the like of the workpiece.
And inputting the target polishing standard information and the characteristic information of the workpiece to be processed into the characteristic analysis logic layer, wherein the characteristic analysis logic layer is used for performing matching analysis on the target polishing standard information and the characteristic information of the workpiece to be processed to obtain the polishing scheme of the workpiece to be processed output by the logic layer. And outputting the polishing scheme of the workpiece to be processed as an output result through the output layer, analyzing and formulating the polishing scheme of the workpiece to be processed by constructing a multi-level applicable workpiece polishing analysis model, and improving scheme formulation accuracy and formulation efficiency.
Further, in the step S4342, obtaining feature information of the workpiece to be processed further includes:
step S43421: obtaining a preset convolution characteristic set according to the target polishing standard information, wherein the preset convolution characteristic set comprises a workpiece flatness characteristic, a curvature characteristic and a hole defect characteristic;
step S43422: inputting the standard workpiece image information to be processed into the convolution image network layer as input information for feature extraction;
step S43423: and obtaining output information of the network layer of the convolution image, wherein the output information comprises the workpiece feature information to be processed which accords with the preset convolution feature set.
Specifically, a preset convolution characteristic set is obtained according to the target polishing standard information, and the preset convolution characteristic set is a preset characteristic standard meeting workpiece polishing requirements and comprises a workpiece flatness characteristic, a curvature characteristic and a hole defect characteristic. And taking the required workpiece flatness characteristic, curvature characteristic and hole defect characteristic as a preset convolution characteristic set, which is equivalent to extracting an image characteristic value. The standard workpiece image information to be processed is input into the convolutional image network layer as input information for feature extraction, the convolutional image network layer is a deep feedforward neural network with the characteristics of local connection, weight sharing and the like, and remarkable effects are achieved in the field of image and video analysis, such as various visual tasks of image classification, target detection, image segmentation and the like.
The convolutional image network layer includes two parts, convolution and neural network, where convolution is the feature extractor and neural network can be regarded as classifier. And obtaining output information of the network layer of the convolution image, wherein the output information comprises the workpiece feature information to be processed which accords with the preset convolution feature set. The technical effects that the standard product production image information is subjected to matching analysis in a convolution extraction mode, the workpiece feature matching result to be processed is more accurate, and the accuracy of the polishing scheme of the workpiece to be processed is improved are achieved.
Further, the method further comprises the following steps:
step S810: verifying the analysis effect of the applicable workpiece polishing analysis model to obtain the model analysis accuracy;
step S820: if the model analysis accuracy does not reach the preset analysis accuracy, obtaining a model analysis deviation degree based on the difference value between the model analysis accuracy and the preset analysis accuracy;
step S830: and iteratively updating the applicable workpiece polishing analysis model based on the PSO algorithm and the model analysis deviation degree to obtain the applicable workpiece polishing optimization analysis model.
Specifically, the analysis effect of the applicable workpiece grinding analysis model is verified, namely, the model grinding analysis accuracy is verified to obtain the model analysis accuracy, and the model analysis accuracy indicates the model analysis accuracy. If the model analysis accuracy does not reach the preset analysis accuracy, namely the training output accuracy of the workpiece grinding analysis model does not reach the standard, the difference value between the model analysis accuracy and the preset analysis accuracy is used as the model analysis deviation degree, namely the model analysis accuracy needing to be optimized, and the larger the deviation degree is, the lower the formulation accuracy of the workpiece grinding analysis scheme is.
Because the fitting degree of the applicable workpiece grinding analysis model is low, the model cannot adapt to the analysis of the current workpiece grinding scheme, and the applicable workpiece grinding analysis model is updated iteratively based on the PSO algorithm and the model analysis deviation degree. The PSO algorithm, namely the particle swarm optimization algorithm, is a random optimization algorithm based on a population, can simulate and continuously iterate until a balanced or optimal state is finally reached, the balanced or optimal state is stored, and a workpiece polishing optimization analysis model updated and optimized by the PSO algorithm is obtained. The model is optimized through the PSO algorithm, so that the output deviation degree of the model is reduced, the accuracy and the efficiency of the output result of the model are improved, and the accuracy of workpiece polishing analysis is further improved.
Further, the step S830 of obtaining an analysis model for optimizing polishing of a suitable workpiece further includes:
step S831: constructing a particle optimization space according to the model training parameters of the applicable workpiece polishing analysis model;
step S832: initializing the particle optimization space to obtain particle swarm constraint parameters, and iteratively calculating a particle swarm fitness function according to the model analysis deviation degree and the particle swarm constraint parameters;
step S833: when a preset termination condition is reached, obtaining a first output result of the particle swarm fitness function, wherein the first output result comprises optimal result particles;
step S834: and mapping the optimal result particles to the applicable workpiece polishing analysis model for optimization training to obtain the applicable workpiece polishing optimization analysis model.
Specifically, the model training parameters suitable for the workpiece grinding analysis model are model parameter training dimensions including information such as model training weights, hidden layer numbers and hidden layer node numbers, the particle optimization space is a virtual space for optimizing the workpiece grinding optimization analysis model, and is a multi-dimensional virtual space, and the spatial dimensions are the same as the model training parameter dimensions of the workpiece grinding optimization analysis model. Initializing the particle optimization space, wherein the particle swarm constraint parameter is a model analysis accuracy range, iteratively calculating a particle swarm fitness function according to the model analysis deviation degree and the particle swarm constraint parameter, further updating the positions and the speeds of particles in the particle swarm, inputting all the particles into the model for training, evaluating the quality of the particles by calculating the fitness function of the particle swarm, and adjusting the positions and the speeds of all the particles by the fitness function.
And when a preset termination condition is reached, obtaining a first output result of the particle swarm fitness function, wherein the first output result comprises optimal result particles. Further, the PSO algorithm is stopped to include two possibilities, one is that the particles obtain a balanced or optimal state, the other is that the operation limit is exceeded, the condition of exceeding the operation limit is not specifically analyzed, and the optimal result particles are the optimal state of the particles; and mapping the optimal result particles to the applicable workpiece grinding analysis model for optimization training. The output accuracy of the suitable workpiece polishing analysis model after optimized training is improved, the suitable workpiece polishing analysis model is optimized and trained through a particle swarm optimization algorithm, the output deviation degree of the model is reduced, the accuracy and the efficiency of the output result of the model are improved, and the accuracy of workpiece polishing analysis is further improved.
In summary, the robot control method and system based on the flexible perception neural network provided by the present application have the following technical effects:
the image acquisition of the workpiece to be processed is carried out through the visual information acquisition module, the image information of the workpiece to be processed is uploaded to the polishing robot control system for interactive analysis based on the data interaction module, the target polishing standard information and the image information of the workpiece to be processed are input into an applicable workpiece polishing analysis model, and a model output result, namely a polishing scheme of the workpiece to be processed, is obtained; the flexible grinding robot is controlled by a motion trail control module to grind workpieces according to the to-be-processed workpiece grinding scheme, workpiece grinding force information in the workpiece grinding process is acquired by a force axis sensor module, force compensation analysis is carried out to acquire action force compensation information, meanwhile gesture capture is carried out by a visual information acquisition module to acquire grinding pose spatial information, and the action force compensation information and the grinding pose spatial information are analyzed by a flexible sensing network to carry out grinding trail correction on the to-be-processed workpiece grinding scheme. And then reach and carry out real-time interaction through machine vision collecting information and polishing dynamics collecting information and flexible polishing robot, realize carrying out the power compensation of polishing to the robot of arbitrary gesture, ensure the work piece dynamics of polishing, improve the work piece precision of polishing, and then guarantee the technological effect of the work piece quality of polishing.
Example two
Based on the same inventive concept as the robot control method based on the flexible perception neural network in the foregoing embodiment, the present invention further provides a robot control system based on the flexible perception neural network, as shown in fig. 4, the system includes:
the robot forming module 11 is used for the flexible polishing robot and comprises a visual information acquisition module, a force axis sensor module, a data interaction module and a motion trail control module;
the visual information acquisition module 12 is used for acquiring images of the workpiece to be processed through the visual information acquisition module to obtain image information of the workpiece to be processed;
the data interaction module 13 is used for uploading the image information of the workpiece to be processed to a polishing robot control system for interactive analysis based on the data interaction module;
the model output module 14 is configured to obtain target polishing standard information, and the polishing robot control system inputs the target polishing standard information and the to-be-processed workpiece image information into an applicable workpiece polishing analysis model to obtain a model output result, where the model output result includes a to-be-processed workpiece polishing scheme;
the motion trail control module 15 is used for sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module, and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through the motion trail control module;
the force axis sensor module 16 is used for acquiring workpiece polishing force information in a workpiece polishing process through the force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to acquire action force compensation information;
and the polishing track correction module 17 is configured to capture a posture according to the visual information acquisition module, obtain polishing pose spatial information, and perform polishing track correction on the workpiece polishing scheme to be processed by analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensor network.
Further, the model output module further includes:
the de-noising and filtering unit is used for de-noising and filtering the image information of the workpiece to be processed based on an image filtering algorithm to obtain the image information of the de-noised workpiece to be processed;
the equalization processing unit is used for carrying out gray level equalization processing on the image information of the de-noised workpiece to be processed to obtain the image information of the workpiece to be processed with the mean value;
and the image enhancement unit is used for carrying out image enhancement on the image information of the workpiece to be processed with the mean value based on the generated countermeasure network, and then carrying out image grid segmentation on the enhanced image to obtain the image information of the standard workpiece to be processed.
Further, the system further comprises:
the characteristic classification unit is used for classifying the basic information of the workpiece to be processed through a grinding workpiece characteristic decision tree to obtain grinding workpiece classification characteristic information;
the parameter calibration unit is used for calibrating the workpiece to be processed according to the classification characteristic information of the polished workpiece to determine the calibration parameters of the polished workpiece;
the model calling unit is used for calling the applicable workpiece polishing analysis model from a workpiece polishing analysis model library based on the polishing workpiece calibration parameters;
and the model output unit is used for inputting the target polishing standard information and the standard workpiece image information to be processed into the applicable workpiece polishing analysis model and outputting the workpiece polishing scheme to be processed.
Further, the model output unit further includes:
the model forming unit is used for enabling the applicable workpiece polishing analysis model to comprise an input layer, a convolution image network layer, a characteristic analysis logic layer and an output layer;
the convolution network layer unit is used for inputting the target polishing standard information and the standard workpiece image information to be processed into the convolution image network layer as input layers to obtain the characteristic information of the workpiece to be processed;
the characteristic analysis logic layer unit is used for inputting the target polishing standard information and the characteristic information of the workpiece to be processed into the characteristic analysis logic layer to obtain the polishing scheme of the workpiece to be processed;
and the output layer unit is used for outputting the polishing scheme of the workpiece to be processed as an output result through the output layer.
Further, the convolutional network layer unit further includes:
the convolution characteristic determining unit is used for obtaining a preset convolution characteristic set according to the target polishing standard information, wherein the preset convolution characteristic set comprises a workpiece flatness characteristic, a curvature characteristic and a hole defect characteristic;
the convolution characteristic extraction unit is used for inputting the standard workpiece image information to be processed into the convolution image network layer as input information for characteristic extraction;
and the convolution network output unit is used for obtaining the output information of the convolution image network layer, and the output information comprises the workpiece feature information to be processed which accords with the preset convolution feature set.
Further, the system further comprises:
the model verification unit is used for verifying the analysis effect of the applicable workpiece polishing analysis model to obtain the model analysis accuracy;
the model deviation degree analysis unit is used for obtaining a model analysis deviation degree based on the difference value between the model analysis accuracy degree and a preset analysis accuracy degree if the model analysis accuracy degree does not reach the preset analysis accuracy degree;
and the model updating unit is used for iteratively updating the applicable workpiece polishing analysis model based on the PSO algorithm and the model analysis deviation degree to obtain the applicable workpiece polishing optimization analysis model.
Further, the model updating unit further includes:
the optimization space construction unit is used for constructing a particle optimization space according to the model training parameters of the applicable workpiece polishing analysis model;
a fitness function calculating unit, configured to initialize the particle optimization space, obtain particle swarm constraint parameters, and iteratively calculate a particle swarm fitness function according to the model analysis deviation degree and the particle swarm constraint parameters;
the fitness function output unit is used for obtaining a first output result of the particle swarm fitness function when a preset termination condition is reached, and the first output result comprises optimal result particles;
and the model optimization unit is used for mapping the optimal result particles to the applicable workpiece polishing analysis model for optimization training to obtain the applicable workpiece polishing optimization analysis model.
The application provides a robot control method based on a flexible perception neural network, which comprises the following steps: acquiring an image of a workpiece to be processed through a visual information acquisition module to obtain image information of the workpiece to be processed; uploading the image information of the workpiece to be processed to a polishing robot control system for interactive analysis based on a data interaction module; acquiring target polishing standard information, inputting the target polishing standard information and the to-be-processed workpiece image information into an applicable workpiece polishing analysis model by the polishing robot control system, and acquiring a model output result, wherein the model output result comprises a to-be-processed workpiece polishing scheme; sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module, and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through a motion trail control module; acquiring workpiece polishing force information in a workpiece polishing process through a force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to obtain action force compensation information; and capturing the gesture according to the visual information acquisition module to obtain polishing pose spatial information, and analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network to correct the polishing track of the polishing scheme of the workpiece to be processed. The technical problem of current polishing robot polishing control accuracy in the operation process not high, lead to polishing work piece quality not up to standard is solved. Reach and carry out real-time interaction through machine vision collection information and dynamics of polishing collection information and flexible polishing robot, realize polishing the power compensation to the robot of arbitrary gesture, ensure the work piece dynamics of polishing, improve the work piece precision of polishing, and then guarantee the technological effect of polishing work piece quality.
The specification and drawings are merely illustrative of the present application, and it is intended that the present invention cover modifications and variations of this invention provided they come within the scope of the invention and their equivalents.
Claims (8)
1. A robot control method based on a flexible perception neural network is applied to a grinding robot control system which is in communication connection with a flexible grinding robot, and the method comprises the following steps:
the flexible polishing robot comprises a visual information acquisition module, a force axis sensor module, a data interaction module and a motion trail control module;
acquiring images of a workpiece to be processed through the visual information acquisition module to obtain image information of the workpiece to be processed;
uploading the image information of the workpiece to be processed to the polishing robot control system for interactive analysis based on the data interaction module;
acquiring target polishing standard information, inputting the target polishing standard information and the to-be-processed workpiece image information into an applicable workpiece polishing analysis model by the polishing robot control system, and acquiring a model output result, wherein the model output result comprises a to-be-processed workpiece polishing scheme;
sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module, and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through the motion trail control module;
acquiring workpiece polishing force information in a workpiece polishing process through the force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to obtain action force compensation information;
and capturing the gesture according to the visual information acquisition module to obtain polishing pose spatial information, and analyzing the motion force compensation information and the polishing pose spatial information through a flexible sensing network to correct the polishing track of the polishing scheme of the workpiece to be processed.
2. The method of claim 1, wherein prior to inputting said target sanding criteria information and said workpiece image to be processed information into an applicable workpiece sanding analysis model, comprising:
denoising and filtering the image information of the workpiece to be processed based on an image filtering algorithm to obtain the image information of the workpiece to be processed;
carrying out gray level equalization processing on the image information of the de-noised workpiece to be processed to obtain the image information of the workpiece to be processed with the mean value;
and performing image enhancement on the image information of the workpiece to be processed with the mean value based on the generated countermeasure network, and then performing image grid segmentation on the enhanced image to obtain the image information of the standard workpiece to be processed.
3. The method of claim 2, wherein the method comprises:
classifying the basic information of the workpiece to be processed through a workpiece polishing feature decision tree to obtain classification feature information of the workpiece to be polished;
calibrating the workpiece to be processed according to the classification characteristic information of the polished workpiece to determine the calibration parameters of the polished workpiece;
calling the applicable workpiece polishing analysis model from a workpiece polishing analysis model library based on the polishing workpiece calibration parameters;
and inputting the target polishing standard information and the standard workpiece image information to be processed into the applicable workpiece polishing analysis model, and outputting the workpiece polishing scheme to be processed.
4. The method of claim 3, wherein the method comprises:
the applicable workpiece polishing analysis model comprises an input layer, a convolution image network layer, a characteristic analysis logic layer and an output layer;
inputting the target polishing standard information and the standard workpiece image information to be processed into the convolution image network layer as input layers to obtain the workpiece characteristic information to be processed;
inputting the target polishing standard information and the workpiece feature information to be processed into the feature analysis logic layer to obtain the workpiece polishing scheme to be processed;
and outputting the polishing scheme of the workpiece to be processed as an output result through the output layer.
5. The method of claim 4, wherein obtaining workpiece feature information to be processed comprises:
obtaining a preset convolution characteristic set according to the target polishing standard information, wherein the preset convolution characteristic set comprises a workpiece flatness characteristic, a curvature characteristic and a hole defect characteristic;
inputting the standard workpiece image information to be processed into the convolution image network layer as input information for feature extraction;
and obtaining output information of the network layer of the convolution image, wherein the output information comprises the workpiece feature information to be processed which accords with the preset convolution feature set.
6. The method of claim 1, wherein the method comprises:
verifying the analysis effect of the applicable workpiece polishing analysis model to obtain the model analysis accuracy;
if the model analysis accuracy does not reach the preset analysis accuracy, obtaining a model analysis deviation degree based on the difference value between the model analysis accuracy and the preset analysis accuracy;
and iteratively updating the applicable workpiece polishing analysis model based on the PSO algorithm and the model analysis deviation degree to obtain the applicable workpiece polishing optimization analysis model.
7. The method of claim 6, wherein said obtaining a fit-for-work-polishing optimization analysis model comprises:
constructing a particle optimization space according to the model training parameters of the applicable workpiece polishing analysis model;
initializing the particle optimization space to obtain particle swarm constraint parameters, and iteratively calculating a particle swarm fitness function according to the model analysis deviation degree and the particle swarm constraint parameters;
when a preset termination condition is reached, obtaining a first output result of the particle swarm fitness function, wherein the first output result comprises optimal result particles;
and mapping the optimal result particles to the applicable workpiece polishing analysis model for optimization training to obtain the applicable workpiece polishing optimization analysis model.
8. A robot control system based on a flexible perception neural network, characterized in that the system comprises:
the robot forming module is used for the flexible polishing robot and comprises a visual information acquisition module, a force axis sensor module, a data interaction module and a motion trail control module;
the visual information acquisition module is used for acquiring images of the workpiece to be processed through the visual information acquisition module to obtain image information of the workpiece to be processed;
the data interaction module is used for uploading the image information of the workpiece to be processed to a polishing robot control system for interactive analysis based on the data interaction module;
the polishing robot control system inputs the target polishing standard information and the image information of the workpiece to be processed into an applicable workpiece polishing analysis model to obtain a model output result, wherein the model output result comprises a polishing scheme of the workpiece to be processed;
the motion trail control module is used for sending the polishing scheme of the workpiece to be processed to the flexible polishing robot through the data interaction module and controlling the flexible polishing robot to polish the workpiece according to the polishing scheme of the workpiece to be processed through the motion trail control module;
the force axis sensor module is used for acquiring workpiece polishing force information in a workpiece polishing process through the force axis sensor module, and performing force compensation analysis according to the workpiece polishing force information to obtain action force compensation information;
and the grinding track correction module is used for capturing postures according to the visual information acquisition module, obtaining grinding pose spatial information, and performing grinding track correction on the grinding scheme of the workpiece to be processed through analysis of the action force compensation information and the grinding pose spatial information through a flexible sensing network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210588502.5A CN114905515B (en) | 2022-05-26 | 2022-05-26 | Robot control method and system based on flexible perception neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210588502.5A CN114905515B (en) | 2022-05-26 | 2022-05-26 | Robot control method and system based on flexible perception neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114905515A true CN114905515A (en) | 2022-08-16 |
| CN114905515B CN114905515B (en) | 2023-05-12 |
Family
ID=82768939
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210588502.5A Active CN114905515B (en) | 2022-05-26 | 2022-05-26 | Robot control method and system based on flexible perception neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114905515B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116525517A (en) * | 2023-06-19 | 2023-08-01 | 苏州鸿安机械股份有限公司 | Positioning control method and system for conveying semiconductor wafers |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105643636A (en) * | 2016-04-13 | 2016-06-08 | 广州文冲船厂有限责任公司 | Robot polishing device |
| CN107838920A (en) * | 2017-12-20 | 2018-03-27 | 芜湖哈特机器人产业技术研究院有限公司 | A kind of robot polishing Force control system and method |
| CN108388702A (en) * | 2018-01-30 | 2018-08-10 | 河南工程学院 | Engineering ceramics electrical discharge machining effect prediction method based on PSO neural networks |
| CN108509947A (en) * | 2018-01-29 | 2018-09-07 | 佛山市南海区广工大数控装备协同创新研究院 | A kind of automatic identification polishing process based on artificial neural network |
| CN108972573A (en) * | 2018-06-12 | 2018-12-11 | 浙江大学 | A kind of industrial robot automation wheel hub polishing system and method based on characteristics of image identification |
| CN109732450A (en) * | 2019-02-27 | 2019-05-10 | 重庆理工大学 | A kind of blade polishing processing method neural network based |
| CN111975579A (en) * | 2020-07-29 | 2020-11-24 | 华南理工大学 | Robot constant-force polishing system based on polishing model and iterative algorithm |
| CN112171458A (en) * | 2020-11-27 | 2021-01-05 | 大捷智能科技(广东)有限公司 | Intelligent mold polishing platform and polishing method |
| CN112757057A (en) * | 2021-01-19 | 2021-05-07 | 武汉海默机器人有限公司 | Intelligent manual-teaching-free grinding and polishing method and system integrating visual depth analysis |
| US20220032461A1 (en) * | 2020-07-31 | 2022-02-03 | GrayMatter Robotics Inc. | Method to incorporate complex physical constraints in path-constrained trajectory planning for serial-link manipulator |
| WO2022062915A1 (en) * | 2020-09-23 | 2022-03-31 | 广东博智林机器人有限公司 | Method and apparatus for controlling grinding robot, electronic device, and storage medium |
-
2022
- 2022-05-26 CN CN202210588502.5A patent/CN114905515B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105643636A (en) * | 2016-04-13 | 2016-06-08 | 广州文冲船厂有限责任公司 | Robot polishing device |
| CN107838920A (en) * | 2017-12-20 | 2018-03-27 | 芜湖哈特机器人产业技术研究院有限公司 | A kind of robot polishing Force control system and method |
| CN108509947A (en) * | 2018-01-29 | 2018-09-07 | 佛山市南海区广工大数控装备协同创新研究院 | A kind of automatic identification polishing process based on artificial neural network |
| CN108388702A (en) * | 2018-01-30 | 2018-08-10 | 河南工程学院 | Engineering ceramics electrical discharge machining effect prediction method based on PSO neural networks |
| CN108972573A (en) * | 2018-06-12 | 2018-12-11 | 浙江大学 | A kind of industrial robot automation wheel hub polishing system and method based on characteristics of image identification |
| CN109732450A (en) * | 2019-02-27 | 2019-05-10 | 重庆理工大学 | A kind of blade polishing processing method neural network based |
| CN111975579A (en) * | 2020-07-29 | 2020-11-24 | 华南理工大学 | Robot constant-force polishing system based on polishing model and iterative algorithm |
| US20220032461A1 (en) * | 2020-07-31 | 2022-02-03 | GrayMatter Robotics Inc. | Method to incorporate complex physical constraints in path-constrained trajectory planning for serial-link manipulator |
| WO2022062915A1 (en) * | 2020-09-23 | 2022-03-31 | 广东博智林机器人有限公司 | Method and apparatus for controlling grinding robot, electronic device, and storage medium |
| CN112171458A (en) * | 2020-11-27 | 2021-01-05 | 大捷智能科技(广东)有限公司 | Intelligent mold polishing platform and polishing method |
| CN112757057A (en) * | 2021-01-19 | 2021-05-07 | 武汉海默机器人有限公司 | Intelligent manual-teaching-free grinding and polishing method and system integrating visual depth analysis |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116525517A (en) * | 2023-06-19 | 2023-08-01 | 苏州鸿安机械股份有限公司 | Positioning control method and system for conveying semiconductor wafers |
| CN116525517B (en) * | 2023-06-19 | 2023-10-13 | 苏州鸿安机械股份有限公司 | Positioning control method and system for conveying semiconductor wafers |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114905515B (en) | 2023-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108280856B (en) | Unknown object grabbing pose estimation method based on mixed information input network model | |
| CN110314854B (en) | Workpiece detecting and sorting device and method based on visual robot | |
| CN108921893B (en) | Image cloud computing method and system based on online deep learning SLAM | |
| CN104915957B (en) | A kind of matching antidote for improving industrial robot 3D vision accuracy of identification | |
| CN112819772B (en) | High-precision rapid pattern detection and recognition method | |
| CN114905515B (en) | Robot control method and system based on flexible perception neural network | |
| CN110065068B (en) | Robot assembly operation demonstration programming method and device based on reverse engineering | |
| CN113034600B (en) | Template matching-based texture-free planar structure industrial part identification and 6D pose estimation method | |
| CN116525517B (en) | Positioning control method and system for conveying semiconductor wafers | |
| JP2019057250A (en) | Work-piece information processing system and work-piece recognition method | |
| CN113822810A (en) | Method for positioning workpiece in three-dimensional space based on machine vision | |
| Tiwari et al. | An image processing and computer vision framework for efficient robotic sketching | |
| CN110728286A (en) | Abrasive belt grinding material removal rate identification method based on spark image | |
| CN112530267B (en) | Intelligent mechanical arm teaching method based on computer vision and application | |
| CN115157272B (en) | Automatic programming system based on visual scanning | |
| CN116740044A (en) | Copper pipe milling surface processing method and system based on visual detection and control | |
| CN114926531A (en) | Binocular vision based method and system for autonomously positioning welding line of workpiece under large visual field | |
| US11685605B2 (en) | Reinforcement learning based conveyoring control | |
| CN111553437A (en) | Neural network based image classification method | |
| CN111242159B (en) | Image recognition and robot automatic positioning method based on product features | |
| Guo | Research on vision measurement system of mechanical workpiece based on machine vision | |
| CN117372423A (en) | Visual guiding method and device for machining path of manipulator and manipulator | |
| CN117226655A (en) | Automatic workpiece polishing device and method | |
| CN115091283B (en) | Control and adjustment method and system for efficient grinding of crankshaft | |
| Nie | Perspective of Methods for Improving the Accuracy of Industrial Robots |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |