CN113282173B - Double-arm robot remote real-time control system and method based on virtual reality - Google Patents

Double-arm robot remote real-time control system and method based on virtual reality Download PDF

Info

Publication number
CN113282173B
CN113282173B CN202110556471.0A CN202110556471A CN113282173B CN 113282173 B CN113282173 B CN 113282173B CN 202110556471 A CN202110556471 A CN 202110556471A CN 113282173 B CN113282173 B CN 113282173B
Authority
CN
China
Prior art keywords
double
ros
unity
virtual reality
arm robot
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110556471.0A
Other languages
Chinese (zh)
Other versions
CN113282173A (en
Inventor
温淑焕
贾仕东
方斌
孙富春
刘鑫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yanshan University
Original Assignee
Yanshan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yanshan University filed Critical Yanshan University
Priority to CN202110556471.0A priority Critical patent/CN113282173B/en
Publication of CN113282173A publication Critical patent/CN113282173A/en
Application granted granted Critical
Publication of CN113282173B publication Critical patent/CN113282173B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1602Programme controls characterised by the control system, structure, architecture
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1656Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1664Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition
    • G06V40/28Recognition of hand or arm movements, e.g. recognition of deaf sign language

Abstract

The invention provides a remote real-time control system and a remote real-time control method for a double-arm robot based on virtual reality, which belong to the technical field of teleoperation of mechanical arms, and comprise virtual reality equipment for finishing interaction between a human body and the double-arm robot, a simulation system of the virtual reality equipment, a video acquisition system for acquiring an image of the double-arm robot, a computer and a software control subsystem; the method comprises the steps of planning an experimental area, acquiring arm postures, designing an experimental platform, transmitting posture information, designing a control program of the double-arm robot, displaying a virtual reality picture and the like. The invention introduces the virtual reality technology into the control of the mechanical arm, is quicker and simpler than the control mode of the traditional control lever and control panel, simultaneously feeds back the motion video of the double-arm robot to the virtual reality equipment in real time, is very beneficial to inexperienced operators, realizes a better man-machine interaction mode and has safer system.

Description

Double-arm robot remote real-time control system and method based on virtual reality
Technical Field
The invention relates to the technical field of teleoperation of mechanical arms, in particular to a system and a method for remotely controlling a double-arm robot in real time based on virtual reality.
Background
Nowadays, the robot technology is rapidly developed, more and more robots are applied to various fields, the traditional remote robot control mode usually utilizes a control lever and a control panel, the experience and technical requirements on operators in operation tasks are high, the tasks can not be completed by simulating human actions, and the response to emergencies in remote environments is poor. Particularly in dangerous operating environments, the operator is required to perform remote control and adjust the change of the remote operating environment in real time.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a double-arm robot remote real-time control system and method based on virtual reality, wherein the control task of the double-arm robot is completed by acquiring the posture information of the human arm through virtual reality equipment, and the high-precision target grabbing task of the mechanical arm is realized.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
a remote real-time control system of a double-arm robot based on virtual reality comprises virtual reality equipment for finishing the interaction between a human body and the double-arm robot, a simulation system of the virtual reality equipment, a video acquisition system for acquiring images of the double-arm robot, a computer and a software control subsystem;
the virtual reality equipment acquires the posture information of the human arm and displays the posture information in the virtual reality simulation system;
the video acquisition system acquires image information of the double-arm robot by using the camera and displays the image information in the virtual reality simulation system in real time, and when the double-arm robot is controlled in real time by using the virtual reality equipment, the image acquired by the camera is displayed in the virtual reality equipment in real time;
the double-arm robot is a controlled object of the whole system and is used for realizing corresponding actions according to control instructions; the computer and software control subsystem is used for restoring the motion trail of the human body arm.
The technical scheme of the invention is further improved as follows: the virtual reality equipment comprises a head-mounted display and two handles; the head-mounted display is used for observing the motion state of the double-arm robot in real time; the two handles are used for acquiring the posture information of the arms of the human body;
the video acquisition system comprises a first camera for acquiring a depth image of the double-arm robot and a second camera for acquiring an RGB (red, green and blue) image of the double-arm robot;
2 computers are arranged, one computer uses a Windows system, and the other computer uses a Ubuntu system;
the software control subsystem comprises a Unity system for acquiring attitude information of two handles in the virtual reality equipment and an ROS system for completing communication with the Unity system and control of the double-arm robot.
A remote real-time control method of a double-arm robot based on virtual reality uses a remote real-time control system of the double-arm robot based on virtual reality, and comprises the following steps:
step 1, planning an experimental area: the human-computer interaction between an operator and the double-arm robot is carried out in a test area;
step 2, acquiring the posture of the arm and designing an experimental platform:
firstly, acquiring posture information of a human arm by using two handles of virtual reality equipment, constructing a double-arm robot arm model in a Unity system 3D simulation platform, and observing the motion state of the double-arm robot arm in real time in a head display of the virtual reality equipment;
then, designing data conversion from the six-dimensional attitude data of the tail end to the mechanical arm of the double-arm robot, and constructing an ROS system and a related function package of the double-arm robot control platform by adopting a time-based difference mapping mode;
and 3, transmitting attitude information:
the six-dimensional handle posture information acquired by the Windows system is transmitted to the Ubuntu system in a network communication mode, and then the control of the double-arm robot is completed;
and 4, designing a control program of the double-arm robot:
the data sent in the step 3 is the tail end posture of the handle, when switches on two sides of the handle controller are pressed, the current position and direction are converted into information in the ROS system from a Unity coordinate system and are issued to the nodes through themes, and the mechanical arm of the double-arm robot is moved after a solution is found by using a built-in IK resolver of the double-arm robot ROS system; in order to make the movement of the mechanical arm of the double-arm robot smoother and more natural, a control mode of difference increment is adopted, the obtained attitude data is differed from the previous moment, the control mode is relative, the absolute initial state is not required to be adjusted in each teleoperation task, and meanwhile, the fine adjustment can be carried out on the mechanical arm of the double-arm robot in the control process, so that a more precise task is completed;
and 5, displaying a virtual reality picture:
the camera I and the camera II are connected to a Ubuntu system computer, the Unity platform acquires the posture information of the tail end of the handle in the Windows system computer, and the ROS platform acquires the depth image in the Ubuntu system computer; and then, the posture information of the tail end of the arm of the human body is acquired by the two handles which transmit the information through network communication and transmitted to another computer provided with the ROS, and the depth image is transmitted to a head-mounted display of the virtual reality equipment.
The technical scheme of the invention is further improved as follows: in step 1, planning an experimental area to ensure enough space; the room is sized to have at least 2m x 1.5m experimental zones; moving all the obstacles out of the experimental area, and placing the computer near the experimental area; the virtual reality device cable extends 5 meters from the computer, ensuring that the two-arm robot is in the middle of the two positioning cameras and the operator can do teleoperation tasks in a standing or sitting position.
The technical scheme of the invention is further improved as follows: the ROS system and Unity system use different coordinate systems; the conversion listener first converts the position and rotation of the ROS system, as in equations (1), (2):
position:
x unity =-x ros (1)
y unity =-z r o s
z unity =-y r o s
in the formula, x unity ,y unity ,z unity Representing three-dimensional coordinates, x, in a Unity system ros ,y ros ,z ros Representing three-dimensional coordinates in the ROS system;
quaternion:
qx unity =qx ros (2)
qy unity =-qz r o s
qz unity =qy r o s
qw unity =qw r o s
in the formula, qx unity ,qy unity ,qz unity ,qw unity Representing quaternions, qx, in Unity systems ros ,qy ros ,qz ros ,qw ros Representing quaternions in the ROS system;
the conversion from the Unity system to the ROS system is derived from the formulas (1) and (2), and the rotation angle is obtained through quaternion operation, as shown in the formula (3):
(qx,qy,qz,qw) ros =(qx,qz,-qy,qw) unity *(0,1,0,0) (3)。
due to the adoption of the technical scheme, the invention has the technical progress that:
1. the invention combines the virtual reality equipment and the camera, utilizes the camera to collect teleoperation pictures when controlling the mechanical arm, transmits the collected pictures to the virtual reality environment in real time, and can adjust own operation tasks in real time according to the pictures.
2. The invention introduces the virtual reality technology into the control of the mechanical arm, is quicker and simpler than the control mode of the traditional control lever and control panel, simultaneously feeds back the motion video of the double-arm robot to the virtual reality equipment in real time, is very beneficial to inexperienced operators, realizes a better man-machine interaction mode and has safer system.
3. The invention increases the presence of operators.
Drawings
FIG. 1 is a schematic diagram of an operating system of the present invention;
FIG. 2 is a schematic diagram of the experimental area and the coordinate positioning of the camera implemented in the present invention;
fig. 3 is a flow chart of an embodiment of the present invention.
Detailed Description
The invention is further described in detail below with reference to the drawings and examples:
as shown in fig. 1 to 3, a system and a method for remote real-time control of a two-arm robot based on virtual reality are provided, in this embodiment, the virtual reality device employs an HTC five; the sensing device adopts a Kinect camera, and the two-arm robot is Baxter; the system is roughly divided into the following modules:
designing a software platform: the HTC five pro is first connected to a computer running the Unity system. The Unity system and the ROS system are connected through WebSocket. Connecting the ROS to a remote Windows machine running Unity, designing and reading HTC five pro handle information at Unity, and then designing six-dimensional gestures to be read, which are transmitted on the ROS theme by Socket communication. The gesture information of the handle is sent through the WebSocket connection, and when a user presses buttons on two sides of the clamp holder, the gesture of the clamp holder of the user is sent to the mechanical arm.
And (3) attitude information transmission: the framework uses two computers, one computer uses a Windows system, the other computer uses an ROS system, the six-dimensional handle attitude information acquired by the Windows system is transmitted to the ROS system in a network communication mode, and then the robot is controlled. Firstly, connecting HTC vive pro equipment to a Windows system computer, connecting a Kinect camera to a Ubuntu system computer, acquiring the posture information of the tail end of a handle by a Unity platform in the Windows system computer, and then acquiring the posture information of the tail end of the arm of a human body by two handles to which the information is transmitted by utilizing network communication and transmitting the posture information to another computer provided with ROS.
Designing a control program of the double-arm robot: after data transmission is completed, the robot is controlled and designed, the sent data is the tail end gesture of the handle, when switches (side buttons on the HTC five pro) on two sides of the handle controller are pressed, the current position and direction are converted into information in the ROS system from the Unity coordinate system and are issued to nodes through themes, and the Baxter mechanical arm is moved after a solution is found by using a built-in IK resolver of the ROS system of the robot. In order to make the movement of the Baxter mechanical arm more smooth and natural, a control mode of difference increment is adopted, the obtained attitude data is differed from the last moment, the control mode is relative, the teleoperation task is not required to be adjusted to an absolute initial state every time, and meanwhile, the Baxter mechanical arm can be finely adjusted in the control process, so that a more precise task is completed. The trigger on the handle can also be used to open and close the gripper of the Baxter robot arm, which can also send a message to the robot in the form of a theme.
The ROS system and Unity system use different coordinate systems. The conversion listener therein first converts the position and rotation of the ROS system as in the following equations (1), (2).
Position:
x unity =-x ros (1)
y unity =-z r o s
z unity =-y r o s
wherein [ x ] unity ,y unity ,z unity ]Representing three-dimensional coordinates, [ x ] in the Unity system ros ,y ros ,z ros ]Representing three-dimensional coordinates in the ROS system.
Quaternion:
qx unity =qx ros (2)
qy unity =-qz r o s
qz unity =qy r o s
qw unity =qw r o s
wherein [ qx [ ] unity ,qy unity ,qz unity ,qw unity ]Representing quaternions, [ qx ] in Unity systems ros ,qy ros ,qz ros ,qw ros ]Representing quaternions in the ROS system.
The conversion from the Unity system to the ROS system can be derived from equations (1) and (2), and the rotation angle is obtained by quaternion operation, as equation (3).
(qx,qy,qz,qw) ros =(qx,qz,-qy,qw) unity *(0,1,0,0) (3)
Displaying a virtual reality picture: the Kinect camera is connected to the Ubuntu system computer, the Unity platform acquires the posture information of the tail end of the handle in the Windows system computer, and the ROS platform acquires the depth image in the Ubuntu system computer. Then, network communication is utilized to transmit information to the two handles, the posture information of the tail end of the arm of the human body is obtained and transmitted to another computer provided with ROS, and the depth image is transmitted to a head-mounted display of HTC vivpro;
experimental results show that the teleoperation task based on VR virtual reality technology has higher precision than the teleoperation task based on wearable equipment, and can well complete complex operation tasks.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Claims (3)

1. The utility model provides a long-range real-time control system of double-armed robot based on virtual reality which characterized in that: the system comprises virtual reality equipment for finishing interaction between a human body and a double-arm robot, a simulation system of the virtual reality equipment, a video acquisition system for acquiring an image of the double-arm robot, a computer and a software control subsystem;
the virtual reality equipment acquires the posture information of the human arm and displays the posture information in the virtual reality simulation system;
the video acquisition system acquires image information of the double-arm robot by using the camera and displays the image information in the virtual reality simulation system in real time, and when the double-arm robot is controlled in real time by using the virtual reality equipment, the image acquired by the camera is displayed in the virtual reality equipment in real time;
the double-arm robot is a controlled object of the whole system and is used for realizing corresponding action according to a control instruction; the computer and software control subsystem is used for restoring the motion trail of the human body arm;
the virtual reality equipment comprises a head-mounted display and two handles; the head-mounted display is used for observing the motion state of the double-arm robot in real time; the two handles are used for acquiring the posture information of the arms of the human body;
the video acquisition system comprises a first camera for acquiring a depth image of the double-arm robot and a second camera for acquiring an RGB (red, green and blue) image of the double-arm robot;
2 computers are arranged, one computer uses a Windows system, and the other computer uses a Ubuntu system;
the software control subsystem comprises a Unity system for acquiring posture information of two handles in the virtual reality equipment and an ROS system for completing communication with the Unity system and control of the double-arm robot; the Unity system is connected with the ROS system through WebSocket;
the control method of the double-arm robot remote real-time control system based on the virtual reality comprises the following steps:
step 1, planning an experimental area: the human-computer interaction between an operator and the double-arm robot is carried out in a test area;
step 2, acquiring the posture of the arm and designing an experimental platform:
firstly, acquiring posture information of a human arm by using two handles of virtual reality equipment, constructing a double-arm robot arm model in a Unity system 3D simulation platform, and observing the motion state of the double-arm robot arm in real time in a head display of the virtual reality equipment;
then, designing data conversion from the six-dimensional attitude data of the tail end to the mechanical arm of the double-arm robot, and constructing an ROS system and a related function package of the double-arm robot control platform by adopting a time-based difference mapping mode;
and 3, transmitting attitude information:
the six-dimensional attitude information of the handle acquired by the Windows system is transmitted to the Ubuntu system in a network communication mode, and then the control of the double-arm robot is completed;
and 4, designing a control program of the double-arm robot:
the data sent in the step 3 is the tail end posture of the handle, when switches on two sides of the handle controller are pressed, the current position and direction are converted into information in the ROS system from a Unity coordinate system and are issued to the nodes through themes, and the mechanical arm of the double-arm robot is moved after a solution is found by using a built-in IK resolver of the double-arm robot ROS system; in order to make the movement of the mechanical arm of the double-arm robot smoother and more natural, a control mode of difference increment is adopted, the obtained attitude data is differed from the previous moment, the control mode is relative, the absolute initial state is not required to be adjusted in each teleoperation task, and meanwhile, the fine adjustment can be carried out on the mechanical arm of the double-arm robot in the control process, so that a more precise task is completed;
and 5, displaying a virtual reality picture:
the camera I and the camera II are connected to a Ubuntu system computer, the Unity platform acquires the posture information of the tail end of the handle in the Windows system computer, and the ROS platform acquires the depth image in the Ubuntu system computer; and then, the two handles which transmit information are used for acquiring the posture information of the tail end of the arm of the human body and transmitting the posture information to another computer with ROS (reactive oxygen species), and the depth image is transmitted to a head-mounted display of the virtual reality device.
2. The remote real-time control system for the two-arm robot based on the virtual reality as claimed in claim 1, wherein: in step 1, planning an experimental area to ensure enough space; the room is sized to have at least 2m x 1.5m experimental zones; moving all the obstacles out of the experimental area, and placing a computer near the experimental area; the virtual reality device cable extends 5 meters from the computer, ensuring that the two-arm robot is in the middle of the two positioning cameras and the operator can do teleoperation tasks in standing or sitting position.
3. The remote real-time control system for the two-arm robot based on the virtual reality as claimed in claim 1, wherein: the ROS system and Unity system use different coordinate systems; the conversion listener first converts the position and rotation of the ROS system, as in equations (1), (2):
position:
x unity =-x ros (1)
y unity =-z ros
z unity =-y ros
in the formula, x unity ,y unity ,z unity Representing three-dimensional coordinates, x, in a Unity system ros ,y ros ,z ros Representing three-dimensional coordinates in the ROS system;
quaternion:
qx unity =qx ros (2)
qy unity =-qz ros
qz unity =qy ros
qw unity =qw ros
in the formula, qx unity ,qy unity ,qz unity ,qw unity Representing quaternions, qx, in Unity systems ros ,qy ros ,qz ros ,qw ros Representing quaternions in the ROS system;
the conversion from the Unity system to the ROS system is derived from the formulas (1) and (2), and the rotation angle is obtained through quaternion operation, as shown in the formula (3):
(qx,qy,qz,qw) ros =(qx,qz,-qy,qw) unity *(0,1,0,0) (3)。
CN202110556471.0A 2021-05-21 2021-05-21 Double-arm robot remote real-time control system and method based on virtual reality Active CN113282173B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110556471.0A CN113282173B (en) 2021-05-21 2021-05-21 Double-arm robot remote real-time control system and method based on virtual reality

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110556471.0A CN113282173B (en) 2021-05-21 2021-05-21 Double-arm robot remote real-time control system and method based on virtual reality

Publications (2)

Publication Number Publication Date
CN113282173A CN113282173A (en) 2021-08-20
CN113282173B true CN113282173B (en) 2023-01-03

Family

ID=77280595

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110556471.0A Active CN113282173B (en) 2021-05-21 2021-05-21 Double-arm robot remote real-time control system and method based on virtual reality

Country Status (1)

Country Link
CN (1) CN113282173B (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106527177A (en) * 2016-10-26 2017-03-22 北京控制工程研究所 Multi-functional and one-stop type remote control design, the simulation system and method thereof

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104057450B (en) * 2014-06-20 2016-09-07 哈尔滨工业大学深圳研究生院 A kind of higher-dimension motion arm teleoperation method for service robot
US9643314B2 (en) * 2015-03-04 2017-05-09 The Johns Hopkins University Robot control, training and collaboration in an immersive virtual reality environment
EP3598270A1 (en) * 2018-07-17 2020-01-22 Technische Universität München Method and control unit for controlling a virtual reality display, virtual reality display and virtual reality system
CN110119207A (en) * 2019-05-14 2019-08-13 重庆大学 Virtual reality machines people interactive system and exchange method based on human body natural's signal
CN111459277B (en) * 2020-04-01 2023-05-30 重庆大学 Mechanical arm teleoperation system based on mixed reality and interactive interface construction method

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106527177A (en) * 2016-10-26 2017-03-22 北京控制工程研究所 Multi-functional and one-stop type remote control design, the simulation system and method thereof

Also Published As

Publication number Publication date
CN113282173A (en) 2021-08-20

Similar Documents

Publication Publication Date Title
US11440179B2 (en) System and method for robot teaching based on RGB-D images and teach pendant
CN110405730B (en) Human-computer interaction mechanical arm teaching system based on RGB-D image
Hashimoto et al. Touchme: An augmented reality based remote robot manipulation
CN100484726C (en) Flexible and remote-controlled operation platform for robot based on virtual reality
CN111906784A (en) Pharyngeal swab double-arm sampling robot based on machine vision guidance and sampling method
CN108214445B (en) ROS-based master-slave heterogeneous teleoperation control system
WO2018219194A1 (en) Cyber arm-based teleoperation system for space station robot
Pan et al. Augmented reality-based robot teleoperation system using RGB-D imaging and attitude teaching device
CN111438673B (en) High-altitude operation teleoperation method and system based on stereoscopic vision and gesture control
Naceri et al. Towards a virtual reality interface for remote robotic teleoperation
CN110039547A (en) A kind of human-computer interaction terminal and method of flexible mechanical arm remote operating
CN110682291B (en) Robot teleoperation system based on VR and teleoperation method thereof
CN113103230A (en) Human-computer interaction system and method based on remote operation of treatment robot
CN110977981A (en) Robot virtual reality synchronization system and synchronization method
CN110355750A (en) Interaction control method towards remote operating hand eye coordination
CN111113414B (en) Robot three-dimensional space scale prompting method and system based on screen identification
CN113282173B (en) Double-arm robot remote real-time control system and method based on virtual reality
CN109857258B (en) Virtual remote control method, device and system
CN116197899A (en) Active robot teleoperation system based on VR
CN115157261A (en) Flexible mechanical arm teleoperation man-machine interaction device and method based on mixed reality
CN116252298A (en) Remote operation method and system for robot based on force feedback technology
CN112947117A (en) Multi-ROV underwater cooperative operation simulation system
Zhou et al. Development of a synchronized human-robot-virtuality interaction system using cooperative robot and motion capture device
CN112947238B (en) Industrial robot real-time control system based on VR technique
Bai et al. Kinect-based hand tracking for first-person-perspective robotic arm teleoperation

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant