CN108422435A - Remote monitoring and control system based on augmented reality - Google Patents
Remote monitoring and control system based on augmented reality Download PDFInfo
- Publication number
- CN108422435A CN108422435A CN201810236758.3A CN201810236758A CN108422435A CN 108422435 A CN108422435 A CN 108422435A CN 201810236758 A CN201810236758 A CN 201810236758A CN 108422435 A CN108422435 A CN 108422435A
- Authority
- CN
- China
- Prior art keywords
- manipulator
- monitoring
- module
- augmented reality
- robot body
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 131
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 44
- 238000004088 simulation Methods 0.000 claims abstract description 63
- 230000002452 interceptive effect Effects 0.000 claims abstract description 19
- 238000007689 inspection Methods 0.000 claims abstract description 6
- 230000033001 locomotion Effects 0.000 claims description 17
- 238000003745 diagnosis Methods 0.000 claims description 15
- 238000003860 storage Methods 0.000 claims description 15
- 230000004927 fusion Effects 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 8
- 238000005457 optimization Methods 0.000 claims description 6
- 238000013154 diagnostic monitoring Methods 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 3
- 230000002194 synthesizing effect Effects 0.000 claims description 3
- 238000013480 data collection Methods 0.000 abstract 1
- 230000000007 visual effect Effects 0.000 abstract 1
- 238000000034 method Methods 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000004886 head movement Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000000465 moulding Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/006—Controls for manipulators by means of a wireless system for controlling one or several manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
- B25J19/04—Viewing devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mechanical Engineering (AREA)
- Geometry (AREA)
- Robotics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Computer Networks & Wireless Communication (AREA)
- Manipulator (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention relates to a remote monitoring and control system based on augmented reality, which comprises an interactive simulation control mode and a monitoring mode, wherein in the interactive simulation mode, a server receives depth images shot by an RGBD (red, green and blue) camera on a mobile robot, and synthesizes the depth images shot at different angles into a scene three-dimensional point cloud model; then combining a scene three-dimensional point cloud model, operating a manipulator simulation model, planning a path, driving the manipulator simulation model to move by using the path, and finally superposing the dynamic manipulator simulation model in a color image shot by an RGBD camera to realize virtual-real combined simulation display and path planning; under the monitoring mode, monitoring data are collected through a remote data collection system, the monitoring images of the monitored objects are obtained through patrol and inspection of the mobile robot, the monitoring data are obtained through a server, the monitoring data and the monitoring data are displayed in a superposition mode, and a display interface is more visual.
Description
Technical field
The long-range monitoring and control system that the present invention relates to a kind of based on augmented reality, belong to intelligence manufacture and computer are surveyed
Amount and control field.
Background technology
Current remote is monitored with control extensively using production, however in terms of monitoring, and current monitoring system is prison
It surveys video and monitoring data shows that monitoring personnel needs oneself video content and monitoring data to link together, interface respectively
Reality is not intuitive enough.In addition, in control aspect, controllers are merely able to estimate the three-dimensional letter of remote environment by color image
Breath, plans the path of manipulator, controls remote mechanical hand by this method and completes operation, can cause robot control not accurate enough in this way
Really.
The patent of invention of Publication No. CN106863303A《CN106863303A》, specific technical solution is：Pass through peace
The angular displacement in each joint is acquired mounted in the encoder of each joint of manipulator, and send gathered data to computer disposal, computer
Robot Path is cooked up according to each joint angle displacement data, and control machinery hand completes the welding of weld seam along this path.The skill
Art scheme is device context path planning mode, cannot be satisfied the demand of current enterprise remote path planning.
Invention content
The long-range monitoring and control system that in order to solve the above technical problem, the present invention provides a kind of based on augmented reality,
It realizes that the fusion of the video surveillance and data monitoring of long-range inspection application shows and generates robot distal end in server end to grasp
The local path of work is planned and simulation.
Technical scheme is as follows：
A kind of long-range monitoring and control system based on augmented reality, including interactive simulation control model, the pattern pass through service
Device and the mobile robot in remote equipment and environment complete the local of manipulator far-end operation in such a way that actual situation combines
Path planning and simulation, specifically：
The mobile robot includes robot body, and the manipulator on robot body and RGBD cameras；
The server includes that a distal environment rebuilds module, manipulator operating simulation module and augmented reality display module；
The distal environment rebuilds the depth image that module receives the RGBD cameras shooting, and the depth image that different angle is shot
Synthesize scene three-dimensional point cloud model；The manipulator operating simulation module establishes manipulator simulation model, then in conjunction with scene three
Point cloud model, manipulator's simulation model are tieed up, planning path finally utilizes manipulator simulation model described in the path drives
Movement；Dynamic manipulator simulation model is superimposed upon the coloured image of RGBD cameras shooting by the augmented reality display module
In, realize the simulative display that actual situation combines.
More preferably, collision detection is also completed under the interactive simulation control model, specifically：The manipulator operating simulation
Module is moved in scene three-dimensional point cloud model using the path drives manipulator simulation model, checks manipulator simulation model
Whether interfered with scene three-dimensional point cloud model, realizes actual situation collision detection；If there is interference, then path optimizing will
Path after optimization is sent to mobile robot, if there is no interference, the path is sent to mobile robot, moving machine
Device people completes operation task according to the movement of path clustering manipulator.
More preferably, in the mobile robot further include image capture controller, robot body controller and machinery
Hand controls；Described image collector controls camera head movement, and acquires coloured image and depth on the RGBD cameras
Image, the RGBD cameras are fixed on camera head；The robot body controller control robot body movement, and it is anti-
Present the orientation and posture of robot body；The movement of the Manipulator Controller control machinery hand；
Holder remote operating module, robot body remote operating module and manipulator remote operating mould are additionally provided on the server
Block；The holder remote operating module sends to image capture controller and instructs, and control camera head is moved according to instruction, to change
Become the shooting angle of RGBD cameras；The robot body remote operating module sends to robot body controller and instructs, control
Robot body is moved according to instruction；The manipulator remote operating module sends to Manipulator Controller and instructs, control machinery hand
It is moved according to instruction；
The distal environment rebuilds module when synthesizing scene three-dimensional point cloud model, passes through robot body remote operating module first
Control instruction is sent to corresponding robot body controller and Manipulator Controller respectively with manipulator remote operating module,
So that robot body and manipulator is remained stationary, is then referred to image capture controller transmission by holder remote operating module
It enables, controls camera head according to instruction multi-angle rotation, and in each angle shot depth image, to obtain the depth of different angle
Image is spent, and synthesizes scene three-dimensional point cloud model.
More preferably, the manipulator simulation model that the manipulator operating simulation module is established, including manipulator three-dimensional
Model and inverse kinematics model plan the end locus of points and posture of manipulator in interactive simulation environment, obtain distal point rail
Mark and posture sequence are sought manipulator and are respectively closed using the end locus of points and posture sequence as the input of the inverse kinematics model
The angle sequence of section, then reads the angle sequence in each joint of manipulator in order, and driving manipulator three dimensional virtual models respectively close
Section movement.
More preferably, the augmented reality display module is according to the current position of robot body, posture and camera head
Dynamic manipulator three dimensional virtual models are superimposed upon in the coloured image of RGBD cameras shooting by azimuth information, are then shown
Come, realizes the vision fusion of actual situation.
More preferably, the current position of the robot body, posture are sent by robot body remote operating module and are controlled
It instructs to robot body controller, obtains the information；The camera head azimuth information is by holder remote operating module to figure
It is instructed as acquisition controller is sent, obtains the azimuth information of camera head；
More preferably, the long-range monitoring and control system further include monitoring pattern, by remote equipment and environment
Remote data acquisition system and the server and mobile robot, and complete monitoring data and prison in such a way that actual situation combines
The fusion of altimetric image shows, specifically：
The remote data acquisition system acquires the monitoring data of each monitoring object, is then forwarded to server；Each monitoring object
Place is respectively provided with a label；
The mobile robot receives the control instruction from server, controls robot body and camera head movement, makes machine
Device human body moves at monitoring object, and controls the rotation of camera head, and RGBD cameras is made to take comprising label information
Monitoring image, and it is sent to server；
The server further includes equipment identification module and storage and fault diagnosis module；
The equipment identification module analyzes the monitoring image of the RGBD cameras shooting, and segmentation identifies the label in the monitoring image
Information, and send the inquiry request for including label information；
The storage and fault diagnosis module receive and preserve the monitoring data that the remote data acquisition system is sent, and diagnose
If monitoring object sends warning message with the presence or absence of failure there are failure；It is set when storage and fault diagnosis module receive to come from
When the inquiry request of standby identification module, the corresponding monitoring data of the label information are inquired according to label information, are then forwarded to institute
State augmented reality display module；
The augmented reality display module by monitoring data be superimposed upon RGBD cameras shooting monitoring image on corresponding label it is attached
Near field is shown, realizes that the Overlapping display of monitoring data and monitoring data, display interface are more intuitive.
More preferably, the remote data acquisition system includes industrial personal computer and sensor, described in each sensor accesses
Industrial personal computer is equipped with the sensor at each monitoring object, and each sensor is correspondingly arranged a label；
More preferably, the storage and fault diagnosis module are according to the correspondence between label, monitoring object and monitoring data
The monitoring data that the remote data acquisition system is sent are received and preserve, and according to preset alarm threshold value, by monitoring data
It is compared with corresponding alarm threshold value, if diagnostic monitoring object sends warning message with the presence or absence of failure there are failure；
When storage and fault diagnosis module receive the inquiry request from equipment identification module, label letter is inquired according to label information
Corresponding monitoring data are ceased, the augmented reality display module is then forwarded to.
More preferably, the monitoring image includes picture and video.
The present invention has the advantages that：
1, the present invention a kind of long-range monitoring and control system based on augmented reality, passes through the shifting in remote equipment and environment
Mobile robot obtains the depth image of environment, and the three-dimensional point cloud mould of synthesis remote equipment and environment is then calculated in server end
Type, and then planned by the local path of virtual simulation manipulator model realization mobile robot far-end operation in server end
And simulation, it is a kind of tele-control system that actual situation combines, improves the control accuracy of mobile robot；
2, the present invention a kind of long-range monitoring and control system based on augmented reality, is acquired by remote data acquisition system and is monitored
Data obtain the monitoring image of monitoring object, and in server end identification tag information, obtain then by mobile robot
Monitoring information, is then superimposed upon in monitoring image by the current state information of monitoring object, realizes the number of long-range inspection application
It is shown according to monitoring and live video monitoring or the fusion of data monitoring and image monitoring.
Description of the drawings
Fig. 1 is the system block diagram of the present invention；
Fig. 2 is the flow chart of interactive simulation control model of the present invention；
Fig. 3 is the flow chart of monitoring pattern of the present invention.
Reference numeral is expressed as in figure：
1, robot body；2, manipulator；3, RGBD cameras；4, camera head；5, robot body controller；6, machinery is manual
Device processed；7, image capture controller；10, robot body remote operating module；20, manipulator remote operating module；30, the distant behaviour of holder
Make module；40, distal environment rebuilds module；50, manipulator operating simulation module；60, augmented reality display module；70, equipment
Identification module；80, storage and fault diagnosis module；100, industrial personal computer；200, sensor；300, label.
Specific implementation mode
It is next in the following with reference to the drawings and specific embodiments that the present invention will be described in detail.
Embodiment one：
It please refers to Fig.1 and Fig. 2, a kind of long-range monitoring and control system based on augmented reality, including interactive simulation control model,
The pattern completes manipulator by server and the mobile robot in remote equipment and environment in such a way that actual situation combines
The local path of 2 far-end operations is planned and simulation, specifically：
The mobile robot includes robot body 1, and the manipulator 2 on robot body 1 and RGBD cameras 3；
The server includes that a distal environment rebuilds module 40, manipulator operating simulation module 50 and augmented reality display mould
Block 60；The distal environment rebuilds module 40 and receives the depth image that the RGBD cameras 3 are shot, and different angle is shot
Range image integration scene three-dimensional point cloud model, the manipulator operating simulation module 50 establish manipulator simulation model, then
In conjunction with scene three-dimensional point cloud model, manipulator's simulation model, planning path finally utilizes machinery described in the path drives
Hand simulation model moves；Dynamic manipulator simulation model is superimposed upon RGBD cameras 3 and clapped by the augmented reality display module 60
In the coloured image taken the photograph, the simulative display that actual situation combines is realized.The manipulator 2 include Multi-freedom-degreemanipulator manipulator, such as six from
By degree manipulator, three freedom degree manipulator or two degrees of freedom manipulator, degree of freedom is more, and the flexibility of manipulator is bigger.
Collision detection is also completed under the interactive simulation control model, specifically：The manipulator operating simulation module 50
It is moved using manipulator simulation model described in the path drives in the scene three-dimensional point cloud model, checks manipulator emulation
Whether model interferes with scene three-dimensional point cloud model, realizes actual situation collision detection；If there is interference, path optimizing, so
The path after optimization is sent to mobile robot afterwards, to control the movement of manipulator 2 in mobile robot, completes manipulator
2 operation task；The optimization rear path can also carry out simulative display by the augmented reality display module 60.
Further include image capture controller 7, robot body controller 5 and manipulator control in the mobile robot
Device 6；Described image collector controls camera head 4 and moves, and acquires the coloured image and depth map on the RGBD cameras 3
Picture, the RGBD cameras 3 are fixed on camera head 4；The robot body controller 5 controls robot body 1 and moves, and
Feed back the orientation and posture of robot body 1；The movement of 6 control machinery hand 2 of the Manipulator Controller；
Holder remote operating module 30, robot body remote operating module 10 and manipulator remote operating are additionally provided on the server
Module 20；The holder remote operating module 30 sends to image capture controller 7 and instructs, and control camera head 4 is transported according to instruction
It is dynamic, to change the shooting angle of RGBD cameras 3；The robot body remote operating module 10 is to robot body controller 5
Instruction is sent, control robot body 1 is moved according to instruction；The manipulator remote operating module 20 is sent out to Manipulator Controller 6
Instruction, control machinery hand 2 is sent to be moved according to instruction；
The distal environment rebuilds module 40 when synthesizing scene three-dimensional point cloud model, passes through robot body remote operating mould first
Block 10 and manipulator remote operating module 20 are sent to corresponding robot body controller 5 and Manipulator Controller 6 respectively
Control instruction makes robot body 1 and manipulator 2 remain stationary, and is then adopted to image by holder remote operating module 30
Collection controller 7 sends instruction, controls camera head 4 according to instruction multi-angle rotation, and in each angle shot depth image, thus
The depth image of different angle is obtained, and synthesizes scene three-dimensional point cloud model.
The manipulator simulation model that the manipulator operating simulation module 50 is established, including manipulator three dimensional virtual models and
Inverse kinematics model, in interactive simulation environment（Combine scene three-dimensional point cloud model）Plan the end locus of points of manipulator 2
And posture, the end locus of points and posture sequence are obtained, using the end locus of points and posture sequence as the inverse kinematics model
Input, seeks the angle sequence in 2 each joint of manipulator, reads the angle sequence in 2 each joint of manipulator, driving manipulator in order
Each joint motions of three dimensional virtual models.
The augmented reality display module 60 is according to 4 orientation of the current position of robot body 1, posture and camera head
Dynamic manipulator three dimensional virtual models are superimposed upon in the coloured image that RGBD cameras 3 are shot by information, are then shown, real
The vision fusion of existing actual situation.The current position of the robot body 1, posture are sent by robot body remote operating module 10
Control instruction obtains the information to robot body controller 5.4 azimuth information of the camera head passes through holder remote operating mould
Block 30 sends to image capture controller 7 and instructs, and obtains the azimuth information of camera head 4.The augmented reality display module 60
According to 4 azimuth information of the current position and attitude of robot body 1 and camera head, camera head 4 is obtained in world coordinate system
In（World coordinate system is the absolute coordinate system of system, and the coordinate of all the points is all on picture before not establishing user coordinate system
It is that respective positions are determined with the origin of the coordinate system）Orientation, to realize that augmented reality is registered, and then according to manipulator
Manipulator three dimensional virtual models are superimposed upon in the coloured image that RGBD cameras 3 are shot by the coordinate information of three dimensional virtual models.
Specifically, collision detection, the manipulator operating simulation module are also completed under the interactive simulation control model
50 are moved in the scene three-dimensional point cloud model using the path drives manipulator three dimensional virtual models, check manipulator three
Whether dimension dummy model interferes with scene three-dimensional point cloud model, realizes actual situation collision detection；If there is interference, optimize road
Then path after optimization is sent to mobile robot by diameter, received by the Manipulator Controller 6 of mobile robot and control shifting
The movement of manipulator 2 on mobile robot, completes the operation task of mobile robot.
The operational process of the interactive simulation control model is as follows：
Step 1, manipulator operating simulation module 50 establish the manipulator simulation model of manipulator 2, including manipulator three-dimensional
Model and inverse kinematics model；
Step 2 keeps robot body 1 and manipulator 2 fixed, and distal environment rebuilds module 40 and receives 3 bat of RGBD cameras
The depth image taken the photograph, and be scene three-dimensional point cloud model by the range image integration of different angle shooting；
Step 3, in interactive simulation environment the manipulator three dimensional virtual models of interactive programming the end locus of points and posture, obtain
The end locus of points and posture sequence；
Step 4, using the end locus of points and posture sequence as the input of the inverse kinematics model of manipulator 2, it is each to seek manipulator 2
The angle sequence in joint；
Step 5, the angle sequence for reading 2 each joint of manipulator in order, each joint motions of driving manipulator three dimensional virtual models；
It checks whether manipulator three dimensional virtual models interfere with scene three-dimensional point cloud model in motion process, realizes actual situation collision inspection
It surveys, if there is collision, then the path of optimization is sent to the Manipulator Controller 6 of mobile robot by path optimizing, to
Control machinery hand 2 completes operation task；
At the same time it can also according to the current position of robot body 1, posture and 4 azimuth information of camera head by dynamic machine
Tool hand three dimensional virtual models are superimposed upon in coloured image, and by augmented reality display module 60, realize the simulative display of actual situation.
The present invention a kind of long-range monitoring and control system based on augmented reality can calculate synthesis distal end in server end
The three-dimensional point cloud model of equipment and environment, and then pass through virtual simulation manipulator model realization mobile robot in server end
The local path of far-end operation is planned and simulation, is a kind of actual situation combination tele-control system, improves the control of mobile robot
Accuracy processed.
Embodiment two,
It please refers to Fig.1 and Fig. 3, the difference between this embodiment and the first embodiment lies in, embodiment two can execute interactive simulation control
Monitoring pattern is executed while molding formula, the interactive simulation molding formula repeats no more.
The monitoring pattern passes through remote data acquisition system in remote equipment and environment and the server
And mobile robot, and complete the fusion of monitoring data and monitoring image in such a way that actual situation combines and show, the monitoring image
Can be picture and/or video.Specifically：
The remote data acquisition system includes industrial personal computer 100 and sensor 200, and each sensor 200 accesses the work
Control machine 100 is equipped with the sensor 200 at each monitoring object, and each sensor 200 is correspondingly arranged a label 300；The mark
Label 300 include the label 300 of the unsymmetric forms such as Quick Response Code, bar code；
The mobile robot receives the control instruction from server, controls robot body 1 and RGBD cameras 3, makes machine
Human body 1 moves at monitoring object, and RGBD cameras 3 is made to take the monitoring image comprising label information（The monitoring image
It is a kind of coloured image）, and it is sent to server；Another embodiment is that RGBD cameras 3 are fixed by camera head 4, then
The rotation that camera head 4 is controlled by the control instruction of server, makes RGBD cameras 3 take the monitoring figure comprising label information
Picture.The mobile robot may also receive from the control instruction of server, the movement of control machinery hand 2；Specifically, it services
The robot body remote operating module 10 and holder remote operating module 30 of device are controlled to the robot body of mobile robot respectively
Device 5 and image capture controller 7 send control instruction, and controlling robot body 1 by robot body controller 5 moves to monitoring
At object, camera head 4 is controlled by image capture controller 7 and is rotated, RGBD cameras 3 is made to take the monitoring comprising label information
Image, realize remote control mobile robot acquisition monitoring object includes label information and the currently monitored object operating status
Monitoring image；The manipulator remote operating module 20 of the server can also be sent to the Manipulator Controller 6 of mobile robot
Control instruction realizes remote control manipulator 2 by the movement of 6 control machinery hand 2 of Manipulator Controller；
The server further includes equipment identification module 70 and storage and fault diagnosis module 80；
The equipment identification module 70 analyzes the monitoring image that the RGBD cameras 3 are shot, and segmentation identifies in the monitoring image
Label information, and send the inquiry request for including label information；
The storage and fault diagnosis module 80 connect according to the correspondence between label 300, monitoring object and monitoring data
Receive and preserve the monitoring data that the remote data acquisition system is sent, and according to preset alarm threshold value, by monitoring data with
Corresponding alarm threshold value is compared, and if diagnostic monitoring object sends warning message with the presence or absence of failure there are failure；When
When storage and fault diagnosis module 80 receive the inquiry request from equipment identification module 70, the label is inquired according to label information
The corresponding monitoring data of information are then forwarded to the augmented reality display module 60；
Monitoring data are superimposed upon corresponding label in the monitoring image that RGBD cameras 3 are shot by the augmented reality display module 60
Region is shown near 300, realizes monitoring data and monitoring image（That is the coloured image of monitoring object）Overlapping display, display
Interface is more intuitive.
The operational process of the interactive simulation monitoring pattern is as follows：
Step 10, operator pass through manipulator remote operating module 20, robot sheet in server side operation computer input device
Body remote operating module 10, holder remote operating module 30 are respectively to robot controller, robot body controller 5, Image Acquisition
Controller 7 sends control instruction, and control is located at remote equipment and manipulator 2, robot body 1 and camera head in environment
4 movements；
Step 20, equipment set the monitoring image that module receives 3 shooting of RGBD cameras, and segmentation identifies the label in monitoring image
Information；
The label information that step 30, foundation recognize, the 300 corresponding prison of label is inquired from storage and fault diagnosis module 80
Survey the current information of object；
The information inquired is sent to augmented reality display module 60 by step 40, is superimposed upon the monitoring figure of the shooting of RGBD cameras 3
As region is shown near upper corresponding label 300, the Overlapping display of monitoring information and monitoring image is realized.
The present invention a kind of long-range monitoring and control system based on augmented reality can control distal end in server local
Mobile robot obtains the monitoring image of monitoring object, and in server end identification tag information, obtains the current of monitoring object
Then monitoring information is superimposed upon in monitoring image by state, realize the data monitoring and live video prison of long-range inspection application
It surveys or the fusion of data monitoring and image monitoring is shown.
Example the above is only the implementation of the present invention is not intended to limit the scope of the invention, every to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of long-range monitoring and control system based on augmented reality, it is characterised in that：It, should including interactive simulation control model
Pattern completes manipulator by server and the mobile robot in remote equipment and environment in such a way that actual situation combines
(2) the local path planning and simulation of far-end operation, specifically：
The mobile robot includes robot body (1), and manipulator (2) and RGBD on robot body (1)
Camera (3)；
The server includes that distal environment reconstruction module (40), manipulator operating simulation module (50) and augmented reality are aobvious
Show module (60)；The distal environment rebuilds the depth image that module (40) receives RGBD cameras (3) shooting, and will be different
The range image integration scene three-dimensional point cloud model of angle shot；It is imitative that the manipulator operating simulation module (50) establishes manipulator
True mode, then in conjunction with scene three-dimensional point cloud model, manipulator's simulation model, planning path finally utilizes the path
The manipulator simulation model is driven to move；Dynamic manipulator simulation model is superimposed by the augmented reality display module (60)
In the coloured image that RGBD cameras (3) are shot, simulative display and path planning that actual situation combines are realized.
2. a kind of long-range monitoring and control system based on augmented reality according to claim 1, it is characterised in that：It is described
Collision detection is also completed under interactive simulation control model, specifically：The manipulator operating simulation module (50) is in scene three-dimensional
It is moved using the path drives manipulator simulation model in point cloud model, checks whether manipulator simulation model is three-dimensional with scene
Point cloud model interferes, and realizes actual situation collision detection；If there is interference, then path optimizing is sent out the path after optimization
Mobile robot is given, if there is no interference, the path is sent to mobile robot, mobile robot is controlled according to path
Operation task is completed in the movement of manipulator (2) processed.
3. a kind of long-range monitoring and control system based on augmented reality according to claim 1, it is characterised in that：It is described
Further include image capture controller (7), robot body controller (5) and Manipulator Controller (6) in mobile robot；Institute
It states image acquisition device control camera head (4) to move, and acquires the coloured image and depth image on the RGBD cameras (3),
The RGBD cameras (3) are fixed on camera head (4)；Robot body controller (5) the control robot body (1) is transported
It is dynamic, and feed back the orientation and posture of robot body (1)；The movement of Manipulator Controller (6) control machinery hand (2)；
It is distant that holder remote operating module (30), robot body remote operating module (10) and manipulator are additionally provided on the server
Operation module (20)；The holder remote operating module (30) sends to image capture controller (7) and instructs, and controls camera head
(4) it is moved according to instruction, to change the shooting angle of RGBD cameras (3)；The robot body remote operating module (10) to
Robot body controller (5) sends instruction, and control robot body (1) is moved according to instruction；The manipulator remote operating mould
Block (20) sends to Manipulator Controller (6) and instructs, and control machinery hand (2) is moved according to instruction；
The distal environment rebuilds module (40) when synthesizing scene three-dimensional point cloud model, passes through robot body remote operating first
Module (10) and manipulator remote operating module (20) are respectively to corresponding robot body controller (5) and manipulator control
Device (6) sends control instruction, so that robot body (1) and manipulator (2) is remained stationary, then passes through holder remote operating mould
Block (30) sends to image capture controller (7) and instructs, and controls camera head (4) according to instruction multi-angle rotation, and at each angle
Degree shooting depth image, to obtain the depth image of different angle, and synthesizes scene three-dimensional point cloud model.
4. a kind of long-range monitoring and control system based on augmented reality according to claim 3, it is characterised in that：It is described
The manipulator simulation model that manipulator operating simulation module (50) is established, including manipulator three dimensional virtual models and inverse kinematics mould
Type plans the end locus of points and posture of manipulator (2) in interactive simulation environment, obtains the end locus of points and posture sequence,
Using the end locus of points and posture sequence as the input of the inverse kinematics model, the angle sequence in manipulator (2) each joint is sought
Then row read the angle sequence in manipulator (2) each joint, each joint motions of driving manipulator three dimensional virtual models in order.
5. a kind of long-range monitoring and control system based on augmented reality according to claim 3, it is characterised in that：It is described
Augmented reality display module (60) will according to the current position of robot body (1), posture and camera head (4) azimuth information
Dynamic manipulator three dimensional virtual models are superimposed upon in the coloured image of RGBD cameras (3) shooting, are then shown, are realized empty
Real vision fusion.
6. a kind of long-range monitoring and control system based on augmented reality according to claim 5, it is characterised in that：The machine
The current position of device human body (1), posture send control instruction by robot body remote operating module (10) and give robot sheet
Body controller (5), obtains the information；Camera head (4) azimuth information is adopted by holder remote operating module (30) to image
Collect controller (7) and send instruction, obtains the azimuth information of camera head (4).
7. a kind of long-range monitoring and control system based on augmented reality according to claim 1, it is characterised in that：Further include
Monitoring pattern passes through remote data acquisition system in remote equipment and environment and the server and mobile machine
People, and complete the fusion of monitoring data and monitoring image in such a way that actual situation combines and show, specifically：
The remote data acquisition system acquires the monitoring data of each monitoring object, is then forwarded to server；Each monitoring object
Place is respectively provided with a label (300)；
When the mobile robot inspection, the control instruction from server, control robot body (1) and RGBD cameras are received
(3), so that robot body (1) is moved at monitoring object, RGBD cameras (3) is made to take the monitoring figure comprising label information
Picture, and it is sent to server；
The server further includes equipment identification module (70) and storage and fault diagnosis module (80)；
The equipment identification module (70) analyzes the monitoring image of RGBD cameras (3) shooting, and segmentation identifies the monitoring image
On label information, and send and include the inquiry request of label information；
The storage and fault diagnosis module (80) receive and preserve the monitoring data that the remote data acquisition system is sent, and
If diagnostic monitoring object sends warning message with the presence or absence of failure there are failure；When storage and fault diagnosis module (80) connect
When receiving the inquiry request from equipment identification module (70), the corresponding monitoring data of the label information are inquired according to label information,
It is then forwarded to the augmented reality display module (60)；
Monitoring data are superimposed upon corresponding in the monitoring image of RGBD cameras (3) shooting mark by the augmented reality display module (60)
Region is shown near label (300), realizes that the Overlapping display of monitoring data and monitoring data, display interface are more intuitive.
8. a kind of long-range monitoring and control system based on augmented reality according to claim 7, it is characterised in that：It is described remote
Journey data collecting system includes industrial personal computer (100) and sensor (200), and each sensor (200) accesses the industrial personal computer
(100), the sensor (200) is equipped at each monitoring object, each sensor (200) is correspondingly arranged a label
(300) 。
9. a kind of long-range monitoring and control system based on augmented reality according to claim 7, it is characterised in that：It is described to deposit
Storage and fault diagnosis module (80) are received and are protected according to the correspondence between label (300), monitoring object and monitoring data
Deposit the monitoring data that the remote data acquisition system is sent, and according to preset alarm threshold value, by monitoring data with it is corresponding
Alarm threshold value is compared, and if diagnostic monitoring object sends warning message with the presence or absence of failure there are failure；When storage and
When fault diagnosis module (80) receives the inquiry request from equipment identification module (70), label letter is inquired according to label information
Corresponding monitoring data are ceased, the augmented reality display module (60) is then forwarded to.
10. a kind of long-range monitoring and control system based on augmented reality according to claim 7, it is characterised in that：It is described
Monitoring image includes picture and video.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810236758.3A CN108422435B (en) | 2018-03-21 | 2018-03-21 | Remote monitoring and control system based on augmented reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810236758.3A CN108422435B (en) | 2018-03-21 | 2018-03-21 | Remote monitoring and control system based on augmented reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108422435A true CN108422435A (en) | 2018-08-21 |
| CN108422435B CN108422435B (en) | 2020-05-12 |
Family
ID=63159299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810236758.3A Active CN108422435B (en) | 2018-03-21 | 2018-03-21 | Remote monitoring and control system based on augmented reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108422435B (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109434808A (en) * | 2018-12-13 | 2019-03-08 | 上海菡为智能科技有限公司 | A kind of cloud remote service Study of Intelligent Robot Control network system realization |
| CN109483234A (en) * | 2018-11-02 | 2019-03-19 | 北京卫星制造厂有限公司 | A kind of intelligent manufacturing system and method based on mobile robot |
| CN110047150A (en) * | 2019-04-24 | 2019-07-23 | 大唐环境产业集团股份有限公司 | It is a kind of based on augmented reality complex device operation operate in bit emulator system |
| CN110238831A (en) * | 2019-07-23 | 2019-09-17 | 青岛理工大学 | Robot teaching system and method based on RGB-D image and teaching device |
| CN110434877A (en) * | 2019-08-14 | 2019-11-12 | 纳博特南京科技有限公司 | One kind being based on augmented reality and idiodynamic robot control method |
| WO2020073681A1 (en) * | 2018-10-10 | 2020-04-16 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| CN111459274A (en) * | 2020-03-30 | 2020-07-28 | 华南理工大学 | 5G + AR-based remote operation method for unstructured environment |
| CN111660294A (en) * | 2020-05-18 | 2020-09-15 | 北京科技大学 | Augmented reality control system of hydraulic heavy-duty mechanical arm |
| CN111754647A (en) * | 2020-05-21 | 2020-10-09 | 江苏锐士安防科技有限公司 | Intelligent security robot inspection method based on environment perception |
| US10803314B2 (en) | 2018-10-10 | 2020-10-13 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| CN111818115A (en) * | 2019-04-12 | 2020-10-23 | 华为技术有限公司 | Processing method, device and system |
| US10816994B2 (en) | 2018-10-10 | 2020-10-27 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| CN112549034A (en) * | 2020-12-21 | 2021-03-26 | 南方电网电力科技股份有限公司 | Robot task deployment method, system, equipment and storage medium |
| CN112819966A (en) * | 2021-01-05 | 2021-05-18 | 上海大学 | Environment fusion system and method suitable for man-machine interaction operation of underwater remote control robot |
| CN112839778A (en) * | 2018-10-12 | 2021-05-25 | 丰田研究所股份有限公司 | System and method for conditional robotic teleoperation |
| CN113100934A (en) * | 2021-04-06 | 2021-07-13 | 德智鸿(上海)机器人有限责任公司 | Operation assisting method, device, computer equipment and storage medium |
| CN113276110A (en) * | 2021-04-22 | 2021-08-20 | 国网浙江省电力有限公司嘉兴供电公司 | Transformer substation operation robot control system and method based on AR technology |
| CN113618731A (en) * | 2021-07-22 | 2021-11-09 | 中广核研究院有限公司 | Robot control system |
| CN114283139A (en) * | 2021-12-27 | 2022-04-05 | 常熟理工学院 | Weld joint detection and segmentation method and device based on area array structured light 3D vision |
| CN114434437A (en) * | 2020-10-30 | 2022-05-06 | 西门子（中国）有限公司 | Remote control method and device for robot |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102528811A (en) * | 2011-12-19 | 2012-07-04 | 上海交通大学 | Mechanical arm positioning and obstacle avoiding system in Tokamak cavity |
| CN205405613U (en) * | 2016-03-07 | 2016-07-27 | 广东理工学院 | Robot is rebuild to indoor three -dimensional scene of building |
| US20160349835A1 (en) * | 2015-05-28 | 2016-12-01 | Microsoft Technology Licensing, Llc | Autonomous drones for tactile feedback in immersive virtual reality |
| CN206561409U (en) * | 2017-01-20 | 2017-10-17 | 上海大学 | A kind of VR shoots kinematic robot system |
| CN107309882A (en) * | 2017-08-14 | 2017-11-03 | 青岛理工大学 | Robot teaching programming system and method |
| CN107578487A (en) * | 2017-09-19 | 2018-01-12 | 北京枭龙科技有限公司 | A kind of cruising inspection system based on augmented reality smart machine |
| CN107635133A (en) * | 2017-11-01 | 2018-01-26 | 广州供电局有限公司 | A kind of robot of data center inspection tour system based on augmented reality |
| CN107657682A (en) * | 2017-10-30 | 2018-02-02 | 成都极致空觉科技有限公司 | A kind of power transformation method for inspecting based on augmented reality |
| CN107765854A (en) * | 2017-10-20 | 2018-03-06 | 国网湖北省电力公司检修公司 | A kind of polling transmission line method based on augmented reality |
| CN207087855U (en) * | 2017-08-17 | 2018-03-13 | 苏州中德睿博智能科技有限公司 | Mobile robot platform for the modeling of coal mine roadway three-dimensional live |
-
2018
- 2018-03-21 CN CN201810236758.3A patent/CN108422435B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102528811A (en) * | 2011-12-19 | 2012-07-04 | 上海交通大学 | Mechanical arm positioning and obstacle avoiding system in Tokamak cavity |
| US20160349835A1 (en) * | 2015-05-28 | 2016-12-01 | Microsoft Technology Licensing, Llc | Autonomous drones for tactile feedback in immersive virtual reality |
| CN205405613U (en) * | 2016-03-07 | 2016-07-27 | 广东理工学院 | Robot is rebuild to indoor three -dimensional scene of building |
| CN206561409U (en) * | 2017-01-20 | 2017-10-17 | 上海大学 | A kind of VR shoots kinematic robot system |
| CN107309882A (en) * | 2017-08-14 | 2017-11-03 | 青岛理工大学 | Robot teaching programming system and method |
| CN207087855U (en) * | 2017-08-17 | 2018-03-13 | 苏州中德睿博智能科技有限公司 | Mobile robot platform for the modeling of coal mine roadway three-dimensional live |
| CN107578487A (en) * | 2017-09-19 | 2018-01-12 | 北京枭龙科技有限公司 | A kind of cruising inspection system based on augmented reality smart machine |
| CN107765854A (en) * | 2017-10-20 | 2018-03-06 | 国网湖北省电力公司检修公司 | A kind of polling transmission line method based on augmented reality |
| CN107657682A (en) * | 2017-10-30 | 2018-02-02 | 成都极致空觉科技有限公司 | A kind of power transformation method for inspecting based on augmented reality |
| CN107635133A (en) * | 2017-11-01 | 2018-01-26 | 广州供电局有限公司 | A kind of robot of data center inspection tour system based on augmented reality |
Cited By (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10803314B2 (en) | 2018-10-10 | 2020-10-13 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| CN112770875B (en) * | 2018-10-10 | 2022-03-11 | 美的集团股份有限公司 | Method and system for providing remote robot control |
| WO2020073681A1 (en) * | 2018-10-10 | 2020-04-16 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| US10678264B2 (en) | 2018-10-10 | 2020-06-09 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| US10816994B2 (en) | 2018-10-10 | 2020-10-27 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| CN112839778A (en) * | 2018-10-12 | 2021-05-25 | 丰田研究所股份有限公司 | System and method for conditional robotic teleoperation |
| CN109483234A (en) * | 2018-11-02 | 2019-03-19 | 北京卫星制造厂有限公司 | A kind of intelligent manufacturing system and method based on mobile robot |
| CN109434808A (en) * | 2018-12-13 | 2019-03-08 | 上海菡为智能科技有限公司 | A kind of cloud remote service Study of Intelligent Robot Control network system realization |
| CN111818115A (en) * | 2019-04-12 | 2020-10-23 | 华为技术有限公司 | Processing method, device and system |
| CN111818115B (en) * | 2019-04-12 | 2021-10-22 | 华为技术有限公司 | Processing method, device and system |
| CN110047150A (en) * | 2019-04-24 | 2019-07-23 | 大唐环境产业集团股份有限公司 | It is a kind of based on augmented reality complex device operation operate in bit emulator system |
| CN110047150B (en) * | 2019-04-24 | 2023-06-16 | 大唐环境产业集团股份有限公司 | Complex equipment operation on-site simulation system based on augmented reality |
| CN110238831A (en) * | 2019-07-23 | 2019-09-17 | 青岛理工大学 | Robot teaching system and method based on RGB-D image and teaching device |
| CN110434877A (en) * | 2019-08-14 | 2019-11-12 | 纳博特南京科技有限公司 | One kind being based on augmented reality and idiodynamic robot control method |
| CN111459274A (en) * | 2020-03-30 | 2020-07-28 | 华南理工大学 | 5G + AR-based remote operation method for unstructured environment |
| CN111459274B (en) * | 2020-03-30 | 2021-09-21 | 华南理工大学 | 5G + AR-based remote operation method for unstructured environment |
| CN111660294A (en) * | 2020-05-18 | 2020-09-15 | 北京科技大学 | Augmented reality control system of hydraulic heavy-duty mechanical arm |
| CN111754647A (en) * | 2020-05-21 | 2020-10-09 | 江苏锐士安防科技有限公司 | Intelligent security robot inspection method based on environment perception |
| CN114434437A (en) * | 2020-10-30 | 2022-05-06 | 西门子（中国）有限公司 | Remote control method and device for robot |
| CN112549034A (en) * | 2020-12-21 | 2021-03-26 | 南方电网电力科技股份有限公司 | Robot task deployment method, system, equipment and storage medium |
| WO2022135138A1 (en) * | 2020-12-21 | 2022-06-30 | 南方电网电力科技股份有限公司 | Robot task deployment method and system, device, and storage medium |
| CN112549034B (en) * | 2020-12-21 | 2021-09-03 | 南方电网电力科技股份有限公司 | Robot task deployment method, system, equipment and storage medium |
| CN112819966A (en) * | 2021-01-05 | 2021-05-18 | 上海大学 | Environment fusion system and method suitable for man-machine interaction operation of underwater remote control robot |
| CN113100934A (en) * | 2021-04-06 | 2021-07-13 | 德智鸿(上海)机器人有限责任公司 | Operation assisting method, device, computer equipment and storage medium |
| CN113276110A (en) * | 2021-04-22 | 2021-08-20 | 国网浙江省电力有限公司嘉兴供电公司 | Transformer substation operation robot control system and method based on AR technology |
| CN113618731A (en) * | 2021-07-22 | 2021-11-09 | 中广核研究院有限公司 | Robot control system |
| CN114283139A (en) * | 2021-12-27 | 2022-04-05 | 常熟理工学院 | Weld joint detection and segmentation method and device based on area array structured light 3D vision |
| CN114283139B (en) * | 2021-12-27 | 2024-08-20 | 常熟理工学院 | Weld joint detection segmentation method and device based on area array structured light 3D vision |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108422435B (en) | 2020-05-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108422435A (en) | Remote monitoring and control system based on augmented reality | |
| US7714895B2 (en) | Interactive and shared augmented reality system and method having local and remote access | |
| WO2020221311A1 (en) | Wearable device-based mobile robot control system and control method | |
| CN114728417B (en) | Method and apparatus for autonomous object learning by remote operator triggered robots | |
| CN105665970B (en) | For the path point automatic creation system and method for welding robot | |
| CN109164829B (en) | Flying mechanical arm system based on force feedback device and VR sensing and control method | |
| JP7337654B2 (en) | Maintenance activity support system and maintenance activity support method | |
| US20180190014A1 (en) | Collaborative multi sensor system for site exploitation | |
| KR101768958B1 (en) | Hybird motion capture system for manufacturing high quality contents | |
| CN112634318B (en) | Teleoperation system and method for underwater maintenance robot | |
| CN111716365B (en) | Immersive remote interaction system and method based on natural walking | |
| CN110977981A (en) | Robot virtual reality synchronization system and synchronization method | |
| CN107703956A (en) | A kind of virtual interaction system and its method of work based on inertia capturing technology | |
| CN102221884A (en) | Visual tele-existence device based on real-time calibration of camera and working method thereof | |
| CN111383348A (en) | Method for remotely and synchronously controlling robot through virtual reality | |
| CN114775724A (en) | Well control emergency rescue remote cooperative operation method based on digital twins | |
| CN102566552A (en) | Road tunnel intelligent overhaul robot facing Internet of things and 3D GIS | |
| CN116160440A (en) | Remote operation system of double-arm intelligent robot based on MR remote control | |
| CN116225213A (en) | Power grid element universe system and construction method | |
| CN210361314U (en) | Robot teaching device based on augmented reality technology | |
| CN113276110B (en) | Transformer substation operation robot control system and method based on AR technology | |
| CN110532905A (en) | A kind of self-loading-unloading mechanism operation Image-aided control method | |
| CN110445982A (en) | A kind of tracking image pickup method based on six degree of freedom equipment | |
| JPH0421105A (en) | Stereoscopic teaching device for manipulator | |
| Marin-Urias et al. | Towards shared attention through geometric reasoning for human robot interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200417 Address after: No. 63, Xingyu Road, Dongjiang street, Longkou City, Yantai City, Shandong Province Applicant after: Yantai Longwen Auto Parts Co.,Ltd. Address before: 266555 Jialing River Road 777, Qingdao economic and Technological Development Zone, Qingdao, Shandong Applicant before: Qingdao University of Technology |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Remote monitoring and control system based on augmented reality Effective date of registration: 20200618 Granted publication date: 20200512 Pledgee: Shandong Longkou Rural Commercial Bank Co.,Ltd. Pledgor: Yantai Longwen Auto Parts Co.,Ltd. Registration number: Y2020980003223 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20210520 Granted publication date: 20200512 Pledgee: Shandong Longkou Rural Commercial Bank Co.,Ltd. Pledgor: Yantai Longwen Auto Parts Co.,Ltd. Registration number: Y2020980003223 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A remote monitoring and control system based on Augmented Reality Effective date of registration: 20210531 Granted publication date: 20200512 Pledgee: Shandong Longkou Rural Commercial Bank Co.,Ltd. Pledgor: Yantai Longwen Auto Parts Co.,Ltd. Registration number: Y2021980004111 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20220624 Granted publication date: 20200512 Pledgee: Shandong Longkou Rural Commercial Bank Co.,Ltd. Pledgor: Yantai Longwen Auto Parts Co.,Ltd. Registration number: Y2021980004111 |