CN116679587A - Mixed reality teleoperation simulation system and method for tour inspection of extraterrestrial star meter - Google Patents
Mixed reality teleoperation simulation system and method for tour inspection of extraterrestrial star meter Download PDFInfo
- Publication number
- CN116679587A CN116679587A CN202310707178.9A CN202310707178A CN116679587A CN 116679587 A CN116679587 A CN 116679587A CN 202310707178 A CN202310707178 A CN 202310707178A CN 116679587 A CN116679587 A CN 116679587A
- Authority
- CN
- China
- Prior art keywords
- simulation
- extraterrestrial
- teleoperation
- module
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004088 simulation Methods 0.000 title claims abstract description 165
- 238000007689 inspection Methods 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000012937 correction Methods 0.000 claims abstract description 36
- 238000001514 detection method Methods 0.000 claims abstract description 35
- 238000012360 testing method Methods 0.000 claims abstract description 27
- 238000005259 measurement Methods 0.000 claims abstract description 24
- 238000012795 verification Methods 0.000 claims abstract description 21
- 238000010276 construction Methods 0.000 claims abstract description 8
- 238000004891 communication Methods 0.000 claims abstract description 7
- 230000003993 interaction Effects 0.000 claims description 31
- 238000012876 topography Methods 0.000 claims description 10
- 230000003190 augmentative effect Effects 0.000 claims description 5
- 238000006073 displacement reaction Methods 0.000 claims description 5
- 238000005286 illumination Methods 0.000 claims description 5
- 238000003384 imaging method Methods 0.000 claims description 4
- 238000009877 rendering Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 3
- 238000013461 design Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 230000003416 augmentation Effects 0.000 claims description 2
- 238000005457 optimization Methods 0.000 abstract description 3
- 230000005540 biological transmission Effects 0.000 description 4
- 230000033001 locomotion Effects 0.000 description 4
- 238000011022 operating instruction Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 238000011217 control strategy Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B17/00—Systems involving the use of models or simulators of said systems
- G05B17/02—Systems involving the use of models or simulators of said systems electric
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention discloses a mixed reality teleoperation simulation system and method for the inspection and detection of an extraterrestrial star, aiming at the ground verification requirement of a mixed reality quasi-real-time teleoperation mobile control scheme and algorithm under the communication time delay condition of the earth and the extraterrestrial star, by constructing an extraterrestrial star inspection and detection mixed reality teleoperation digital simulation system and an extraterrestrial star inspection and detection time delay teleoperation closed-loop semi-physical simulation system, virtual-real combination and relay verification of remote visible, closed-loop and quasi-real-time extraterrestrial star teleoperation processes, operators have stronger telepresence. Further, the test field environment and the inspection device state parameters obtained through external measurement are used for evaluating the calibration virtual scene construction and prediction correction algorithm, so that support is provided for verification and optimization of a teleoperation method, and teleoperation efficiency and reliability are improved.
Description
Technical Field
The invention relates to an extraterrestrial star-meter patrol detection mixed reality teleoperation simulation system and method, and belongs to the technical field of deep space detection simulation.
Background
The ground control personnel remotely operate the extraterrestrial star-meter mobile robot which is an important mode of extraterrestrial star-meter detection, a planning control strategy is generated by the ground and then is sent to an extraterrestrial detection site for execution, the ground control personnel can hardly effectively respond in real time according to the execution effect of the phase, but can only correct the execution process of the next phase according to the data obtained by the later telemetry, and the operation efficiency, the operation reliability and the operation safety of the mobile robot are all the problems to be solved urgently in the current extraterrestrial star remote operation detection task.
Disclosure of Invention
The technical solution of the invention is as follows: the method comprises the steps of establishing a digital simulation system and a semi-physical simulation system, verifying availability of a teleoperation mobile control scheme and algorithm design by virtual-real combination and adopting a relay verification method, and improving operation efficiency, reliability and safety of operation in a real environment.
The technical scheme of the invention is as follows:
the digital simulation system comprises an extraterrestrial star meter operation field digital simulation module, an extraterrestrial time delay prediction positioning module and a manual teleoperation simulation module;
the extraterrestrial star table operation field digital simulation module is used for simulating an extraterrestrial star table environment and comprises a virtual extraterrestrial star table landform three-dimensional environment, a virtual mobile robot positioned on the extraterrestrial star table and a virtual sensor;
the extraterrestrial time delay prediction positioning module is used for dynamically updating local scenes on the basis of an extraterrestrial star table environment based on a preset positioning and attitude determination algorithm, a prediction correction algorithm and a virtual scene construction algorithm according to the measurement data of the virtual sensor and an operation instruction sent by the remote control operation simulation module, outputting the positioning track and planning path information of the robot, and performing enhanced display in the remote control operation simulation module;
the remote control operation simulation module is operated through man-machine interaction equipment, analyzes the operation instruction to generate a virtual mobile robot wheel speed driving instruction, sends the virtual mobile robot wheel speed driving instruction to the extraterrestrial star meter operation field digital simulation module, and the virtual mobile robot moves on the virtual extraterrestrial star meter in real time according to the driving instruction.
Preferably, a virtual three-dimensional environment of the topography of the extraterrestrial star is created by using robot simulation software Gazebo.
Preferably, the extraterrestrial time delay prediction positioning module comprises a ground remote measurement and control simulation sub-module, an extraterrestrial time delay positioning and attitude determining sub-module, an extraterrestrial time delay prediction correction sub-module and a three-dimensional reconstruction and enhancement sub-module;
the ground remote measurement and remote control simulation sub-module analyzes the wheel speed direct operation instruction of the virtual mobile robot, generates a driving instruction, uploads the driving instruction to the extraterrestrial star meter operation field digital simulation module, and the virtual mobile robot moves according to the driving instruction;
the underground time delay positioning and attitude determination sub-module is used for determining the position and the attitude of the robot based on a preset positioning and attitude determination algorithm, simultaneously determining a reference standard, calibrating a sensor and planning an autonomous path, outputting a positioning and attitude determination result of the mobile robot and generating guide information of a planned path;
the external delay prediction correction sub-module analyzes the operation instruction to generate a virtual mobile robot direct operation instruction, and sends the virtual mobile robot direct operation instruction to the ground remote measurement and remote control simulation sub-module; meanwhile, based on a preset prediction correction algorithm, predicting the pose of the virtual mobile robot after the control of the driving instruction, and providing the pose for the three-dimensional reconstruction and enhancement submodule to perform scene matching and dynamic updating;
the three-dimensional reconstruction and enhancement submodule is used for carrying out multi-view feature recognition and matching based on the predicted pose information of the virtual mobile robot, the positioning track of the robot, the planning path information and the sensor measurement data, carrying out three-dimensional reconstruction and local dynamic update on the virtual scene based on a preset virtual scene construction algorithm, and generating enhancement information and guide information on the basis.
Preferably, the remote control operation simulation module comprises a mixed reality simulation sub-module and a man-machine interaction sub-module;
the mixed reality simulation submodule carries out virtual scene rendering and augmented information reality based on the extraterrestrial star table environment augmented information given by the three-dimensional reconstruction and augmentation submodule, and carries out mixed reality simulation by combining with the man-machine interaction submodule;
and the man-machine interaction submodule generates an operation instruction according to the displacement and rotation of the man-machine interaction equipment caused by external force.
Preferably, the minimized hardware of the mixed reality teleoperation digital simulation system for constructing the extraterrestrial star table patrol detection is configured into three workstations and a total control Ethernet connected with the workstations;
the extraterrestrial star table operation field digital simulation module and the three-dimensional reconstruction and enhancement submodule are operated on an algorithm and simulation environment workstation under a Linux operation system; the external delay prediction correction sub-module, the ground remote measurement and remote control simulation sub-module and the external delay positioning and attitude determination sub-module run on a positioning prediction algorithm workstation under a Linux operating system; the remote control operation simulation module operates on a graphic workstation under a Windows operating system.
The remote operation closed loop semi-physical simulation system for the inspection and detection time delay of the extraterrestrial star meter comprises an extraterrestrial star meter operation physical simulation site and a ground control simulation center which are positioned at different geographic positions;
the extraterrestrial star table operation physical simulation site is used for simulating an extraterrestrial star table patrol control task scene and comprises three subsystems, namely an extraterrestrial star table terrain physical simulation subsystem, a mobile robot subsystem and an unmanned aerial vehicle, wherein the mobile robot subsystem comprises a mobile platform and a mechanical arm; the exterior star meter topography physical simulation subsystem comprises a topography simulation test field and an illumination simulation module, wherein the topography simulation test field simulates flat areas, protrusions, pits and slopes of the exterior star meter, and the illumination simulation module simulates sunlight under different altitude angles; the unmanned aerial vehicle is used for shooting sequence images in a task scene;
the ground control simulation center comprises a prediction correction subsystem, a mixed reality simulation subsystem, a man-machine interaction subsystem and a test master control subsystem; a prediction correction subsystem operates a positioning and attitude determination algorithm and a prediction correction algorithm under the external time delay; the mixed reality simulation subsystem operates a task scene three-dimensional reconstruction, local scene update and enhancement information generation algorithm; the man-machine interaction subsystem comprises man-machine interaction equipment, a graphic display workstation and a display screen, and generates an operation instruction according to the displacement and rotation of the man-machine interaction equipment caused by external force; the test total control subsystem comprises a test total control computer, a wireless communication network, an Ethernet local area network, a data storage server and test monitoring equipment, wherein the wireless communication network is used for transmitting data and instructions of an extraterrestrial star table operation simulation site and a ground control simulation center, and the Ethernet local area network is used for transmitting data among all subsystems of the ground control simulation center.
Preferably, the mixed reality simulation subsystem performs three-dimensional reconstruction and local dynamic update on a virtual scene based on the predicted pose information of the mobile robot, the positioning track and planning path information of the robot and sensor measurement data, performs three-dimensional reconstruction and local dynamic update on the virtual scene based on a preset virtual scene construction algorithm, generates enhancement information and guide information on the basis, performs virtual scene rendering and enhancement information reality, and performs mixed reality simulation in combination with the man-machine interaction subsystem.
Preferably, the prediction correction subsystem is used for determining the position and the posture of the robot based on a preset positioning and posture determining algorithm, simultaneously performing reference standard determination, sensor calibration and autonomous path planning, outputting the positioning and posture determining result of the mobile robot and the planned path generation information, analyzing an operation instruction sent by the man-machine interaction subsystem to generate a direct operation instruction of the mobile robot, sending the direct operation instruction to the mobile robot, predicting the position and the posture of the mobile robot after the control of a driving instruction based on the preset prediction correction algorithm, and providing the position and the posture to the mixed reality simulation subsystem for scene matching and dynamic updating.
An extraterrestrial star-meter patrol detection mixed reality teleoperation simulation method comprises the following steps:
based on the extraterrestrial star meter inspection detection mixed reality teleoperation digital simulation system, teleoperation and algorithm digital simulation verification are executed, if the pose of a robot in the mixed reality sub-module is consistent with the state of a virtual mobile robot of an extraterrestrial star meter operation field digital simulation module, and the instruction execution is correct, the robot operation route is safe and accords with the expectation, the teleoperation behavior is confirmed to be safe and the algorithm is correct, and then based on the extraterrestrial star meter inspection detection time delay teleoperation closed-loop semi-physical simulation system, the same teleoperation and algorithm digital simulation verification is executed, and the correctness of the teleoperation and the algorithm is confirmed; and if the two teleoperation executions are safe and the algorithm is correct, the usability of the teleoperation mobile design and the algorithm is proved.
Preferably, the teleoperation and algorithm digital simulation verification is executed based on the extraterrestrial star meter patrol detection mixed reality teleoperation digital simulation system, and the method comprises the following steps:
creating an extraterrestrial star table operation field digital simulation module by using simulation software, and generating a virtual extraterrestrial star table environment, a virtual mobile robot positioned in the extraterrestrial star table and a virtual sensor;
simulating unmanned aerial vehicle flight to perform monocular camera sequence imaging on the virtual extraterrestrial star table environment based on the virtual extraterrestrial star table environment, and performing three-dimensional reconstruction on the virtual extraterrestrial star table environment;
the man-machine interaction sub-module displays the extraterrestrial star table operation site in real time by using graphic rendering software, and operators control man-machine interaction equipment which outputs an operation instruction to the extraterrestrial time delay prediction correction sub-module; the extraterrestrial time delay prediction correction submodule analyzes the operation instruction to generate a wheel speed direct operation instruction of the mobile robot, and predicts the position of the virtual robot after the virtual robot moves under the control of the instruction; the ground remote measurement and remote control simulation sub-module receives the wheel speed direct operation instruction of the virtual mobile robot, analyzes and generates a driving instruction, uploads the driving instruction to the extraterrestrial star meter operation field digital simulation module, and the virtual mobile robot moves according to the driving instruction.
Compared with the prior art, the invention has the advantages that:
(1) The invention establishes a mixed reality teleoperation digital simulation system for the tour inspection of the extraterrestrial star table, completes the measurement simulation of a virtual task scene, a virtual mobile robot and a virtual sensor through digital modeling, and realizes infinite simulation verification of the effectiveness of teleoperation reconstruction, positioning and predictive control in the extraterrestrial star table virtual environment.
(2) The invention establishes an extraterrestrial star meter inspection detection time delay teleoperation closed-loop semi-physical simulation system, realizes virtual-real parallelism of teleoperation task sites by constructing an extraterrestrial star meter environment and a mobile robot and reconstructing virtual task operation scenes, and meets the closed-loop test verification requirement of extraterrestrial time delay quasi-real-time teleoperation control performance.
(3) According to the invention, through the virtual-real combination and relay verification modes and combining the test field environment and the robot state parameters obtained by physical field external measurement, the performance indexes of processing algorithms such as ground scene modeling, motion prediction positioning and display can be evaluated, and meanwhile, calibration information and verification means can be further provided for the algorithm in a further optimization mode, so that the digital simulation modeling precision can be further improved, and a more realistic virtual verification environment is provided for digital simulation.
Drawings
FIG. 1 is a schematic diagram of a mixed reality teleoperation digital simulation system for tour inspection of an extraterrestrial star meter according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a teleoperation closed-loop semi-physical simulation system for detecting time delay for a tour of an extraterrestrial star table according to an embodiment of the present invention;
FIG. 3 is a flow chart of a mixed reality teleoperation simulation method according to an embodiment of the invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings:
aiming at the requirements of the detection task of the earth celestial bodies such as moon and other planets, the invention provides an extraterrestrial star-table patrol detection mixed reality teleoperation simulation system and method. By constructing the extraterrestrial star meter patrol detection mixed reality teleoperation digital simulation system and the extraterrestrial star meter patrol detection time delay teleoperation closed-loop semi-physical simulation system, virtual-real combination and relay verification of remote visual, closed-loop and quasi-real-time extraterrestrial star meter teleoperation processes, operators have stronger presence. Further, the test field environment and the inspection device state parameters obtained through external measurement are used for evaluating the calibration virtual scene construction and prediction correction algorithm, so that support is provided for verification and optimization of a teleoperation method, and teleoperation efficiency and reliability are improved.
Taking lunar surface detection task as an example, the technical scheme of the invention is described. As shown in FIG. 1, the extraterrestrial star-meter patrol detection mixed reality teleoperation digital simulation system is constructed, and the digital simulation of a physical scene and the parallel virtual scene of three-dimensional reconstruction are realized, so that the closed loop teleoperation control verification requirement in the digital simulation environment is met. The system comprises an extraterrestrial star meter operation field digital simulation module, an extraterrestrial time delay prediction positioning module and a manual teleoperation simulation module. The remote control operation simulation module comprises a man-machine interaction sub-module and a mixed reality simulation sub-module. The extraterrestrial time delay prediction positioning module comprises a ground remote measuring and control simulation sub-module, an extraterrestrial time delay positioning and attitude determining sub-module, an extraterrestrial time delay prediction correction sub-module and a three-dimensional reconstruction and enhancement sub-module.
The minimized hardware of the system is configured as a total control ethernet network between three workstations. The extraterrestrial star table operation field digital simulation module and the three-dimensional reconstruction and enhancement submodule are operated on an algorithm and simulation environment workstation under a Linux operation system; the underground time delay positioning and attitude determination sub-module, the underground time delay prediction correction sub-module and the ground remote control telemetry simulation sub-module run on a positioning prediction algorithm workstation under a Linux operating system; the remote control operation simulation module operates on a graphic workstation under a Windows operating system. The three workstations realize data exchange through the master control Ethernet.
The implementation way of the system comprises the following steps:
in an algorithm and simulation environment workstation under a Linux operation system, a extraterrestrial star table operation site digital simulation module is created by using commercial robot simulation software Gazebo, wherein the digital simulation module comprises a virtual lunar surface environment, a virtual mobile robot and virtual sensor measurement, and the virtual sensor comprises a monocular camera, a binocular camera, a laser radar and an inertial navigation sensor (IMU).
In an algorithm and simulation environment workstation under a Linux operation system, aiming at a virtual lunar surface environment under a Gazebo environment, simulating unmanned aerial vehicle flight to perform monocular camera sequence imaging on the virtual lunar surface environment, and completing three-dimensional reconstruction of a virtual scene based on sequence images.
In a positioning prediction algorithm workstation under a Linux operating system, aiming at the virtual sensor measurement data simulated in the algorithm and simulation environment workstation, the positioning pose determination of the virtual mobile robot and the prediction correction under the earth-month time delay are carried out.
In an algorithm and simulation environment workstation under a Linux operating system, according to simulated virtual sensor measurement data, simultaneously receiving a predicted position of the mobile robot obtained by earth-moon time delay prediction correction, and carrying out dynamic update and enhancement information generation of a local scene on the original three-dimensional reconstruction terrain through multi-view feature recognition and matching; meanwhile, the robot positioning track and the planned path output by the out-of-ground time delay positioning and attitude determination module can be used for generating guide information.
In a graphic display workstation under a Windows operating system, three-dimensional reconstructed and locally and dynamically updated virtual scenes and enhancement information are displayed in real time by using Unity graphic rendering software and are presented to ground operators. The control personnel directly face the screen or through a mixed reality (VR) helmet by utilizing man-machine interaction equipment, and realize the real-time movement of the virtual mobile robot on the virtual lunar surface topography through directly controlling a keyboard or a control handle under the rendered lunar surface task scene, and transmit the control instruction to the extraterrestrial time delay prediction correction module in real time.
In a positioning prediction algorithm workstation under a Linux operating system, an extraterrestrial time delay prediction correction sub-module analyzes operating instructions such as displacement, rotation angle and the like of an operating handle to generate direct operating instructions such as wheel speed and the like of a mobile robot, and the direct operating instructions are uploaded to a lunar surface operating field digital simulation module through a ground remote sensing and remote control simulation sub-module to simulate the quasi-real-time teleoperation mobile control of the mobile robot.
In an algorithm and simulation environment workstation under the Linux operating system environment, a teleoperation instruction is analyzed and uploaded to a digital lunar surface task scene under Gazebo, and quasi-real-time teleoperation mobile control of the mobile robot is simulated.
Based on the simulation environment and the workflow, the digital simulation verification of the extraterrestrial star-meter patrol detection mixed reality teleoperation flow and related algorithms is completed.
The method comprises the steps of constructing an extraterrestrial star meter patrol detection time delay teleoperation closed loop semi-physical simulation system, wherein the extraterrestrial star meter operation physical simulation sites and ground control simulation centers are positioned at different geographic positions, and the simulation center is shown in figure 2. The extraterrestrial star-watch operation physical simulation site is mainly used for simulating a lunar surface inspection control task scene and comprises a lunar surface terrain simulation subsystem, a mobile robot subsystem and an unmanned aerial vehicle subsystem. The ground control simulation center uses a real mobile robot, sensor equipment and a ground month transmission simulation link to replace a corresponding virtual simulation module in the digital simulation environment on the basis of the mixed reality teleoperation digital simulation environment, realizes the dynamic parallelism of a virtual scene and a physical scene, and achieves the purpose of real-time teleoperation mobile control ground verification through prediction and mixed reality technology. The simulation system implementation way comprises:
and establishing a mobile robot subsystem, wherein the hardware mainly comprises a mobile robot platform and a mechanical arm. The mobile robot platform comprises an eight-wheel mobile robot body, a driver, a controller, an upper computer (an Injeida industrial personal computer), various sensors (an IMU, a binocular camera, a laser radar) and the like; the mechanical arm part comprises a UR-5 mechanical arm body, a control cabinet, an upper computer (a grinding and warfare industrial personal computer), a hand-eye binocular camera, a paw and the like.
The lunar surface environment simulation subsystem is established and mainly comprises a lunar surface topography simulation test field and an illumination simulation system. The former includes simulated lunar surface flat areas, bumps such as stones, pits, slopes and other topographical features, and the latter simulates sunlight at different elevation angles.
And establishing a ground control simulation center, wherein the ground control simulation center comprises a prediction correction subsystem, a mixed reality simulation subsystem, a man-machine interaction subsystem and a test master control subsystem. The prediction and correction subsystem mainly operates a positioning and attitude determination algorithm and a position and attitude real-time prediction and correction algorithm under the time delay of earth and month; the mixed reality simulation subsystem mainly runs algorithms such as three-dimensional reconstruction of task scenes, local scene update, enhancement information generation and the like; the man-machine interaction subsystem comprises an operation handle (steering wheel), a virtual reality helmet and processing equipment, a graphic display workstation and a display screen; the test total control subsystem comprises a test total control computer, a wireless communication network (used for data and instruction transmission of a lunar surface operation simulation site and a ground control simulation center), an Ethernet local area network (used for data transmission among computers of the ground control simulation center), a data storage server and test monitoring equipment. The prediction correction subsystem, the mixed reality simulation subsystem and the man-machine interaction subsystem can replace the original lunar surface operation field digital simulation module with a physical test field and real mobile robot and sensor measurement information thereof based on an established digital simulation environment. The positioning and attitude determination algorithm can be operated on the ground control simulation center or an industrial personal computer of the mobile robot, and positioning and attitude determination information is transmitted back to the ground control simulation center through a wireless communication network.
And operating a physical test field on the lunar surface, utilizing a real unmanned aerial vehicle to fly along a designed nominal track, and carrying out sequential imaging on the simulated terrain of the lunar surface of the test field by a high-resolution monocular camera at the lower end of the unmanned aerial vehicle, wherein the simulated terrain is used as input data to verify the performance of a three-dimensional reconstruction algorithm of the lunar surface scene.
Based on a mixed reality teleoperation digital simulation environment, a steering wheel is utilized to control a virtual mobile robot to move in real time in a virtual scene, the instruction and data transmission between the physical test field mobile robot and a ground control center are realized through a wireless network, and teleoperation instructions are uploaded to the physical test field in a delayed mode, so that the quasi-real-time movement control of the real mobile robot is realized.
Based on the simulation system and the workflow, a mixed reality teleoperation simulation method is designed, and as shown in fig. 3, the semi-physical simulation verification of the extraterrestrial star-meter patrol detection mixed reality teleoperation flow and related algorithms is completed. Meanwhile, the environment of the ground test field and the state data of the mobile robot are acquired through external measuring equipment, and compared with related information in a virtual scene of a ground control center, so that performance indexes of processing algorithms such as ground scene modeling, motion prediction positioning and display can be evaluated, and meanwhile, calibration information and verification means can be further optimized and provided for the algorithms.
The foregoing is merely illustrative of the best embodiments of the present invention, and the present invention is not limited thereto, but any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be construed as falling within the scope of the present invention.
Claims (10)
1. The digital simulation system for the mixed reality teleoperation of the tour inspection of the extraterrestrial star meter is characterized by comprising a digital simulation module of an extraterrestrial star meter operation site, an extraterrestrial time delay prediction positioning module and a manual teleoperation simulation module;
the extraterrestrial star table operation field digital simulation module is used for simulating an extraterrestrial star table environment and comprises a virtual extraterrestrial star table landform three-dimensional environment, a virtual mobile robot positioned on the extraterrestrial star table and a virtual sensor;
the extraterrestrial time delay prediction positioning module is used for dynamically updating local scenes on the basis of an extraterrestrial star table environment based on a preset positioning and attitude determination algorithm, a prediction correction algorithm and a virtual scene construction algorithm according to the measurement data of the virtual sensor and an operation instruction sent by the remote control operation simulation module, outputting the positioning track and planning path information of the robot, and performing enhanced display in the remote control operation simulation module;
the remote control operation simulation module is operated through man-machine interaction equipment, analyzes the operation instruction to generate a virtual mobile robot wheel speed driving instruction, sends the virtual mobile robot wheel speed driving instruction to the extraterrestrial star meter operation field digital simulation module, and the virtual mobile robot moves on the virtual extraterrestrial star meter in real time according to the driving instruction.
2. The extraterrestrial star-based patrol exploration mixed reality teleoperation digital simulation system according to claim 1, wherein the robot simulation software Gazebo is utilized to create a virtual extraterrestrial star-based topography three-dimensional environment.
3. The extraterrestrial star-watch patrol detection mixed reality teleoperation digital simulation system according to claim 1, wherein the extraterrestrial time delay prediction positioning module comprises a ground remote measurement and control simulation sub-module, an extraterrestrial time delay positioning and attitude determination sub-module, an extraterrestrial time delay prediction correction sub-module and a three-dimensional reconstruction and enhancement sub-module;
the ground remote measurement and remote control simulation sub-module analyzes the wheel speed direct operation instruction of the virtual mobile robot, generates a driving instruction, uploads the driving instruction to the extraterrestrial star meter operation field digital simulation module, and the virtual mobile robot moves according to the driving instruction;
the underground time delay positioning and attitude determination sub-module is used for determining the position and the attitude of the robot based on a preset positioning and attitude determination algorithm, simultaneously determining a reference standard, calibrating a sensor and planning an autonomous path, outputting a positioning and attitude determination result of the mobile robot and generating guide information of a planned path;
the external delay prediction correction sub-module analyzes the operation instruction to generate a virtual mobile robot direct operation instruction, and sends the virtual mobile robot direct operation instruction to the ground remote measurement and remote control simulation sub-module; meanwhile, based on a preset prediction correction algorithm, predicting the pose of the virtual mobile robot after the control of the driving instruction, and providing the pose for the three-dimensional reconstruction and enhancement submodule to perform scene matching and dynamic updating;
the three-dimensional reconstruction and enhancement submodule is used for carrying out multi-view feature recognition and matching based on the predicted pose information of the virtual mobile robot, the positioning track of the robot, the planning path information and the sensor measurement data, carrying out three-dimensional reconstruction and local dynamic update on the virtual scene based on a preset virtual scene construction algorithm, and generating enhancement information and guide information on the basis.
4. A extraterrestrial star tour detection mixed reality teleoperation digital simulation system according to claim 3, wherein the human-controlled teleoperation simulation module comprises a mixed reality simulation sub-module and a human-computer interaction sub-module;
the mixed reality simulation submodule carries out virtual scene rendering and augmented information reality based on the extraterrestrial star table environment augmented information given by the three-dimensional reconstruction and augmentation submodule, and carries out mixed reality simulation by combining with the man-machine interaction submodule;
and the man-machine interaction submodule generates an operation instruction according to the displacement and rotation of the man-machine interaction equipment caused by external force.
5. The extraterrestrial star tour inspection and detection mixed reality teleoperation digital simulation system according to claim 4, wherein the minimized hardware of the extraterrestrial star tour inspection and detection mixed reality teleoperation digital simulation system is configured as a total control Ethernet of three work stations and a connection work station;
the extraterrestrial star table operation field digital simulation module and the three-dimensional reconstruction and enhancement submodule are operated on an algorithm and simulation environment workstation under a Linux operation system; the external delay prediction correction sub-module, the ground remote measurement and remote control simulation sub-module and the external delay positioning and attitude determination sub-module run on a positioning prediction algorithm workstation under a Linux operating system; the remote control operation simulation module operates on a graphic workstation under a Windows operating system.
6. The remote operation closed-loop semi-physical simulation system for the tour detection time delay of the extraterrestrial star meters is characterized by comprising an extraterrestrial star meter operation physical simulation site and a ground control simulation center which are positioned at different geographic positions;
the extraterrestrial star table operation physical simulation site is used for simulating an extraterrestrial star table patrol control task scene and comprises three subsystems, namely an extraterrestrial star table terrain physical simulation subsystem, a mobile robot subsystem and an unmanned aerial vehicle, wherein the mobile robot subsystem comprises a mobile platform and a mechanical arm; the exterior star meter topography physical simulation subsystem comprises a topography simulation test field and an illumination simulation module, wherein the topography simulation test field simulates flat areas, protrusions, pits and slopes of the exterior star meter, and the illumination simulation module simulates sunlight under different altitude angles; the unmanned aerial vehicle is used for shooting sequence images in a task scene;
the ground control simulation center comprises a prediction correction subsystem, a mixed reality simulation subsystem, a man-machine interaction subsystem and a test master control subsystem; a prediction correction subsystem operates a positioning and attitude determination algorithm and a prediction correction algorithm under the external time delay; the mixed reality simulation subsystem operates a task scene three-dimensional reconstruction, local scene update and enhancement information generation algorithm; the man-machine interaction subsystem comprises man-machine interaction equipment, a graphic display workstation and a display screen, and generates an operation instruction according to the displacement and rotation of the man-machine interaction equipment caused by external force; the test total control subsystem comprises a test total control computer, a wireless communication network, an Ethernet local area network, a data storage server and test monitoring equipment, wherein the wireless communication network is used for transmitting data and instructions of an extraterrestrial star table operation simulation site and a ground control simulation center, and the Ethernet local area network is used for transmitting data among all subsystems of the ground control simulation center.
7. The extraterrestrial star-based patrol detection time delay teleoperation closed-loop semi-physical simulation system according to claim 6, wherein the mixed reality simulation subsystem is based on predicted pose information of a mobile robot, positioning track and planning path information of the robot and sensor measurement data, performs multi-view feature recognition and matching based on a preset virtual scene construction algorithm, performs three-dimensional reconstruction and local dynamic update on a virtual scene, generates enhancement information and guide information on the basis, performs virtual scene rendering and augmented information reality, and performs mixed reality simulation in combination with a man-machine interaction subsystem.
8. The extraterrestrial star-based patrol detection time delay teleoperation closed-loop semi-physical simulation system is characterized in that the prediction correction subsystem is used for determining the position and the posture of a robot based on a preset positioning and posture-fixing algorithm, simultaneously performing reference standard determination, sensor calibration and autonomous path planning, outputting a mobile robot positioning and posture-fixing result and planning path generation information, analyzing an operation instruction sent by the man-machine interaction subsystem to generate a mobile robot direct operation instruction, sending the mobile robot direct operation instruction to the mobile robot, predicting the position and the posture of the mobile robot after control of a driving instruction based on the preset prediction correction algorithm, and providing the mobile robot direct operation instruction to the mixed reality simulation subsystem for scene matching and dynamic updating.
9. The method for simulating the mixed reality teleoperation of the tour inspection of the extraterrestrial star table is characterized by comprising the following steps of:
based on the extraterrestrial star meter inspection detection mixed reality teleoperation digital simulation system, teleoperation and algorithm digital simulation verification are executed, if the pose of a robot in the mixed reality sub-module is consistent with the state of a virtual mobile robot of an extraterrestrial star meter operation field digital simulation module, and the instruction execution is correct, the robot operation route is safe and accords with the expectation, the teleoperation behavior is confirmed to be safe and the algorithm is correct, and then based on the extraterrestrial star meter inspection detection time delay teleoperation closed-loop semi-physical simulation system, the same teleoperation and algorithm digital simulation verification is executed, and the correctness of the teleoperation and the algorithm is confirmed; and if the two teleoperation executions are safe and the algorithm is correct, the usability of the teleoperation mobile design and the algorithm is proved.
10. The method for mixed reality teleoperation simulation of outdoor star-based tour-inspection and detection according to claim 9, wherein the step of performing teleoperation and algorithm digital simulation verification based on the outdoor star-based tour-inspection and detection mixed reality teleoperation digital simulation system comprises the steps of:
creating an extraterrestrial star table operation field digital simulation module by using simulation software, and generating a virtual extraterrestrial star table environment, a virtual mobile robot positioned in the extraterrestrial star table and a virtual sensor;
simulating unmanned aerial vehicle flight to perform monocular camera sequence imaging on the virtual extraterrestrial star table environment based on the virtual extraterrestrial star table environment, and performing three-dimensional reconstruction on the virtual extraterrestrial star table environment;
the man-machine interaction sub-module displays the extraterrestrial star table operation site in real time by using graphic rendering software, and operators control man-machine interaction equipment which outputs an operation instruction to the extraterrestrial time delay prediction correction sub-module; the extraterrestrial time delay prediction correction submodule analyzes the operation instruction to generate a wheel speed direct operation instruction of the mobile robot, and predicts the position of the virtual robot after the virtual robot moves under the control of the instruction; the ground remote measurement and remote control simulation sub-module receives the wheel speed direct operation instruction of the virtual mobile robot, analyzes and generates a driving instruction, uploads the driving instruction to the extraterrestrial star meter operation field digital simulation module, and the virtual mobile robot moves according to the driving instruction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310707178.9A CN116679587A (en) | 2023-06-14 | 2023-06-14 | Mixed reality teleoperation simulation system and method for tour inspection of extraterrestrial star meter |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310707178.9A CN116679587A (en) | 2023-06-14 | 2023-06-14 | Mixed reality teleoperation simulation system and method for tour inspection of extraterrestrial star meter |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116679587A true CN116679587A (en) | 2023-09-01 |
Family
ID=87778928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310707178.9A Pending CN116679587A (en) | 2023-06-14 | 2023-06-14 | Mixed reality teleoperation simulation system and method for tour inspection of extraterrestrial star meter |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116679587A (en) |
-
2023
- 2023-06-14 CN CN202310707178.9A patent/CN116679587A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7082416B2 (en) | Real-time 3D that expresses the real world Two-way real-time 3D interactive operation of real-time 3D virtual objects in a virtual world | |
| Nguyen et al. | Virtual reality interfaces for visualization and control of remote vehicles | |
| US8190295B1 (en) | Apparatus and method for modifying the operation of a robotic vehicle in a real environment, to emulate the operation of the robotic vehicle operating in a mixed reality environment | |
| Allan et al. | Planetary rover simulation for lunar exploration missions | |
| CN101630146B (en) | Simulation control system for remote tele-operation of lunar rover | |
| CN111580519B (en) | Quasi-real-time teleoperation system for lunar surface detection under earth-moon time delay | |
| CN110321000B (en) | Virtual simulation system for complex tasks of intelligent system | |
| CN113703462B (en) | Unknown space autonomous exploration system based on quadruped robot | |
| Wright et al. | Driving on Mars with RSVP | |
| CN114488848A (en) | Unmanned aerial vehicle autonomous flight system and simulation experiment platform for indoor building space | |
| Rossmann | erobotics: The symbiosis of advanced robotics and virtual reality technologies | |
| Volpe | Rover technology development and mission infusion beyond MER | |
| Kryuchkov et al. | Simulation of the «cosmonaut-robot» system interaction on the lunar surface based on methods of machine vision and computer graphics | |
| Weclewski et al. | Sample Fetch Rover guidance, navigation and control subsystem-an overview | |
| Ramos et al. | Internet-based solutions in the development and operation of an unmanned robotic airship | |
| CN116679587A (en) | Mixed reality teleoperation simulation system and method for tour inspection of extraterrestrial star meter | |
| Sewtz et al. | Ursim-a versatile robot simulator for extra-terrestrial exploration | |
| Correal et al. | Autonomy for ground-level robotic space exploration: framework, simulation, architecture, algorithms and experiments | |
| Bualat et al. | Robotic recon for human exploration: Method, assessment, and lessons learned | |
| Hartman et al. | Data visualization for effective rover sequencing | |
| Correal et al. | Onboard Autonomous Navigation Architecture for a Planetary Surface Exploration Rover and Functional Validation Using Open-Source Tools | |
| Koch et al. | Helicopter and Rover Operations on Mars Using the Robot Sequencing and Visualization Program (RSVP) | |
| Weller et al. | Vr-interactions for planning planetary swarm exploration missions in vamex-vtb | |
| Zheng et al. | Virtual Prototyping-Based Path Planning of Unmanned Aerial Vehicles for Building Exterior Inspection | |
| Teuber et al. | Vamex-vtb-a modular virtual testbed for multimodal autonomous planetary missions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |