CN116362109B - Intelligent unmanned system and method based on digital twinning - Google Patents
Intelligent unmanned system and method based on digital twinning Download PDFInfo
- Publication number
- CN116362109B CN116362109B CN202310147580.6A CN202310147580A CN116362109B CN 116362109 B CN116362109 B CN 116362109B CN 202310147580 A CN202310147580 A CN 202310147580A CN 116362109 B CN116362109 B CN 116362109B
- Authority
- CN
- China
- Prior art keywords
- twin
- data
- module
- model
- intelligent
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000004088 simulation Methods 0.000 claims abstract description 63
- 230000009471 action Effects 0.000 claims abstract description 40
- 230000003993 interaction Effects 0.000 claims abstract description 36
- 238000011156 evaluation Methods 0.000 claims abstract description 34
- 238000011217 control strategy Methods 0.000 claims abstract description 30
- 238000013136 deep learning model Methods 0.000 claims description 60
- 238000004422 calculation algorithm Methods 0.000 claims description 23
- 230000033001 locomotion Effects 0.000 claims description 21
- 238000009826 distribution Methods 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 230000004927 fusion Effects 0.000 claims description 10
- 238000005516 engineering process Methods 0.000 description 13
- 238000004590 computer program Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 12
- 238000004458 analytical method Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 6
- 238000004821 distillation Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000007613 environmental effect Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000003860 storage Methods 0.000 description 4
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000013140 knowledge distillation Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000003340 mental effect Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013499 data model Methods 0.000 description 2
- 238000013209 evaluation strategy Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000037081 physical activity Effects 0.000 description 1
- 230000000704 physical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Geometry (AREA)
- Medical Informatics (AREA)
- Computer Hardware Design (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides an intelligent unmanned system and a method based on digital twinning, wherein the system is deployed on target unmanned equipment and comprises the following steps: an interaction layer for acquiring physical state data; the digital intelligence twin layer comprises a physical twin model and an intelligent twin model; the physical twin model is used for modeling the target unmanned equipment according to the physical state data to obtain a physical twin body; the intelligent twin model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies; the simulation layer comprises an environment simulation module, an evaluation module and a matching module; the environment simulation module is used for obtaining the unmanned equipment operation environment through simulation; the matching module is used for respectively matching the unmanned equipment operation environments with all the intelligent twin bodies to obtain a plurality of comprehensive twin bodies; the evaluation module is used for evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body.
Description
Technical Field
The invention relates to the technical field of digital twinning, in particular to an intelligent unmanned system and method based on digital twinning.
Background
In recent years, along with the continuous enhancement of social intelligence, unmanned equipment control systems appear in blowout, are widely applied to various fields such as medical treatment, catering, logistics and the like, and provide great convenience for life of people. Meanwhile, limited computing and storage capacity of the unmanned equipment cannot meet the requirements of a resource-intensive artificial intelligent algorithm, and the unmanned equipment has low-efficiency development iteration flow, high maintenance and repair cost and cooperative complexity of unmanned equipment with different models, so that the unmanned equipment becomes a bottleneck for restricting intelligent development of the unmanned equipment.
Most of the existing schemes are to control and instruct an unmanned equipment control system manually in a real environment, so that the problem of unmanned equipment in a corresponding scene is solved, the labor and the effort are wasted, and unnecessary cost is easily caused. In addition, the existing technical scheme related to unmanned equipment control is mostly realized based on single-class data (such as video data or image data) of the unmanned equipment, and fusion processing on multi-source heterogeneous data is lacked, so that information analysis on the entity of the unmanned equipment is not comprehensive and accurate enough. In addition, as the number of unmanned devices increases, the difficulty of collaborative work increases in geometric progression. Because unmanned equipment is large in energy and calculation amount, the traditional algorithm cannot provide an optimal strategy to realize intelligent collaborative operation under a plurality of unmanned equipment. Accordingly, it is desirable to provide a digital twinning-based intelligent unmanned system and method for implementing intelligent control of unmanned devices.
Disclosure of Invention
In view of the above, embodiments of the present invention provide a digital twinning-based intelligent unmanned system and method to overcome or at least partially solve the above-described problems.
A first aspect of an embodiment of the present invention provides an intelligent unmanned system based on digital twinning, deployed on a target unmanned device, the system comprising:
an interaction layer, configured to obtain physical state data of the target unmanned device;
the digital intelligence twin layer comprises a physical twin model and an intelligent twin model;
the physical twin model is used for modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, and the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment;
the intelligent twin model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies, wherein each intelligent twin body is a digital model for representing the physical twin bodies to act according to one action strategy;
the simulation layer comprises an environment simulation module, an evaluation module and a matching module;
the environment simulation module is used for obtaining the unmanned equipment operation environment through simulation;
The matching module is used for respectively matching the unmanned equipment operation environments with each intelligent twin to obtain a plurality of comprehensive twin, wherein the comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environments;
the evaluation module is used for evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body.
Optionally, the interaction layer further includes a control module;
the evaluation module is further configured to send the optimal control policy to a control module of the interaction layer;
the control module is used for controlling the target unmanned equipment according to the optimal control strategy.
Optionally, the interaction layer includes a data conversion module and a data acquisition module;
the data acquisition module is used for acquiring original state data of the target unmanned equipment, wherein the original state data is heterogeneous data transmitted by a plurality of sensors deployed on the target unmanned equipment, and the heterogeneous data comprises one or more of infrared data, radar data, video data, image data, text data and position data;
The data conversion module is used for converting the original state data into the physical state data, wherein the physical state data is the data which can be identified by the physical twin model.
Optionally, the data acquisition module is further configured to acquire real environment data of the target unmanned device;
the environment simulation module further comprises a similarity calculation unit; the similarity calculation unit is used for calculating semantic similarity between the real environment data and a plurality of prestored simulation environments respectively, and determining the simulation environment with the highest similarity as the unmanned equipment operation environment.
Optionally, the evaluation module evaluates the multiple comprehensive twin bodies respectively through the following formula to obtain an optimal twin body;
Argmi n(cost(f2-f1),cost(S(cpu、battery、memory，env)))
the cost (f 2, f 1) represents the cost of changing the target unmanned equipment from the current state to the next state, represents the cost required by executing the strategy corresponding to the comprehensive twin, f2 represents the next state of the target unmanned equipment, and f1 represents the current state of the target unmanned equipment; cost (S (cpu, battery, memory, env)) represents the cost of the target drone itself attribute and the cost of the operating environment.
Optionally, the intelligent twin model comprises an algorithm selection module and a plurality of deep learning models, and each deep learning model is a multi-modal fusion deep learning model;
the algorithm selection module is used for determining a proper target deep learning model from the multiple deep learning models based on a meta learning method according to the unmanned equipment operation environment and the physical state data;
the target deep learning model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies.
Optionally, each of the deep learning models is a deep learning model obtained by distilling BEV in advance.
Optionally, the digital twin layer further comprises a data distribution service module, and data interaction is performed between each model/module in the digital twin layer through the data distribution service module.
Optionally, each model/module in the digital intelligent twin layer is used as a micro-service module to form a whole micro-service framework system, and each micro-service module obtains the required data of the micro-service module deployed on other unmanned equipment or platforms through the RPC.
Optionally, the physical twin model further comprises a data display module and a model data synchronization module;
The data display module is used for displaying the acquired physical state data of the target unmanned equipment;
the model data synchronization module is used for acquiring model parameter data deployed on other unmanned equipment or platforms by utilizing the RPC, and updating a corresponding model according to the model parameter data.
Optionally, the intelligent twin model includes a report generation module;
the report generation module is used for generating an evaluation report of the digital intelligent twin body and an evaluation report of each deep learning model.
A second aspect of the present embodiment provides a method for generating a control policy of an unmanned device based on digital twinning, where the method includes:
acquiring physical state data of the target unmanned equipment;
modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, wherein the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment;
obtaining a plurality of digital twins according to the physical twins and a plurality of action strategies, wherein each digital twins is a digital model for representing the action of the physical twins according to one action strategy;
Simulating to obtain an unmanned equipment operation environment;
respectively matching the unmanned equipment operation environments with each intelligent twin to obtain a plurality of comprehensive twin, wherein the comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environments;
and evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body.
The third aspect of the embodiment of the invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory, wherein the processor executes the computer program to realize the steps in the unmanned device control strategy generation method based on digital twinning according to the second aspect of the embodiment of the invention.
The fourth aspect of the embodiment of the present invention further provides a computer readable storage medium, on which a computer program/instruction is stored, which when executed by a processor, implements the steps in the method for generating a digital twin-based unmanned device control policy according to the second aspect of the embodiment of the present invention.
The embodiment of the invention provides an intelligent unmanned system based on digital twinning, which is deployed on target unmanned equipment, and comprises the following components: an interaction layer, configured to obtain physical state data of the target unmanned device; the digital intelligence twin layer comprises a physical twin model and an intelligent twin model; the physical twin model is used for modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, and the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment; the intelligent twin model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies, wherein each intelligent twin body is a digital model for representing the physical twin bodies to act according to one action strategy; the simulation layer comprises an environment simulation module, an evaluation module and a matching module; the environment simulation module is used for obtaining the unmanned equipment operation environment through simulation; the matching module is used for respectively matching the unmanned equipment operation environments with each intelligent twin to obtain a plurality of comprehensive twin, wherein the comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environments; the evaluation module is used for evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body.
The concrete beneficial effects are that:
1) The unmanned equipment is endowed with higher intelligence. According to the embodiment of the invention, the physical state data is acquired through the interaction layer, then the digital intelligent twin layer is utilized, virtual modeling of the unmanned equipment is realized through the physical twin model and the intelligent twin model based on the acquired multidimensional data, and the digital intelligent twin is generated, so that the unmanned equipment is endowed with higher intelligence, and more complex and difficult operations such as target tracking, automatic patrol and the like can be executed by the unmanned equipment.
2) And carrying out simulation on the real environment. According to the embodiment of the invention, the simulation environment is combined with the digital intelligent twin body through the simulation environment, the optimal comprehensive twin body is evaluated and produced according to the set strategy, the optimal control strategy for the target unmanned equipment is determined, the influence of the environment of the unmanned equipment is fully considered, and the determined control strategy is more intelligent and reliable.
3) The technical effect of virtual control is achieved. According to the embodiment of the invention, the high-fidelity digital twin model of the corresponding physical entity (the target unmanned equipment) is constructed, and the generated twin is evaluated to obtain the optimal control strategy, so that the target unmanned equipment can be controlled according to the control strategy, the interactive co-fusion and intelligent operation of the physical entity space and the digital virtual space is realized, and the technical effect of virtual control is achieved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a digital twinning-based intelligent unmanned system according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a functional setting of an intelligent unmanned system according to an embodiment of the present invention;
fig. 3 is a step flowchart of a method for generating a control policy of an unmanned device according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings in the embodiments of the present invention. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
First, related terms according to the embodiments of the present invention will be briefly explained.
Digital twinning is a virtual representation that models the state of a physical entity or system. Data is collected from physical entities through sensors and digitized to make predictions of them, providing a low cost, high efficiency and high quality solution for the target area. The digital twin technology is a multidisciplinary integration technology based on big data, artificial intelligence, simulation modeling and the like, a digital method is adopted to establish a virtual model representing a physical entity, the physical activity of the physical entity is simulated through simulation analysis, and the technical means of virtual-real interaction feedback, data fusion analysis, decision process iteration optimization and the like are comprehensively utilized to realize interaction fusion and intelligent control from the physical entity to the virtual digital model.
The bird's eye view distillation method (Bird's Eye Vi ew Di st i l l, BEV Di st il), hereinafter collectively referred to as BEV distillation, is a trans-modal knowledge distillation method, mainly uses the characteristics of a uniform image and a laser radar in a Bird's Eye View (BEV) space, and adaptively transfers knowledge across heterogeneous characterizations in a master-slave paradigm. In particular, converting all features into BEV space, both geometric and semantic information, features from different modalities can be naturally aligned without losing much information through shared BEV representations. On the basis, the space knowledge is adaptively transferred through two modes of dense and sparse supervision. Thereby realizing knowledge distillation of the deep learning model and compressing the model into a smaller model.
The data distribution service (Data Di str i but i on Servi ce, DDS), hereinafter collectively referred to as DDS, is a new generation of distributed real-time communication middleware technical specification formulated by an Object Management Group (OMG) on the basis of standards such as HLA and CORBA, and the DDS adopts a publish/subscribe architecture, emphasizes on centering on data, provides a rich QoS quality of service policy, can ensure real-time, efficient and flexible distribution of the data, and can meet various distributed real-time communication application requirements. The DDS information distribution middleware is a lightweight middleware technology capable of providing real-time information transmission.
Remote procedure call protocol (Remote Procedure Ca l l, RPC), hereinafter referred to collectively as RPC, is a protocol that requests services from a remote computer program over a network without requiring knowledge of underlying network technology. The RPC protocol assumes the existence of certain transport protocols, such as TCP or UDP, to carry information data between communication programs.
The following is a specific content of the embodiment of the present invention.
The embodiment of the invention provides an intelligent unmanned system based on digital twinning, which is deployed on target unmanned equipment, and referring to fig. 1, fig. 1 shows a schematic structural diagram of the intelligent unmanned system based on digital twinning, as shown in fig. 1, the system comprises:
And the interaction layer is used for acquiring the physical state data of the target unmanned equipment.
In this embodiment, the unmanned device refers to a device that can realize remote control by a set policy or a remotely transmitted control instruction, such as an unmanned plane or an unmanned vehicle. The physical state data is real state data of the target unmanned equipment at the current moment, and specifically may include various data such as a movement speed, a movement position, a movement direction, a battery state, a CPU state and the like. The interaction layer is used for carrying out data interaction with the target unmanned equipment, and the connection between the physical entity space of the target unmanned equipment and the digital virtual space of the intelligent unmanned system is realized, so that physical state data are acquired from the target unmanned equipment in real time.
The digital intelligence twin layer comprises a physical twin model and an intelligent twin model.
The physical twin model is used for modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, and the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment;
the intelligent twin model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies, wherein each intelligent twin body is a digital model for representing the physical twin bodies to act according to one action strategy;
In the embodiment, the digital intelligent twin layer performs modeling and analysis based on the acquired multidimensional data, builds a corresponding high-fidelity digital twin model of the physical entity, and can realize rapid simulation of large-scale unmanned equipment nodes by adopting a simplified motion and control model under the condition of ensuring the basic simulation precision requirement. Meanwhile, based on an artificial intelligence technology, a digital intelligent twin body is generated through means of data fusion analysis, virtual-real interaction feedback, decision iterative optimization and the like, and interaction co-fusion intelligent operation of a physical entity space and a digital virtual space is realized.
Specifically, the digital intelligent twin layer can be divided into a physical twin model and an intelligent twin model. The physical twin model is used for generating a physical twin body according to the acquired physical state data. The physical twin body is a digital model for representing the real motion state of the target unmanned aerial vehicle at the current moment, namely the physical twin body is a simulation of the target unmanned aerial vehicle, and basic physical properties of the target unmanned aerial vehicle, such as corresponding speed, position, battery state and the like, can be obtained through analysis according to the physical twin body model. The target unmanned aerial vehicle is an unmanned aerial vehicle, and the physical twin model models according to various physical state data of the unmanned aerial vehicle, so that a virtual unmanned aerial vehicle digital model is obtained through simulation and is used as a corresponding physical twin body. If the speed of the unmanned aerial vehicle at the current moment is 5 meters per second, the corresponding physical attribute of the speed in the physical twin is 5 meters per second. When the speed of the unmanned aerial vehicle changes, the speed attribute of the physical twin body also changes.
In this embodiment, the digital twinning layer further comprises a smart twinning model. The intelligent twin model is used for generating a plurality of corresponding digital twin bodies according to the physical twin bodies and different action strategies, and each digital twin body is a digital model representing the physical twin bodies to act according to one action strategy. The action strategy can be a simple next action of the target unmanned equipment, a guiding strategy of a next flight track, namely, a digital model of the next action or action track of the target unmanned equipment which is simulated or predicted according to the action strategy, for example, more complex and intelligent actions or operations of the target unmanned equipment for executing target tracking, automatic patrol, target search and rescue and the like can be simulated. As a plurality of action strategies exist, a plurality of generated intelligent twin bodies exist, and each intelligent twin body corresponds to one action strategy. With the increase of the number of unmanned devices, the difficulty of collaborative operation is increased in geometric progression, and because the energy and calculation amount of the unmanned devices are large, the traditional algorithm cannot provide an optimal strategy, the embodiment provides that the digital twin technology is utilized, virtual modeling of the unmanned devices is realized through a physical twin model and an intelligent twin model, and a digital intelligent twin body is generated, so that the unmanned devices are endowed with higher intelligence, and the unmanned devices realize functions such as target tracking, automatic patrol, target search and rescue and the like which cannot be realized originally.
The simulation layer comprises an environment simulation module, an evaluation module and a matching module;
the environment simulation module is used for obtaining the unmanned equipment operation environment through simulation;
the matching module is used for respectively matching the unmanned equipment operation environments with each intelligent twin to obtain a plurality of comprehensive twin, wherein the comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environments;
the evaluation module is used for evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body.
In this embodiment, the environment simulation module performs environment simulation by using a simulation engine, builds operation scenes of different twin bodies, and obtains an unmanned equipment operation environment, where the unmanned equipment operation environment is obtained by simulating a real environment where the target unmanned equipment is located. The matching module respectively matches the unmanned equipment operation environments with all the intelligent twin bodies to obtain a plurality of comprehensive twin bodies. Specifically, the digital intelligent twin is matched with the simulated virtual environment, and the comprehensive twin is obtained. The comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environment. The intelligent twin body is a data model for simulating the unmanned aerial vehicle A to execute the target tracking task, the unmanned equipment operation environment is a forest environment, the matching module matches the intelligent twin body and the unmanned aerial vehicle A to execute the target tracking data model in the forest environment. And finally, evaluating the obtained multiple comprehensive twins by utilizing an evaluation module, selecting an optimal comprehensive twins from the multiple comprehensive twins, and determining an action strategy corresponding to the comprehensive twins as an optimal control strategy, wherein the optimal control strategy represents the most suitable strategy of the target unmanned equipment in the multiple action strategies, or namely the strategy with the minimum action cost. Therefore, the simulation layer in the embodiment combines the simulation environment with the digital intelligent twin through the simulation environment, evaluates the comprehensive twin with the optimal output according to the set strategy, and determines the optimal control strategy for the target unmanned equipment.
For unmanned equipment, as the application of the unmanned equipment is mature gradually, the limited computing and storage capacity of the unmanned equipment cannot meet the requirements of a resource-intensive artificial intelligent algorithm, and the unmanned equipment has low-efficiency development iteration flow, high maintenance and repair cost and cooperative complexity of unmanned equipment with different models, so that the unmanned equipment becomes a bottleneck for restricting the intelligent development of the unmanned equipment. With the rapid development of digital twin technology in recent years, the digital twin technology becomes a research hot spot in various fields, and is actively and widely applied in various fields from aircraft engines to spacecrafts. Based on the above, the embodiment proposes that the digital twin technology is applied to a complex intelligent unmanned system to control the unmanned equipment, so that the unmanned equipment can realize a more intelligent function. The embodiment is based on a digital twin technology, an intelligent unmanned system is built, and physical state data of unmanned equipment are obtained through an interaction layer; virtual modeling is carried out through a digital twin technology of a digital twin layer and an intelligent collaboration framework, and the motion state of unmanned equipment in each action strategy is simulated and predicted to obtain a digital twin body, so that resource-intensive operation is effectively converted into a virtual model; the simulation layer simulates a real scene, based on machine learning and artificial intelligence technology, training and learning are carried out on a complex real scene in a virtual scene, the established twin body is combined with the simulated scene, the influence of different environments on the operation of the unmanned equipment is fully considered, an optimal control strategy is determined according to a specified strategy, the control of the unmanned equipment is completed according to the strategy, the unmanned equipment in the real world is guided by virtual control, the unmanned equipment is controlled to execute corresponding tasks until the running state of the unmanned equipment and the task execution condition reach the required effect, and an optimal operation scheme is provided for the unmanned equipment. On one hand, the development and maintenance efficiency of the intelligent unmanned system is improved, and the collaborative design of multiple unmanned devices is realized more efficiently; on the other hand, the unmanned equipment can be well set, simulated and updated, and observation, judgment and decision making of command control can be effectively assisted.
In one embodiment, the interaction layer further comprises a control module;
the evaluation module is further configured to send the optimal control policy to a control module of the interaction layer;
the control module is used for controlling the target unmanned equipment according to the optimal control strategy.
In this embodiment, the interaction layer further includes a control module, and after the evaluation module in the simulation layer determines the optimal control policy, the evaluation module sends the optimal control policy to the control module of the interaction layer, so that the control module controls the target unmanned device according to the received optimal controllable policy. By way of example, if the optimal control policy determined by the evaluation module is "move right", the control module controls the target unmanned device to move right according to the policy, so that the control policy is automatically generated by virtual control and reality, that is, by using a virtual model and a simulation environment, so as to realize control of the real unmanned device, thereby endowing the unmanned device with higher intelligence, and enabling the unmanned device to participate in more complex and more difficult operations.
In one embodiment, the interaction layer includes a data conversion module and a data acquisition module;
The data acquisition module is used for acquiring original state data of the target unmanned equipment, wherein the original state data is heterogeneous data transmitted by a plurality of sensors deployed on the target unmanned equipment, and the heterogeneous data comprises one or more of infrared data, radar data, video data, image data, text data and position data;
the data conversion module is used for converting the original state data into the physical state data, wherein the physical state data is the data which can be identified by the physical twin model.
In this embodiment, the interaction layer obtains the original state data through the data obtaining module, and then the data conversion module converts the original state data into the physical state data. The original state data is data obtained directly through a sensor of the target unmanned aerial vehicle, the data is state information of the target unmanned aerial vehicle is collected in real time by the sensor, and the state information is sent to an interaction layer by each sensor. In general, an unmanned device is deployed with a plurality of sensors to acquire various status data, and the structure and type of the data of each sensor are different, so that the original status data is multi-source heterogeneous data, for example, radar data acquired through a radar, video data or image data acquired through a camera, so that the original status data includes one or more of infrared data, radar data, video data, image data, text data and position data.
After the data acquisition module of the interaction layer acquires the original state data, the original state data needs to be converted into physical state data through the data conversion module. The method is characterized in that the original state data cannot be directly used for a digital intelligent twin layer, such as radar data transmitted by a radar sensor, and a physical twin model in the digital intelligent twin layer cannot be directly processed, so that a data conversion module is required to preprocess the original state data, and data which cannot be directly utilized by the physical twin model in the digital intelligent twin layer is converted into data which can be directly utilized.
In one embodiment, the data acquisition module is further configured to acquire real environment data of the target unmanned device;
the environment simulation module further comprises a similarity calculation unit; the similarity calculation unit is used for calculating semantic similarity between the real environment data and a plurality of prestored simulation environments respectively, and determining the simulation environment with the highest similarity as the unmanned equipment operation environment.
In this embodiment, the real environment data where the target unmanned device is located is obtained through the data obtaining module of the interaction layer. The real environment data may be manually input or obtained through the target unmanned device, which indicates the real environment state of the unmanned device, and specifically may include, but is not limited to, environmental temperature data, environmental brightness data, environmental geographic data, and the like.
The environment simulation module in the simulation layer is pre-stored with a plurality of simulation environments, namely, the module is pre-simulated for different environments, and virtual environments obtained through simulation are stored. And the similarity calculation unit performs semantic similarity calculation with a plurality of stored simulation environments according to the acquired real environment data, so as to determine a simulation environment which is the most similar, and the simulation environment is used as an unmanned equipment operation environment for subsequent matching with a digital intelligent twin body. Therefore, the embodiment realizes that the simulation environment is established based on the acquired real-world environment data, so that the operation environment of the unmanned equipment obtained by simulation has higher similarity with the real environment where the unmanned equipment is located, and the influence of environmental factors is reduced.
In one embodiment, the evaluation module evaluates the plurality of integrated twins to obtain an optimal twins, respectively, by the following formula;
Argmi n(cost(f2-f1),cost(S(cpu、battery、memory，env)))
the cost (f 2, f 1) represents the cost of changing the target unmanned equipment from the current state to the next state, represents the cost required by executing the strategy corresponding to the comprehensive twin, f2 represents the next state of the target unmanned equipment, and f1 represents the current state of the target unmanned equipment; cost (S (cpu, battery, memory, env)) represents the cost of the target drone itself attribute and the cost of the operating environment.
In this embodiment, the evaluation module in the simulation layer may evaluate each comprehensive twin according to the formula, so as to determine an optimal control policy. According to the formula, on one hand, the evaluation module considers the cost required by the target unmanned equipment to change from the current state to the next state, and characterizes the cost required by executing the strategy corresponding to the comprehensive twin. Illustratively, the control strategy corresponding to the comprehensive twin is acceleration, and according to the comprehensive twin, the cost required by changing from the current state (current speed) to the next state (after acceleration) is determined; on the other hand, the evaluation module considers the cost generated by the attribute of the target unmanned device and the cost generated by the running environment (S (cpu, battery, memory, env)), wherein the cost generated by factors such as cpu performance, battery state, memory and environment are included. The evaluation module takes the formula as an evaluation strategy, and determines one with the minimum cost as an optimal twin according to each comprehensive twin, so that the strategy corresponding to the twin is determined as an optimal control strategy, and the obtained strategy enables the target unmanned equipment to finish tasks with the minimum cost.
In one embodiment, the intelligent twin model comprises an algorithm selection module and a plurality of deep learning models, each of the deep learning models being a multimodal fusion of deep learning models;
the algorithm selection module is used for determining a proper target deep learning model from the multiple deep learning models based on a meta learning method according to the unmanned equipment operation environment and the physical state data;
the target deep learning model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies.
In this embodiment, the intelligent twin model in the digital intelligent twin layer includes an algorithm selection module, and the model can determine an appropriate target deep learning model from a plurality of deep learning models according to the unmanned device operating environment and physical state data based on a meta learning method. Specifically, since the acquired physical state data is acquired by various sensors, one or more of infrared data, radar data, video data, image data, text data, and position data are included. The intelligent twin model comprises a plurality of deep learning models, model parameters in different deep learning models are different, and weights of various data are different. How to select an algorithm meeting application requirements from a large number of possible algorithms is an important challenge to be solved in various fields, namely, an algorithm selection problem. Algorithm selection problems are typically solved by either manual or automatic selection methods. The manual selection method relies on experimental trial and error or expert knowledge for selection, and is high in cost and poor in flexibility; and the method based on meta learning automatically selects a proper algorithm according to the characteristics of the problem by designing the algorithm and the model, and has the advantages of low cost, high flexibility and the like. In the night environment, for example, the environment brightness is weak, the definition of the obtained video data or image data is low, and the radar data or infrared data is more effective, so that the algorithm selection module analyzes and determines that a deep learning model with higher weight occupied by the radar data or the infrared data should be selected as a target deep learning model according to the unmanned equipment operation environment and the physical state data. And then, generating a corresponding digital intelligent twin body by utilizing the target deep learning model according to the physical twin body and a plurality of action strategies.
Correspondingly, the physical twin model also comprises a corresponding algorithm selection module and a plurality of deep learning models, and each deep learning model is a multi-modal fusion deep learning model; the algorithm selection module is used for determining a proper target deep learning model from the multiple deep learning models according to the unmanned equipment operation environment and the physical state data based on a meta learning method; the target deep learning model is used for obtaining a corresponding physical twin body according to the physical state data. Specifically, the algorithm selection module in the physical twin model determines a suitable target deep learning model according to the obtained physical state data. Illustratively, in a forest environment, light is darker, target tracking is realized for example, and infrared data has higher utilization value. The algorithm analysis module takes the deep learning model which is more good at processing the infrared data as a target deep learning model, so that a physical twin corresponding to the target unmanned equipment is obtained.
Further, each of the deep learning models in the present embodiment is a multimodal fusion deep learning model. This is because the original state data obtained by the plurality of sensors, and thus the structure and type of each data in the converted physical state data are different, resulting in the physical state data being multi-source heterogeneous data. Thus, by setting the deep learning model as a multi-modal fusion model, because they can accept a plurality of different input modes (e.g., language, image, voice, video) and in some cases produce different output modalities, the deep learning model can generate corresponding twins from multi-source heterogeneous data, so that the twins are more accurate and comprehensive, and more accurate, complete and reliable evaluation results than single modalities can be obtained.
In one embodiment, each of the deep learning models is a deep learning model obtained in advance by BEV distillation.
In this embodiment, since the deep learning model is a multi-modal model, the multi-modal model often has a huge parameter, and if the multi-modal model is directly deployed on the unmanned device, it occupies too much memory and computing power. In order to enable the multi-modal model to be better deployed on small unmanned equipment, knowledge distillation is performed on each model in advance through a BEV distillation mode, and the effect of the small model obtained after distillation is ensured while model parameters are reduced through calibration among different modal data.
In one embodiment, the digital twin layer further comprises a data distribution service module, and data interaction is performed between each model/module in the digital twin layer through the data distribution service module.
In this embodiment, the digital intelligence twin layer further includes a data distribution service module, the model adopts a data distribution service based on DDS technology, and uses data and messages as centers, so that different models/modules can subscribe and publish messages to the data distribution service model continuously, thereby realizing data interaction between the models/modules, and simultaneously providing rich QoS quality of service policies, and guaranteeing real-time, efficient and flexible distribution of data.
In one embodiment, each model/module in the digital intelligent twin layer is used as a micro-service module to form a whole micro-service framework system, and each micro-service module obtains data of the micro-service module deployed on other unmanned equipment or platforms through an RPC.
In this embodiment, the digital intelligent twin layer is based on RPC, and micro-services each model/module therein, so that each model/module constitutes the whole micro-service framework system. Microservices (mi cross vi) is an architectural style, a large complex software application is made up of one or more microservices, each of which can be deployed independently, with loose coupling between each microservice. Each micro-service is focused on completing only one task and performing that task well. In the micro-service framework, each micro-service module acquires data of the micro-service modules deployed on other unmanned equipment or platforms through the RPC, and the micro-services of different algorithms are not affected by each other by utilizing the characteristics of the micro-services, so that the stability of the intelligent unmanned system is improved.
In one embodiment, the physical twinning model further comprises a data presentation module and a model data synchronization module;
The data display module is used for displaying the acquired physical state data of the target unmanned equipment;
the model data synchronization module is used for acquiring model parameter data deployed on other unmanned equipment or platforms by utilizing the RPC, and updating a corresponding model according to the model parameter data.
In this embodiment, the physical twin model further includes a data display module, where the module is configured to display physical state data. In addition, the generated relevant attribute data of the physical twin can be displayed. Therefore, the visual display of the result obtained after each step of data processing is performed as much as possible, and when errors occur, the data of each step can be checked in time to find out the problem. The physical twin model further comprises a model data synchronization module, and the intelligent unmanned system can be deployed on unmanned equipment or an unmanned platform, so that when the deep learning model in the intelligent unmanned system needs to be updated, the model data synchronization module can acquire model parameter data deployed on other unmanned equipment or platforms through an RPC remote call protocol, and update the corresponding deep learning model by utilizing the acquired model parameters, and the automatic model update in the intelligent unmanned system is realized.
In one embodiment, the intelligent twin model includes a report generation module;
the report generation module is used for generating an evaluation report of the digital intelligent twin body and an evaluation report of each deep learning model.
In this embodiment, the intelligent twin model in the intelligent twin layer further includes a report generating module, which is configured to evaluate the intelligent twin and generate an evaluation report, and further evaluate each deep learning model and generate a corresponding evaluation report. Specifically, a deep learning model is utilized to obtain a digital intelligent twin, the integrity and the accuracy of each attribute in the digital intelligent twin are evaluated, and then the deep learning model is evaluated according to an evaluation report of the digital intelligent twin. Illustratively, if the model generates a majority of the mental twins that are evaluated as being excellent, the model is evaluated as being excellent. Selection and training of the deep learning model can thereby be achieved.
Referring to fig. 2, fig. 2 shows a schematic diagram of a functional setting of an intelligent unmanned system, and as shown in fig. 2, a virtual-real interaction layer represents an interaction layer of the intelligent unmanned system, and is mainly used for performing data transmission, data conversion and device control, and performing data interaction with a real physical layer, specifically including sensor data, unmanned device data, environmental data, expert knowledge, and the like; the digital intelligent twin layer is mainly divided into a physical twin model and an intelligent twin model and is used for establishing a digital intelligent twin body according to data acquired by the interaction layer; the operation scene simulation module in the simulation layer represents an environment simulation module in the intelligent unmanned system and is used for simulating to obtain the operation environment of the unmanned equipment, and the simulation system further comprises a similarity calculation unit which is used for calculating to obtain the simulation environment which is most similar to the real environment, and the operation scene generation module in fig. 2 represents a matching module in the intelligent unmanned system and is used for combining the operation environment with the digital intelligent twin to obtain a plurality of comprehensive twin. The twin evaluation module represents an evaluation module in the intelligent unmanned system and is used for evaluating the comprehensive twin by using a set evaluation strategy to obtain an optimal twin, and obtaining an optimal control strategy according to the optimal twin.
The embodiment also provides a method for generating the control strategy of the unmanned equipment based on digital twinning, referring to fig. 3, fig. 3 shows a step flow chart of the method for generating the control strategy of the unmanned equipment, as shown in fig. 3, and the method comprises the following steps:
step S101, acquiring physical state data of the target unmanned equipment;
step S102, modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, wherein the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment;
step S103, obtaining a plurality of digital twins according to the physical twins and a plurality of action strategies, wherein each digital twins is a digital model representing the physical twins to act according to one action strategy;
step S104, simulating to obtain an unmanned equipment operation environment;
step S105, the unmanned equipment operation environment is respectively matched with each intelligent twin to obtain a plurality of comprehensive twin bodies, wherein the comprehensive twin bodies are digital models for representing the motion states of the intelligent twin bodies in the unmanned equipment operation environment;
and S106, evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body.
In one embodiment, the method further comprises
And controlling the target unmanned equipment according to the optimal control strategy.
In one embodiment, obtaining physical state data of the target unmanned device includes:
acquiring original state data of the target unmanned equipment, wherein the original state data is heterogeneous data transmitted by a plurality of sensors deployed on the target unmanned equipment, and the heterogeneous data comprises one or more of infrared data, radar data, video data, image data, text data and position data;
and converting the original state data into the physical state data, wherein the physical state data is the data which can be identified by the physical twin model.
In one embodiment, the method further comprises:
acquiring real environment data of the target unmanned equipment;
simulating to obtain an unmanned equipment operation environment, including:
and respectively carrying out semantic similarity calculation on the real environment data and a plurality of prestored simulation environments, and determining the simulation environment with the highest similarity as the unmanned equipment operation environment.
In one embodiment, evaluating the plurality of integrated twins to obtain an optimal twins comprises:
Respectively evaluating the plurality of comprehensive twins according to the following formula to obtain an optimal twins;
Argmi n(cost(f2-f1),cost(S(cpu、battery、memory，env)))
the cost (f 2, f 1) represents the cost of changing the target unmanned equipment from the current state to the next state, represents the cost required by executing the strategy corresponding to the comprehensive twin, f2 represents the next state of the target unmanned equipment, and f1 represents the current state of the target unmanned equipment; cost (S (cpu, battery, memory, env)) represents the cost of the target drone itself attribute and the cost of the operating environment.
In one embodiment, deriving a plurality of mental twins from the physical twins, and a plurality of action strategies, comprises:
based on a meta learning method, determining a proper target deep learning model from a plurality of deep learning models according to the unmanned equipment operation environment and the physical state data;
and obtaining a plurality of intelligent twins by utilizing the target deep learning model according to the physical twins and a plurality of action strategies, wherein each deep learning model is a multi-modal fusion deep learning model.
In one embodiment, each of the deep learning models is a deep learning model obtained in advance by BEV distillation.
In one embodiment, the method further comprises:
displaying the acquired physical state data of the target unmanned equipment;
acquiring model parameter data deployed on other unmanned equipment or platforms by utilizing the RPC;
and updating the corresponding model according to the model parameter data.
In one embodiment, the method further comprises:
generating an assessment report for the mental twin, and an assessment report for each of the deep learning models.
The embodiment of the application also provides electronic equipment, and referring to fig. 4, fig. 4 shows a schematic diagram of the electronic equipment according to the embodiment of the application. As shown in fig. 4, the electronic device 100 includes: the system comprises a memory 110 and a processor 120, wherein the memory 110 is in communication connection with the processor 120 through a bus, and a computer program is stored in the memory 110 and can run on the processor 120, so that the steps in the unmanned equipment control strategy generation method based on digital twinning disclosed by the embodiment of the application are realized.
The embodiment of the application also provides a computer readable storage medium, on which a computer program/instruction is stored, which when executed by a processor, implements the steps in a digital twin-based unmanned equipment control strategy generation method as disclosed in the embodiment of the application.
The embodiment of the invention also provides a computer program product, which when being run on the electronic equipment, causes a processor to realize the steps in the digital twin-based unmanned equipment control strategy generation method disclosed by the embodiment of the invention when being executed.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, electronic devices, and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The above description of the digital twin-based intelligent unmanned system and method provided by the invention applies specific examples to illustrate the principle and implementation of the invention, and the above examples are only used to help understand the method and core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (9)
1. An intelligent unmanned system based on digital twinning, characterized in that the system is deployed on a target unmanned device, the system comprising:
an interaction layer, configured to obtain physical state data of the target unmanned device;
the digital intelligence twin layer comprises a physical twin model and an intelligent twin model;
the physical twin model is used for modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, and the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment;
the intelligent twin model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies, wherein each intelligent twin body is a digital model for representing the physical twin bodies to act according to one action strategy;
the simulation layer comprises an environment simulation module, an evaluation module and a matching module;
the environment simulation module is used for obtaining the unmanned equipment operation environment through simulation;
the matching module is used for respectively matching the unmanned equipment operation environments with each intelligent twin to obtain a plurality of comprehensive twin, wherein the comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environments;
The evaluation module is used for evaluating the comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body;
the evaluation module is used for evaluating the comprehensive twin bodies respectively through the following formulas to obtain an optimal twin body;
Argmin(cost(f2-f1),cost(S(cpu、battery、memory，env)))；
the cost (f 2, f 1) represents the cost of changing the target unmanned equipment from the current state to the next state, represents the cost required by executing the strategy corresponding to the comprehensive twin, f2 represents the next state of the target unmanned equipment, and f1 represents the current state of the target unmanned equipment; cost (S (cpu, battery, memory, env)) represents the cost of the target drone itself attribute and the cost of the operating environment.
2. The intelligent unmanned system of claim 1, wherein the interaction layer further comprises a control module;
the evaluation module is further configured to send the optimal control policy to a control module of the interaction layer;
the control module is used for controlling the target unmanned equipment according to the optimal control strategy.
3. The intelligent unmanned system of claim 1, wherein the interaction layer comprises a data conversion module and a data acquisition module;
The data acquisition module is used for acquiring original state data of the target unmanned equipment, wherein the original state data is heterogeneous data transmitted by a plurality of sensors deployed on the target unmanned equipment, and the heterogeneous data comprises one or more of infrared data, radar data, video data, image data, text data and position data;
the data conversion module is used for converting the original state data into the physical state data, wherein the physical state data is the data which can be identified by the physical twin model.
4. The intelligent unmanned system of claim 3, wherein the data acquisition module is further configured to acquire real environment data of the target unmanned device;
the environment simulation module further comprises a similarity calculation unit; the similarity calculation unit is used for calculating semantic similarity between the real environment data and a plurality of prestored simulation environments respectively, and determining the simulation environment with the highest similarity as the unmanned equipment operation environment.
5. The intelligent unmanned system of claim 1, wherein the intelligent twin model comprises an algorithm selection module and a plurality of deep learning models, each of the deep learning models being a multimodal fusion of deep learning models;
The algorithm selection module is used for determining a proper target deep learning model from the multiple deep learning models based on a meta learning method according to the unmanned equipment operation environment and the physical state data;
the target deep learning model is used for obtaining a plurality of intelligent twin bodies according to the physical twin bodies and a plurality of action strategies.
6. The intelligent unmanned system of claim 5, wherein the digital twinning layer further comprises a data distribution service module, and wherein data interaction is performed between each model/module in the digital twinning layer through the data distribution service module.
7. The intelligent unmanned system of claim 1, wherein each model/module in the digital intelligent twin layer is used as a micro-service module to form a whole micro-service framework system, and each micro-service module obtains the required data of the micro-service module deployed on other unmanned devices or platforms through the RPC.
8. The intelligent unmanned system of claim 7, wherein the physical twinning model further comprises a data presentation module and a model data synchronization module;
the data display module is used for displaying the acquired physical state data of the target unmanned equipment;
The model data synchronization module is used for acquiring model parameter data deployed on other unmanned equipment or platforms by utilizing the RPC, and updating a corresponding model according to the model parameter data.
9. A method for generating a control strategy of an unmanned device based on digital twinning, which is characterized by comprising the following steps:
acquiring physical state data of target unmanned equipment;
modeling the target unmanned equipment according to the physical state data to obtain a physical twin body, wherein the physical twin body is a digital model for representing the real motion state of the target unmanned equipment at the current moment;
obtaining a plurality of digital twins according to the physical twins and a plurality of action strategies, wherein each digital twins is a digital model for representing the action of the physical twins according to one action strategy;
simulating to obtain an unmanned equipment operation environment;
respectively matching the unmanned equipment operation environments with each intelligent twin to obtain a plurality of comprehensive twin, wherein the comprehensive twin is a digital model for representing the motion state of the intelligent twin in the unmanned equipment operation environments;
evaluating the multiple comprehensive twin bodies to obtain an optimal twin body, and obtaining an optimal control strategy according to the optimal twin body;
Wherein the evaluating the plurality of integrated twins to obtain an optimal twins comprises: respectively evaluating the plurality of comprehensive twins through the following formula to obtain an optimal twins;
Argmin(cost(f2-f1),cost(S(cpu、battery、memory，env)))；
the cost (f 2, f 1) represents the cost of changing the target unmanned equipment from the current state to the next state, represents the cost required by executing the strategy corresponding to the comprehensive twin, f2 represents the next state of the target unmanned equipment, and f1 represents the current state of the target unmanned equipment; cost (S (cpu, battery, memory, env)) represents the cost of the target drone itself attribute and the cost of the operating environment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310147580.6A CN116362109B (en) | 2023-02-09 | 2023-02-09 | Intelligent unmanned system and method based on digital twinning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310147580.6A CN116362109B (en) | 2023-02-09 | 2023-02-09 | Intelligent unmanned system and method based on digital twinning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116362109A CN116362109A (en) | 2023-06-30 |
| CN116362109B true CN116362109B (en) | 2023-09-12 |
Family
ID=86931286
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310147580.6A Active CN116362109B (en) | 2023-02-09 | 2023-02-09 | Intelligent unmanned system and method based on digital twinning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116362109B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111208759A (en) * | 2019-12-30 | 2020-05-29 | 中国矿业大学(北京) | Digital twin intelligent monitoring system for unmanned fully mechanized coal mining face of mine |
| CN113205260A (en) * | 2021-05-07 | 2021-08-03 | 北京首都机场节能技术服务有限公司 | Electric automobile charging pile health state assessment method based on digital twins |
| WO2021227325A1 (en) * | 2020-05-13 | 2021-11-18 | 中国电子科技集团公司第十四研究所 | Digital twin-based production process simulation and optimization method |
| CN115392476A (en) * | 2022-07-07 | 2022-11-25 | 中国人民解放军军事科学院系统工程研究院 | Intelligent twin in unmanned cooperative combat system |
-
2023
- 2023-02-09 CN CN202310147580.6A patent/CN116362109B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111208759A (en) * | 2019-12-30 | 2020-05-29 | 中国矿业大学(北京) | Digital twin intelligent monitoring system for unmanned fully mechanized coal mining face of mine |
| WO2021227325A1 (en) * | 2020-05-13 | 2021-11-18 | 中国电子科技集团公司第十四研究所 | Digital twin-based production process simulation and optimization method |
| CN113205260A (en) * | 2021-05-07 | 2021-08-03 | 北京首都机场节能技术服务有限公司 | Electric automobile charging pile health state assessment method based on digital twins |
| CN115392476A (en) * | 2022-07-07 | 2022-11-25 | 中国人民解放军军事科学院系统工程研究院 | Intelligent twin in unmanned cooperative combat system |
Non-Patent Citations (1)
| Title |
|---|
| 刘婷 等." 基于数字孪生的航空发动机全生命周期管理".《航空动力》.2018,全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116362109A (en) | 2023-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tang et al. | Survey on digital twin edge networks (DITEN) toward 6G | |
| Wu et al. | Multi UAV cluster control method based on virtual core in improved artificial potential field | |
| US20180293498A1 (en) | For hiearchical decomposition deep reinforcement learning for an artificial intelligence model | |
| CN111522258A (en) | Multi-unmanned aerial vehicle cooperative control simulation system and construction method and simulation method thereof | |
| CN101083019A (en) | Rapid evaluating system based on roomage state sensing | |
| CN112766499A (en) | Method for realizing autonomous flight of unmanned aerial vehicle through reinforcement learning technology | |
| CN109376852A (en) | Arithmetic unit and operation method | |
| CN112034888A (en) | Autonomous control cooperation strategy training method for fixed wing unmanned aerial vehicle | |
| Cheng et al. | Research status of artificial neural network and its application assumption in aviation | |
| CN105068421A (en) | Two-degree-of-freedom cooperative control method for multiple mobile robots | |
| CN112068587B (en) | Man/unmanned aerial vehicle co-converged cluster interaction method based on European 26891bird communication mechanism | |
| CN112990485A (en) | Knowledge strategy selection method and device based on reinforcement learning | |
| CN113741533A (en) | Unmanned aerial vehicle intelligent decision-making system based on simulation learning and reinforcement learning | |
| CN112257846A (en) | Neuron model, topology, information processing method, and retinal neuron | |
| CN115099124A (en) | Multi-agent distribution collaborative training simulation method | |
| Liu et al. | Task offloading optimization of cruising UAV with fixed trajectory | |
| Xu et al. | Living with artificial intelligence: A paradigm shift toward future network traffic control | |
| CN109144099A (en) | Unmanned aerial vehicle group action scheme fast evaluation method based on convolutional neural networks | |
| CN115168720A (en) | Content interaction prediction method and related equipment | |
| CN107623590A (en) | The collocation method and system monitored in real time for unmanned aerial vehicle group | |
| CN110806758A (en) | Unmanned aerial vehicle cluster autonomous level self-adaptive adjustment method based on scene fuzzy cognitive map | |
| CN116362109B (en) | Intelligent unmanned system and method based on digital twinning | |
| Meng et al. | Task-oriented metaverse design in the 6G era | |
| CN116700340A (en) | Track planning method and device and unmanned aerial vehicle cluster | |
| CN114967476B (en) | Multi-agent distributed coordination control method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |