CN109085766A - A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system and method - Google Patents
A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system and method Download PDFInfo
- Publication number
- CN109085766A CN109085766A CN201810973690.7A CN201810973690A CN109085766A CN 109085766 A CN109085766 A CN 109085766A CN 201810973690 A CN201810973690 A CN 201810973690A CN 109085766 A CN109085766 A CN 109085766A
- Authority
- CN
- China
- Prior art keywords
- unmanned plane
- control
- control instruction
- manipulator
- airborne processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B17/00—Systems involving the use of models or simulators of said systems
- G05B17/02—Systems involving the use of models or simulators of said systems electric
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a kind of multi-modal unmanned plane human-computer interactions to manipulate semi-matter simulating system, comprising: airborne processor is mounted on unmanned plane；Gesture identification equipment and EEG signals measuring device are connect by serial communication module with interconnecting module, and interconnecting module is connect with airborne processor；Remote controler is connect by PPM interface with airborne processor；Host computer is emulated, carries AirSim simulator thereon, and connect with airborne processor by serial communication module；Ground control station is connect by serial communication module with airborne processor, and is connect by udp protocol with emulation host computer.The invention also discloses a kind of multi-modal unmanned plane human-computer interactions to manipulate Hardware In The Loop Simulation Method.Beneficial effects of the present invention: it realizes the emulation of the brain electricity and gesture control and remote control control unmanned plane during flying process under multiple modalities control, can also realize the emulation of ground control station control unmanned plane during flying process.
Description
Technical field
The present invention relates to air vehicle technique fields, in particular to a kind of multi-modal unmanned plane human-computer interaction manipulation half
Matter simulating system and method.
Background technique
Unmanned plane attacks target as simulation aircraft class, can appoint for the winged, shooting in fire control radar school etc. of air defence weapon system
Business provides aerial target, is Weapon System Development, essential equipment in qualification process.Core composition portion as unmanned plane
/ mono-, flight control system passes through the posture of control unmanned plane, to complete the aerial mission under the various mode of unmanned plane.At nobody
In the development process of machine, human-computer interaction is very important a link.Gesture identification and brain electric control technology be a new generation from
A key technology of right human-computer interaction, relative to traditional mouse, the mode of operation of the contacts such as keyboard, gesture and brain electricity tool
Have the advantages that nature is intuitive, be readily appreciated that, be easy to operate, experience property it is good, be more in line with the daily exchange habit of the mankind, gesture identification
Have become the research hotspot of human-computer interaction scheme with brain electric control.By connected applications between manipulator and unmanned plane, have good
Good application prospect.In the prior art, it can only realize the emulation during ground control station control unmanned plane during flying, cannot achieve
The emulation of manipulator's brain electricity and gesture control unmanned plane during flying process under multiple modalities control, thus can not be to human-computer interaction
Control instruction is targetedly corrected, and then improves the reliability of identification, avoids the generation of false triggering.
Summary of the invention
To solve the above problems, the purpose of the present invention is to provide a kind of multi-modal unmanned plane human-computer interaction manipulation half is in kind
Analogue system and method realize brain electricity and gesture control, remote control control unmanned plane during flying process under multiple modalities control
Emulation, while the emulation of ground control station control unmanned plane during flying process may be implemented.
The present invention provides a kind of multi-modal unmanned plane human-computer interactions to manipulate semi-matter simulating system, comprising:
Airborne processor, is mounted on unmanned plane, for receiving gesture identification equipment, EEG signals measuring device and distant
The Characteristics Control instruction that the control instruction and ground control station that control receiver is sent issue, according to corresponding control instruction or
Specific control instruction controls unmanned plane, and control instruction is transmitted to ground control station by serial communication module and is emulated upper
Machine；
Gesture identification equipment is connect by serial communication module with interconnecting module input terminal, the interconnecting module output
End is connect with the airborne processor communication, for will be converted into control to unmanned plane after collected hand signal fusion treatment
System instruction, and control instruction is passed into the airborne processor；
EEG signals measuring device is connect by serial communication module with interconnecting module input terminal, the interconnecting module
Output end is connect with the airborne processor communication, for will be converted into after the processing of collected eeg signal classification to unmanned plane
Control instruction, and control instruction is passed into the airborne processor；
Remote controler is connect, for referring to the control of the remote controler with the airborne processor communication by PPM interface
Order passes to the airborne processor；
Host computer is emulated, thereon equipped with AirSim simulator, and passes through serial communication module and the airborne processor
Communication connection, for receiving the control instruction of the airborne processor and the specific control instruction of ground control station transmission,
Simulation model, the setting simulation parameter that unmanned plane during flying is established according to control instruction or specific control instruction, carry out unmanned plane
Flight simulation, and the Simulation Interface of display control unmanned plane during flying process；
Ground control station connect by serial communication module with the airborne processor communication, and pass through udp protocol and
The emulation host computer communication connection can be according to demand for receiving the unmanned aerial vehicle (UAV) control instruction of the airborne processor transmission
Specific control instruction is issued to the airborne processor, and specific control instruction is transmitted to the emulation host computer.
As further improvement of the invention, the gesture identification equipment includes:
Electromyography signal measuring device comprising be worn on several electromyographic electrodes on two arms of manipulator, to disappear
Except the driving circuit of common mode interference, modulus conversion chip, micro-control unit MCU and the vibration horse being worn on manipulator's leg
It reaches, several electromyographic electrodes are pasted on manipulator's left forearm and right forearm in the form of annular array, for acquiring manipulator's
Electromyography signal, several electromyographic electrodes and the driving circuit are connected with the modulus conversion chip, the analog-to-digital conversion core
Piece passes through SPI interface and the micro-control unit MCU two-way communication link, the micro-control unit MCU and the vibration motor
It is connected；
Attitude signal measuring device comprising several IMU being worn on two arms of manipulator；
Gesture identification decoding device, if input terminal passes through serial communication module and the micro-control unit MCU and described
Dry IMU is connected, and output end is connected by serial ports with the interconnecting module input terminal, the interconnecting module output end with it is described airborne
Processor communication connection.
As further improvement of the invention, the interconnecting module selects the Arduino mega based on AVR processor
2560 chips have four road serial ports and four tunnel interrupt sources.
As further improvement of the invention, the remote controler is eight channel remote controlers, wherein channel one is control rolling
Turn, channel two is control pitching, and channel three is control throttle, and channel four is control course, and channel five is switching offline mode, packet
It includes from steady offline mode, height-lock control mode and fixed point offline mode, channel six is switching input mode, including remote control side
Formula, brain machine control mode and gesture control mode, channel seven and channel eight are alternate channel.
As further improvement of the invention, the serial communication module uses APC2770 communication module.
Improved as of the invention further, the EEG signals measuring device include be pasted onto manipulator's brain A2 it is dry
The dry electrode of electrode, Fp2 and the dry electrode of C4 and brain electro-detection and processing chip, the brain electro-detection and processing chip will be described
The human thinking information of manipulator is obtained after the dry electrode of A2, the dry electrode of Fp2 and the collected EEG Processing of the dry electrode of C4.
The present invention also provides a kind of multi-modal unmanned plane human-computer interactions to manipulate Hardware In The Loop Simulation Method, comprising:
Step 1, the hand signal of gesture identification equipment acquisition manipulator, obtains the gesture information of manipulator, realizes to behaviour
The identification of control person's gesture, and gesture information is converted to the control instruction to unmanned plane, interconnecting module receives and processes the hand
The control instruction agreement that gesture identification equipment is sent is converted into the identifiable mavlink instruction of airborne processor, and logical by serial ports
News module is sent to the airborne processor；
Step 2, the EEG signals of EEG signals measuring device acquisition manipulator, obtain the human thinking information of manipulator,
And human thinking information is converted to the control instruction to unmanned plane, the interconnecting module receives and processes the EEG signals and surveys
The control instruction agreement that amount equipment is sent is converted into the identifiable mavlink instruction of airborne processor, and passes through serial communication mould
Block is sent to the airborne processor；
Step 3, the interconnecting module reads the encoded PPM wave signal of control instruction of remote controler, is directly forwarded to institute
Airborne processor is stated, the airborne processor obtains the control instruction of the remote controler after decoding to PPM wave signal；
Step 4, the airborne processor is by the gesture identification equipment, the EEG signals measuring device and the remote control
The control instruction of device is transmitted to ground control station and emulation host computer by serial communication module；
Step 5, the ground control station receives the unmanned plane control of the airborne processor transmission by serial communication module
System instruction, and with text and graphical display, if demand when, specific control instruction is transmitted to the emulation by udp protocol
Host computer；
Step 6, the airborne processor controls unmanned plane according to the control instruction received or specific control instruction, described
Interconnecting module receives the mavlink instruction that the airborne processor returns, and reads unmanned plane during flying by serial communication module
Posture information is sent to the gesture identification equipment for feeding back according to certain agreement；
Step 7, the emulation host computer receives control instruction and the institute of the airborne processor by serial communication module
The specific control instruction for stating ground control station transmission, the emulation of unmanned plane during flying is established according to control instruction or specific control instruction
Model, setting simulation parameter carry out flight simulation, and the Simulation Interface of display control unmanned plane during flying process to unmanned plane.
It is further improved as of the invention, in step 1, when gesture identification equipment acquires the hand signal of manipulator, packet
It includes:
Electromyography signal measuring device acquires the electromyography signal of manipulator, and the electromyography signal after analog-to-digital conversion is exported to gesture
It identifies that decoding device carries out fusion treatment, and receives the feedback command from the gesture identification decoding device, according to the feedback
The vibration frequency of vibration motor of the instruction control for being fed back to manipulator's touch-control；
Attitude signal measuring device acquires the upper arm and forearm motion-sensing signal of manipulator, and upper arm and forearm are moved
Transducing signal exports to the gesture identification decoding device and carries out fusion treatment；
The gesture identification decoding device receives the data transmitted from the electromyography signal measuring device, obtains manipulator
Hand exercise status information, receive data transmit from the attitude signal measuring device, obtain manipulator's upper arm with before
Arm motion-sensing signal and after handling operation, obtains the upper arm and forearm movement state information of operator, by hand exercise state
Information and upper arm and forearm movement state information carry out fusion treatment, obtain the gesture information of manipulator, realize to manipulator's hand
The identification of gesture.
It is further improved as of the invention, in step 3, the interconnecting module is read on each PPM waveform using interrupting
Edge and failing edge signal are risen, register is stored in, obtains each channel width of PPM waveform in real time；
When six channels for detecting the remote controler enter remote control mode, simulation pin forwards PPM waveform；
When six channels for detecting the remote controler enter gesture control or brain machine control model, the airborne place is notified
Reason device enters external control model, and serial communication module forwards control instruction, and the gesture identification equipment or the myoelectricity are believed
The control instruction of number measuring device is converted into unmanned plane expectation posture, is packaged into mavlink communications protocol, is sent to by serial ports
The airborne processor after the airborne processor analyzes the instruction, is sent on the airborne processor by uORB communication streams
MIXER mixing device, control unmanned plane motor changes and executes corresponding control instruction.
It is further improved as of the invention, in step 6, passes through the AirSim simulator carried on the emulation host computer
The api interface that carries obtains the flight attitude information of unmanned plane, attitude angle including quaternary number format and is based on NED coordinate
The location information of system, and the library rpclib carried by AirSim simulator reads flight attitude information and is transmitted to serial communication
Module, the interconnecting module receives the flight attitude information of serial communication module forwarding, and sends according to mavlink communications protocol
To the gesture identification equipment for feeding back.
The invention has the benefit that
The manipulator's brain electricity and gesture control and remote control control unmanned plane during flying under multiple modalities control may be implemented
The control instruction of human-computer interaction is targetedly corrected in the emulation of process, realization, and then improves the reliability of identification, avoids missing
The generation of triggering；The emulation of ground control station control unmanned plane during flying process may be implemented simultaneously；
Can intuitively show simulation process, it is intuitive to show unmanned plane during flying overall process, it is subsequent can be to flight course qualitative analysis；
Using HIL HWIL simulation mode, emulation cost is saved, while also improving the confidence of entire analogue system
Degree.
Detailed description of the invention
Fig. 1 is a kind of showing for multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system described in the embodiment of the present invention
It is intended to；
Fig. 2 is the schematic diagram of gesture identification equipment described in the embodiment of the present invention.
Specific embodiment
Embodiment 1, as shown in Figure 1, a kind of multi-modal unmanned plane human-computer interaction manipulates semi-matter simulating system, comprising: hand
Gesture identifies equipment, EEG signals measuring device, remote controler, emulation host computer, ground control station and airborne processor, wherein hand
Gesture identifies that equipment, EEG signals measuring device and remote controler realize the Multi-mode control to unmanned plane.
Airborne processor is mounted on unmanned plane, for receiving gesture identification equipment, EEG signals measuring device and remote control
The Characteristics Control instruction that the control instruction and ground control station that receiver is sent issue, according to corresponding control instruction or spy
Determine control instruction control unmanned plane, and control instruction is transmitted to ground control station by serial communication module and is emulated upper
Machine.
Gesture identification equipment is connect by serial communication module with interconnecting module input terminal, interconnecting module output end with it is airborne
Processor communication connection, for control instruction to unmanned plane will to be converted into after collected hand signal fusion treatment, and will
Control instruction passes to airborne processor.
EEG signals measuring device is connect by serial communication module with interconnecting module input terminal, interconnecting module output end and
Airborne processor communication connection, for being converted into the control instruction to unmanned plane after handling collected eeg signal classification,
And control instruction is passed into airborne processor.
Remote controler is connect by PPM interface with airborne processor communication, for the control instruction of remote controler to be passed to machine
Borne processor.
It emulates on host computer equipped with AirSim simulator, and is connected by serial communication module and airborne processor communication
Connect, the specific control instruction that the control instruction and ground control station for receiver borne processor are transmitted, according to control instruction or
Specific control instruction establishes the simulation model of unmanned plane during flying, setting simulation parameter, carries out flight simulation to unmanned plane, and show
Control the Simulation Interface of unmanned plane during flying process.
Ground control station is connect by serial communication module with airborne processor communication, and by udp protocol and emulation
Position machine communication connection can issue specific control according to demand and refer to for the unmanned aerial vehicle (UAV) control instruction of receiver borne processor transmission
It enables to airborne processor, and specific control instruction is transmitted to emulation host computer.
As shown in Fig. 2, gesture identification equipment includes: that electromyography signal measuring device, attitude signal measuring device and gesture are known
Other decoding device.
Electromyography signal measuring device includes 8 dry a electromyographic electrodes being worn on two arms of manipulator, to eliminate altogether
Driving circuit, modulus conversion chip, micro-control unit MCU and the vibration motor being worn on manipulator's leg of mould interference, 8
Electromyographic electrode is pasted on manipulator's left forearm and right forearm in the form of annular array, for acquiring the electromyography signal of manipulator, 8
A electromyographic electrode and driving circuit are connected with modulus conversion chip, and modulus conversion chip passes through SPI interface and micro-control unit
MCU two-way communication link, micro-control unit MCU are connected with vibration motor.Electromyography signal measuring device is for acquiring manipulator's
Electromyography signal after analog-to-digital conversion is exported to gesture identification and decodes progress fusion treatment, and received and come from gesture by electromyography signal
Identify decoded feedback command, the vibration frequency of the vibration motor according to feedback command control for feeding back to manipulator's touch-control
Rate.
Attitude signal measuring device includes 6 IMU (i.e. Inertial Measurement Unit) being worn on two arms of manipulator, often
The wrist joint of arm nearby, elbow joint nearby and shoulder joint nearby one IMU of bondage.It is used by three on every arm
Property measuring unit measurement data, reject the influence of relative motion between each joint, the movement of arm can be gone out with accurate Analysis
Posture information.IMU includes power module, three axis accelerometer, three-axis gyroscope, three axis magnetometer, and power module is inertia survey
It measures unit and power supply is provided, three axis accelerometer, three-axis gyroscope, three axis magnetometer are respectively used to acquisition manipulator's left arm or the right side
Acceleration, angular speed and magnetic field strength when the movement of arm.Attitude signal measuring device be used for acquire manipulator upper arm and
Forearm motion-sensing signal, and upper arm and forearm motion-sensing signal are exported to gesture identification decoding device and carried out at fusion
Reason.
Gesture identification decoding device input terminal is connected by serial communication module with micro-control unit MCU and several IMU, defeated
Outlet is connected by serial ports with interconnecting module input terminal, and interconnecting module output end is connect with airborne processor communication.For to flesh
Electric signal carries out processing and operation, obtains the hand exercise status information of operator.Gesture identification decoding device is to upper arm with before
Arm motion-sensing signal carries out processing and operation, the upper arm and forearm movement state information of operator is obtained, to hand exercise shape
State information and upper arm and forearm movement state information carry out fusion treatment, obtain the gesture information of manipulator, realize to manipulator
The identification of gesture, and gesture information is converted to the control instruction to UAV Flight Control System, and control instruction is issued
To the airborne processor of unmanned plane, the real-time control to UAV Flight Control System is realized, receive at unmanned aerial vehicle onboard
The status information of device is managed, and feedback command is issued to electromyography signal measuring device according to status information.
EEG signals measuring device includes the dry electrode of A2, the dry electrode of Fp2 and the dry electrode of C4 for being pasted onto manipulator's brain, with
And brain electro-detection and processing chip, brain electro-detection with handle chip the dry electrode of A2, the dry electrode of Fp2 and the dry electrode of C4 is collected
After the region A2 EEG signals, the region Fp2 EEG signals and the processing of the region C4 eeg signal classification, it is converted into corresponding control instruction
It exports to airborne processor.
Remote controler is eight channel remote controlers, wherein channel one is control rolling, and channel two is control pitching, and channel three is
Throttle is controlled, channel four is control course, and channel five is switching offline mode, including from steady offline mode, height-lock control mode
With fixed point offline mode, channel six is switching input mode, including remote control mode (passing through remote control control), brain machine control
Mode processed and (being controlled by manipulator's EEG signals) gesture control mode (being controlled by manipulator's hand signal), channel
Seven and channel eight be alternate channel.
Interconnecting module selects 2560 chip of Arduino mega based on AVR processor, has in four road serial ports and four tunnels
Disconnected source.Effect is: receiving and processing EEG signals measuring device and command protocols that gesture identification equipment is sent, is converted into airborne
The identifiable mavlink instruction of processor, and winged control is sent to by serial ports, while in order to guarantee simulated flight effect more true to nature
Fruit reads the PPM wave signal of remote controler, is directly forwarded to airborne processor, guarantees that remote control distributor feel is most true, passes through serial ports
UAV Attitude information is read, package informatin is solved, is sent to EEG signals measuring device and gesture identification equipment according to certain agreement
Processor for feeding back.The reasons why selecting Arduino 2560 chip of mega: 2560 processor of Arduino mega has body
Small, the light-weight feature of product, it is easy to carry, with when prototype flight can be worn at, while there are four road serial ports, four tunnels are interrupted
Source perfect can handle and communicate (serial ports 3) with EEG signals measuring device and gesture identification equipment, communicate with airborne processor
(serial ports 2), Debugging message print (serial ports 0), and four tunnels are interrupted the PPM wave signal that can guarantee to intercept remote controler and forwarded.
Serial communication module uses APC2770 communication module, and Serial Port Information is converted 2.4G RI radio intelligence by effect,
Up to 1 kilometer or more outside operating distance room.
Embodiment 2, a kind of multi-modal unmanned plane human-computer interaction manipulation Hardware In The Loop Simulation Method, comprising:
Step 1, the hand signal of gesture identification equipment acquisition manipulator, obtains the gesture information of manipulator, realizes to behaviour
The identification of control person's gesture, and gesture information is converted to the control instruction to unmanned plane, interconnecting module receives and processes gesture knowledge
The control instruction agreement that other equipment is sent is converted into the identifiable mavlink instruction of airborne processor, and passes through serial communication mould
Block is sent to airborne processor；
Step 2, the EEG signals of EEG signals measuring device acquisition manipulator, obtain the human thinking information of manipulator,
And human thinking information is converted to the control instruction to unmanned plane, interconnecting module receives and processes EEG signals measuring device hair
The control instruction agreement come is converted into the identifiable mavlink instruction of airborne processor, and is sent to by serial communication module
Airborne processor；
Step 3, interconnecting module reads the encoded PPM wave signal of control instruction of remote controler, is directly forwarded to airborne place
Device is managed, airborne processor obtains the control instruction of remote controler after decoding to PPM wave signal；
Step 4, airborne processor passes through the control instruction of gesture identification equipment, EEG signals measuring device and remote controler
Serial communication module is transmitted to ground control station and emulation host computer；
Step 5, the unmanned aerial vehicle (UAV) control instruction that ground control station is transmitted by serial communication module receiver borne processor, and
With text and graphical display, if demand when, specific control instruction is transmitted to emulation host computer by udp protocol；
Step 6, airborne processor controls unmanned plane, interconnecting module according to the control instruction received or specific control instruction
The mavlink instruction that receiver borne processor returns, and unmanned plane during flying posture information is read by serial communication module, according to
Certain agreement is sent to gesture identification equipment for feeding back；
Step 7, emulation host computer passes through the control instruction and ground control station of serial communication module receiver borne processor
The specific control instruction of transmission establishes the simulation model of unmanned plane during flying according to control instruction or specific control instruction, setting is imitated
True parameter carries out flight simulation, and the Simulation Interface of display control unmanned plane during flying process to unmanned plane.
Wherein, in step 1, when gesture identification equipment acquires the hand signal of manipulator, comprising:
Electromyography signal measuring device acquires the electromyography signal of manipulator, and the electromyography signal after analog-to-digital conversion is exported to gesture
It identifies that decoding device carries out fusion treatment, and receives the feedback command from gesture identification decoding device, according to the feedback command
Control the vibration frequency of the vibration motor for feeding back to manipulator's touch-control；
Attitude signal measuring device acquires the upper arm and forearm motion-sensing signal of manipulator, and upper arm and forearm are moved
Transducing signal exports to gesture identification decoding device and carries out fusion treatment；
Gesture identification decoding device receives the data transmitted from electromyography signal measuring device, obtains the hand fortune of manipulator
Dynamic status information, receives the data transmitted from attitude signal measuring device, obtains manipulator's upper arm and forearm motion-sensing letter
Number and after handling operation, obtain the upper arm and forearm movement state information of operator, by hand exercise status information and upper arm and
Forearm movement state information carries out fusion treatment, obtains the gesture information of manipulator, realizes the identification to manipulator's gesture.
PPM waveform is that the high level square wave fixed by several impulse amplitudes forms, and low level length is each logical
Road value, a PPM wave period is 20ms, can theoretically read 10 channel values.Wherein, in step 3, in interconnecting module utilization
It is disconnected, each PPM waveform rising edge and failing edge signal are read, register is stored in, obtains each channel width of PPM waveform in real time；
When six channels for detecting remote controler enter remote control mode, simulation pin forwards PPM waveform；When detecting remote controler
When six channels enter gesture control or brain machine control model, airborne processor is notified to enter external control model, serial communication mould
Block forwards control instruction, converts unmanned plane expectation appearance for the control instruction of gesture identification equipment or electromyography signal measuring device
State is packaged into mavlink communications protocol, is sent to airborne processor after airborne processor analyzes the instruction by serial ports and passes through
UORB communication streams are sent to the MIXER mixing device on airborne processor, and control unmanned plane motor, which changes, executes corresponding control
System instruction.
Wherein, in step 6, nothing is obtained by the api interface that the AirSim simulator carried on emulation host computer carries
Man-machine flight attitude information, the attitude angle including quaternary number format and the location information based on NED coordinate system, and pass through
The library rpclib that AirSim simulator carries reads flight attitude information and is transmitted to serial communication module, and interconnecting module receives string
The flight attitude information of mouth communication module forwarding, and gesture identification equipment is sent to for feeding back according to mavlink communications protocol.
AirSim simulator parameter of the invention is as follows: serial port baud rate 11500, data bit 8, stop position 1, school
Testing position is NONE, flow control NONE.
Emulation of the invention uses HIL HWIL simulation, and HWIL simulation is compared with pure matter emulation, most prominent feature
It is to significantly reduce testing cost, another advantage is that reducing the interference of system.Since there are many equipment building block such as aircraft,
Degree of freedom in system is relatively high.When being emulated, although all-real object platform can obtain it is more true comprehensive as a result,
The complexity for improving data simultaneously, increases the difficulty of analysis.And HWIL simulation then can make simulation process more controllable,
It obtains relatively true comprehensive data simultaneously, reduces the freedom degree of data.In addition, because semi-matter simulating system price is opposite
It is cheap, greatly improve testing efficiency.Compared with pure digi-tal emulation, although HIL emulates higher cost, it will need to emulate hard
Part introduces emulation circuit, improves the confidence level of entire analogue system.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of multi-modal unmanned plane human-computer interaction manipulates semi-matter simulating system characterized by comprising
Airborne processor, is mounted on unmanned plane, connects for receiving gesture identification equipment, EEG signals measuring device and remote control
The control instruction that device is sent and the Characteristics Control instruction that ground control station issues are received, according to corresponding control instruction or specific
Control instruction controls unmanned plane, and control instruction is transmitted to ground control station and emulation host computer by serial communication module；
Gesture identification equipment is connect by serial communication module with interconnecting module input terminal, the interconnecting module output end and
The airborne processor communication connection, for that will be converted into referring to the control of unmanned plane after collected hand signal fusion treatment
It enables, and control instruction is passed into the airborne processor；
EEG signals measuring device is connect by serial communication module with interconnecting module input terminal, the interconnecting module output
End is connect with the airborne processor communication, for being converted into the control to unmanned plane after handling collected eeg signal classification
System instruction, and control instruction is passed into the airborne processor；
Remote controler is connect by PPM interface with the airborne processor communication, for passing the control instruction of the remote controler
Pass the airborne processor；
Host computer is emulated, thereon equipped with AirSim simulator, and passes through serial communication module and the airborne processor communication
Connection, for receiving the control instruction of the airborne processor and the specific control instruction of ground control station transmission, according to
Control instruction or specific control instruction establish the simulation model of unmanned plane during flying, setting simulation parameter, fly to unmanned plane
Simulation, and the Simulation Interface of display control unmanned plane during flying process；
Ground control station is connect by serial communication module with the airborne processor communication, and pass through udp protocol with it is described
Host computer communication connection is emulated, for receiving the unmanned aerial vehicle (UAV) control instruction of the airborne processor transmission, can be issued according to demand
Specific control instruction is transmitted to the emulation host computer to the airborne processor by specific control instruction.
2. multi-modal unmanned plane human-computer interaction according to claim 1 manipulates semi-matter simulating system, which is characterized in that institute
Stating gesture identification equipment includes:
Electromyography signal measuring device comprising several electromyographic electrodes for being worn on two arms of manipulator, to eliminate altogether
Driving circuit, modulus conversion chip, micro-control unit MCU and the vibration motor being worn on manipulator's leg of mould interference, if
A dry electromyographic electrode is pasted on manipulator's left forearm and right forearm in the form of annular array, and the myoelectricity for acquiring manipulator is believed
Number, several electromyographic electrodes and the driving circuit are connected with the modulus conversion chip, and the modulus conversion chip passes through
SPI interface and the micro-control unit MCU two-way communication link, the micro-control unit MCU are connected with the vibration motor；
Attitude signal measuring device comprising several IMU being worn on two arms of manipulator；
Gesture identification decoding device, input terminal pass through serial communication module and the micro-control unit MCU and several IMU
It is connected, output end is connected by serial ports with the interconnecting module input terminal, the interconnecting module output end and the airborne processing
Device communication connection.
3. multi-modal unmanned plane human-computer interaction according to claim 2 manipulates semi-matter simulating system, which is characterized in that institute
It states interconnecting module and selects 2560 chip of Arduino mega based on AVR processor, there are four road serial ports and four tunnel interrupt sources.
4. multi-modal unmanned plane human-computer interaction according to claim 1 manipulates semi-matter simulating system, which is characterized in that institute
Stating remote controler is eight channel remote controlers, wherein channel one is control rolling, and channel two is control pitching, and channel three is control oil
Door, channel four are control course, and channel five is switching offline mode, including from steady offline mode, height-lock control mode and fixed point
Offline mode, channel six are switching input mode, including remote control mode, brain machine control mode and gesture control mode, are led to
Road seven and channel eight are alternate channel.
5. multi-modal unmanned plane human-computer interaction according to claim 1 manipulates semi-matter simulating system, which is characterized in that institute
Serial communication module is stated using APC2770 communication module.
6. multi-modal unmanned plane human-computer interaction according to claim 1 manipulates semi-matter simulating system, which is characterized in that institute
Stating EEG signals measuring device includes the dry electrode of A2 for being pasted onto manipulator's brain, the dry electrode of Fp2 and the dry electrode of C4 and brain electricity
Detection and processing chip, the brain electro-detection collect the dry electrode of the A2, the dry electrode of Fp2 and the dry electrode of C4 with processing chip
EEG Processing after obtain manipulator human thinking information.
7. a kind of multi-modal unmanned plane human-computer interaction manipulates Hardware In The Loop Simulation Method characterized by comprising
Step 1, the hand signal of gesture identification equipment acquisition manipulator, obtains the gesture information of manipulator, realizes to manipulator
The identification of gesture, and gesture information is converted to the control instruction to unmanned plane, interconnecting module, which receives and processes the gesture, to be known
The control instruction agreement that other equipment is sent is converted into the identifiable mavlink instruction of airborne processor, and passes through serial communication mould
Block is sent to the airborne processor；
Step 2, the EEG signals of EEG signals measuring device acquisition manipulator obtain the human thinking information of manipulator, and will
Human thinking information is converted to the control instruction to unmanned plane, and the interconnecting module receives and processes the EEG signals measurement and sets
The control instruction agreement that preparation is come is converted into the identifiable mavlink instruction of airborne processor, and is sent out by serial communication module
Give the airborne processor；
Step 3, the interconnecting module reads the encoded PPM wave signal of control instruction of remote controler, is directly forwarded to the machine
Borne processor, the airborne processor obtain the control instruction of the remote controler after decoding to PPM wave signal；
Step 4, the airborne processor is by the gesture identification equipment, the EEG signals measuring device and the remote controler
Control instruction is transmitted to ground control station and emulation host computer by serial communication module；
Step 5, the ground control station is referred to by the unmanned aerial vehicle (UAV) control that serial communication module receives the airborne processor transmission
Enable, and with text and graphical display, if demand when, by specific control instruction by udp protocol be transmitted to it is described emulate it is upper
Machine；
Step 6, the airborne processor controls unmanned plane, the switching according to the control instruction received or specific control instruction
Module receives the mavlink instruction that the airborne processor returns, and reads unmanned plane during flying posture by serial communication module
Information is sent to the gesture identification equipment for feeding back according to certain agreement；
Step 7, the emulation host computer receives the control instruction of the airborne processor and described by serial communication module
The specific control instruction of face control station transmission, the emulation mould of unmanned plane during flying is established according to control instruction or specific control instruction
Type, setting simulation parameter carry out flight simulation, and the Simulation Interface of display control unmanned plane during flying process to unmanned plane.
8. multi-modal unmanned plane human-computer interaction according to claim 7 manipulates Hardware In The Loop Simulation Method, which is characterized in that step
In rapid 1, when gesture identification equipment acquires the hand signal of manipulator, comprising:
Electromyography signal measuring device acquires the electromyography signal of manipulator, and the electromyography signal after analog-to-digital conversion is exported to gesture identification
Decoding device carries out fusion treatment, and receives the feedback command from the gesture identification decoding device, according to the feedback command
Control the vibration frequency of the vibration motor for feeding back to manipulator's touch-control；
Attitude signal measuring device acquires the upper arm and forearm motion-sensing signal of manipulator, and by upper arm and forearm motion-sensing
Signal exports to the gesture identification decoding device and carries out fusion treatment；
The gesture identification decoding device receives the data transmitted from the electromyography signal measuring device, obtains the hand of manipulator
Portion's movement state information receives the data transmitted from the attitude signal measuring device, obtains manipulator's upper arm and forearm fortune
After moving transducing signal and handling operation, the upper arm and forearm movement state information of operator are obtained, by hand exercise status information
And upper arm and forearm movement state information carry out fusion treatment, obtain the gesture information of manipulator, realize to manipulator's gesture
Identification.
9. multi-modal unmanned plane human-computer interaction according to claim 7 manipulates Hardware In The Loop Simulation Method, which is characterized in that step
In rapid 3, the interconnecting module reads each PPM waveform rising edge and failing edge signal using interrupting, and is stored in register, in real time
Obtain each channel width of PPM waveform；
When six channels for detecting the remote controler enter remote control mode, simulation pin forwards PPM waveform；
When six channels for detecting the remote controler enter gesture control or brain machine control model, the airborne processor is notified
Into external control model, serial communication module forwards control instruction, and the gesture identification equipment or the electromyography signal are surveyed
The control instruction of amount equipment is converted into unmanned plane expectation posture, is packaged into mavlink communications protocol, is sent to by serial ports described
Airborne processor after the airborne processor analyzes the instruction, is sent on the airborne processor by uORB communication streams
MIXER mixing device, control unmanned plane motor, which changes, executes corresponding control instruction.
10. multi-modal unmanned plane human-computer interaction according to claim 7 manipulates Hardware In The Loop Simulation Method, which is characterized in that
In step 6, the flight of unmanned plane is obtained by the api interface that the AirSim simulator carried on the emulation host computer carries
Posture information, the attitude angle including quaternary number format and the location information based on NED coordinate system, and pass through AirSim simulator
The library rpclib carried reads flight attitude information and is transmitted to serial communication module, and the interconnecting module receives serial communication mould
The flight attitude information of block forwarding, and the gesture identification equipment is sent to for feeding back according to mavlink communications protocol.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810973690.7A CN109085766A (en) | 2018-08-24 | 2018-08-24 | A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810973690.7A CN109085766A (en) | 2018-08-24 | 2018-08-24 | A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109085766A true CN109085766A (en) | 2018-12-25 |
Family
ID=64794578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810973690.7A Pending CN109085766A (en) | 2018-08-24 | 2018-08-24 | A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109085766A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109778932A (en) * | 2019-03-22 | 2019-05-21 | 江苏徐工工程机械研究院有限公司 | A kind of contactless digging machine arm control system and control method based on arm posture |
| CN110471444A (en) * | 2019-08-19 | 2019-11-19 | 西安微电子技术研究所 | UAV Intelligent barrier-avoiding method based on autonomous learning |
| CN110706551A (en) * | 2019-10-25 | 2020-01-17 | 中国航空工业集团公司沈阳飞机设计研究所 | Multi-mode unmanned aerial vehicle operator training simulator |
| CN110900627A (en) * | 2019-11-29 | 2020-03-24 | 哈尔滨工程大学 | Shooting robot device based on brain control technology and remote control technology |
| CN112987705A (en) * | 2021-03-02 | 2021-06-18 | 北京航空航天大学 | Verification system of automatic airplane sliding running-away technology based on 5G transmission |
| CN113253750A (en) * | 2021-06-28 | 2021-08-13 | 北京科技大学 | Multi-mode control system for flapping wing aircraft |
| CN115859699A (en) * | 2023-03-02 | 2023-03-28 | 北京理工大学 | Large-scale fine-grained clustered ammunition simulation system providing three-dimensional scene rendering |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102163381A (en) * | 2011-03-17 | 2011-08-24 | 山东电力研究院 | Unmanned aerial vehicle electric power pipeline walking simulation training system |
| CN203551961U (en) * | 2013-09-24 | 2014-04-16 | 中国兵器工业计算机应用技术研究所 | Avionics hardware-in-the-loop simulation system for unmanned plane |
| CN106773787A (en) * | 2016-12-28 | 2017-05-31 | 中国航空工业集团公司西安飞机设计研究所 | A kind of unmanned plane during flying simulation demo verification platform based on serial communication |
| CN107203665A (en) * | 2017-05-22 | 2017-09-26 | 成都飞机工业（集团）有限责任公司 | Unmanned plane all round computer, computer based control system and design method |
| CN107643695A (en) * | 2017-09-07 | 2018-01-30 | 天津大学 | Someone/unmanned plane cluster formation VR emulation modes and system based on brain electricity |
| CN107817731A (en) * | 2017-11-27 | 2018-03-20 | 中国兵器工业计算机应用技术研究所 | Merge unmanned the platform control system and control method of myoelectricity and brain electric information |
| CN107943294A (en) * | 2017-11-27 | 2018-04-20 | 中国兵器工业计算机应用技术研究所 | The UAV Flight Control System and control method that can be manipulated with sign language |
| CN108008646A (en) * | 2017-12-22 | 2018-05-08 | 成都才智圣有科技有限责任公司 | A kind of unmanned plane avionics semi-matter simulating system |
| CN108053714A (en) * | 2017-11-10 | 2018-05-18 | 广东电网有限责任公司教育培训评价中心 | Multi-rotor unmanned aerial vehicle based on polling transmission line makes an inspection tour operation simulation training system |
-
2018
- 2018-08-24 CN CN201810973690.7A patent/CN109085766A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102163381A (en) * | 2011-03-17 | 2011-08-24 | 山东电力研究院 | Unmanned aerial vehicle electric power pipeline walking simulation training system |
| CN203551961U (en) * | 2013-09-24 | 2014-04-16 | 中国兵器工业计算机应用技术研究所 | Avionics hardware-in-the-loop simulation system for unmanned plane |
| CN106773787A (en) * | 2016-12-28 | 2017-05-31 | 中国航空工业集团公司西安飞机设计研究所 | A kind of unmanned plane during flying simulation demo verification platform based on serial communication |
| CN107203665A (en) * | 2017-05-22 | 2017-09-26 | 成都飞机工业（集团）有限责任公司 | Unmanned plane all round computer, computer based control system and design method |
| CN107643695A (en) * | 2017-09-07 | 2018-01-30 | 天津大学 | Someone/unmanned plane cluster formation VR emulation modes and system based on brain electricity |
| CN108053714A (en) * | 2017-11-10 | 2018-05-18 | 广东电网有限责任公司教育培训评价中心 | Multi-rotor unmanned aerial vehicle based on polling transmission line makes an inspection tour operation simulation training system |
| CN107817731A (en) * | 2017-11-27 | 2018-03-20 | 中国兵器工业计算机应用技术研究所 | Merge unmanned the platform control system and control method of myoelectricity and brain electric information |
| CN107943294A (en) * | 2017-11-27 | 2018-04-20 | 中国兵器工业计算机应用技术研究所 | The UAV Flight Control System and control method that can be manipulated with sign language |
| CN108008646A (en) * | 2017-12-22 | 2018-05-08 | 成都才智圣有科技有限责任公司 | A kind of unmanned plane avionics semi-matter simulating system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109778932A (en) * | 2019-03-22 | 2019-05-21 | 江苏徐工工程机械研究院有限公司 | A kind of contactless digging machine arm control system and control method based on arm posture |
| CN110471444A (en) * | 2019-08-19 | 2019-11-19 | 西安微电子技术研究所 | UAV Intelligent barrier-avoiding method based on autonomous learning |
| CN110471444B (en) * | 2019-08-19 | 2022-07-12 | 西安微电子技术研究所 | Unmanned aerial vehicle intelligent obstacle avoidance method based on autonomous learning |
| CN110706551A (en) * | 2019-10-25 | 2020-01-17 | 中国航空工业集团公司沈阳飞机设计研究所 | Multi-mode unmanned aerial vehicle operator training simulator |
| CN110900627A (en) * | 2019-11-29 | 2020-03-24 | 哈尔滨工程大学 | Shooting robot device based on brain control technology and remote control technology |
| CN112987705A (en) * | 2021-03-02 | 2021-06-18 | 北京航空航天大学 | Verification system of automatic airplane sliding running-away technology based on 5G transmission |
| CN112987705B (en) * | 2021-03-02 | 2022-06-28 | 北京航空航天大学 | Verification system of airplane automatic sliding running and driving-off technology based on 5G transmission |
| CN113253750A (en) * | 2021-06-28 | 2021-08-13 | 北京科技大学 | Multi-mode control system for flapping wing aircraft |
| CN115859699A (en) * | 2023-03-02 | 2023-03-28 | 北京理工大学 | Large-scale fine-grained clustered ammunition simulation system providing three-dimensional scene rendering |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109085766A (en) | A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system and method | |
| CN108983636B (en) | Man-machine intelligent symbiotic platform system | |
| CN106773787A (en) | A kind of unmanned plane during flying simulation demo verification platform based on serial communication | |
| Qian et al. | Developing a gesture based remote human-robot interaction system using kinect | |
| CN107357311B (en) | Unmanned aerial vehicle reconnaissance system based on hybrid control technology | |
| CN104991650A (en) | Gesture controller and virtual reality system | |
| CN104317386A (en) | Action recognition method of posture sequence finite-state machine | |
| CN108961910A (en) | A kind of VR fire drill device | |
| CN104950695A (en) | Universal UAV (unmanned aerial vehicle) vision simulation platform | |
| CN107943282A (en) | A kind of man-machine interactive system and method based on augmented reality and wearable device | |
| CN110033764A (en) | Sound control method, device, system and the readable storage medium storing program for executing of unmanned plane | |
| CN110297540A (en) | A kind of human-computer interaction device and man-machine interaction method | |
| CN208796050U (en) | A kind of multi-modal unmanned plane human-computer interaction manipulation semi-matter simulating system | |
| CN103327099B (en) | A kind of network-based social interaction equipment and its control method | |
| Jie et al. | Design of multi-mode UAV human-computer interaction system | |
| Keir et al. | Gesture-recognition with non-referenced tracking | |
| Jang et al. | Omnipotent virtual giant for remote human–swarm interaction | |
| Müezzinoğlu et al. | Wearable glove based approach for human-UAV interaction | |
| CN108942938A (en) | A kind of four axis robotic arm motion control methods and system based on eeg signal | |
| Alimohammadzadeh et al. | Dronevision: An Experimental 3D Testbed for Flying Light Specks | |
| CN204790857U (en) | Gesture controller and virtual reality system | |
| Wu et al. | Omnidirectional mobile robot control based on mixed reality and semg signals | |
| CN110757471A (en) | Dance robot system based on computer vision and operation method thereof | |
| CN111409068A (en) | Bionic manipulator control system and bionic manipulator | |
| Mughees et al. | Gesture controlled hexacopter using MEMS technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |