CN113552950B

CN113552950B - Virtual and real interaction method for virtual cockpit

Info

Publication number: CN113552950B
Application number: CN202110904548.9A
Authority: CN
Inventors: 李刚; 程刚宁
Original assignee: Shanghai Xuanwu Technology Co ltd
Current assignee: Shanghai Xuanwu Technology Co ltd
Priority date: 2021-08-06
Filing date: 2021-08-06
Publication date: 2022-09-20
Anticipated expiration: 2041-08-06
Also published as: CN113552950A

Abstract

The invention relates to the technical field of virtual cabins, and discloses a virtual and real interaction method for a virtual cabin, which comprises the following steps: calibrating the position angle relation of the whole physical object system of the virtual cockpit and a virtual world corresponding to the tracking system; carrying out operation tracking on the whole physical object of the virtual cabin; adjusting the interaction speed of the physical object operation of the driver and the response of the virtual object in the virtual scene to enable the response of the virtual object in the virtual scene to be synchronously matched with the physical operation state of the driver; and displaying the virtual object in the virtual scene on the display end in a real-time visualized mode in response to the motion. The method tracks the operation of the driver in the cockpit, so that the user observes and experiences the physical world in an operation mode in nature, reasonably designs the operation time, and enables the response of the system to be matched with the state of physical operation, thereby realizing the technical effect of more real immersive feeling.

Description

Virtual and real interaction method for virtual cockpit

Technical Field

The invention relates to the technical field of virtual cabins, in particular to a virtual and real interaction method for a virtual cabin.

Background

Since the birth of the driving simulator, the characteristics of convenience, safety, low cost and high efficiency make the driving simulator become an important auxiliary tool for researching the running performance and the safety performance of the vehicle; in the aspect of traffic civil engineering, the driving simulator has important significance on the design and evaluation of traffic road alignment design, road sign design, environment design and other road related facilities.

However, in actual use, a series of problems that the reliability of the model is low, the image similarity measure and the image quality evaluation are difficult, the construction workload and data of a complex high-simulation virtual environment are huge, a human-computer interaction mechanism is unnatural, the immersion is lacked, the real-time performance and consistency of virtual reality during multiple users are poor and the like exist, and the further expansion of an application scene is always restricted.

Since vision is a significant proportion of the sensory experience of driving. At present, a plurality of companies and research institutions invest in a large amount of driving simulation systems researched and developed by manpower and material resources, and more sense of immersion and sense of reality of a view in the driving system is improved by combining with a virtual reality technology with stereoscopic vision. With the development of stereoscopic vision technology, various technical routes such as mainly using a head-mounted display, surround type stereoscopic projection, real vehicle window modification and the like are formed in the market at present. The solutions of these techniques are all able to meet to some extent the visual needs of current driver simulators. With the benefit, the current automobile driving simulation system gradually turns from a passive driving simulator to an active driving simulator, and the application range also covers the wide range of driving training, development and entertainment.

However, driving is not a pure experience process, and in actual use, very strong demands are made on interaction. Due to the complexity of the cabin itself, it is very difficult to track the driver's movements. The majority of the solutions currently prevailing are the abstraction of the driver's operation as sensor events of the vehicle operating means. This causes some splitting of the interaction in the synchronization of the physical and virtual environments, resulting in a bottleneck for the difficult experience.

Disclosure of Invention

Technical problem to be solved

Among the mainstream virtual systems at present, the main cause of immersive disruption is: the interaction mode and the real driving mode have larger difference; experience logic is not perfect; data exchange has a delay that affects the sense of substitution;

in order to solve the problems and achieve a more real immersive feeling, the operation of a driver in a cockpit is tracked, so that a user can observe and experience the physical world in a natural operation mode, the operation time is reasonably designed, and the response of the system is matched with the state of physical operation.

(II) technical scheme

In order to achieve the purpose, the invention provides the following technical scheme:

a virtual and real interaction method for a virtual cockpit comprises the following steps:

calibrating the position angle relation between the whole physical object system of the virtual cabin and a virtual world corresponding to a tracking system;

step two, performing operation tracking on the whole physical object of the virtual cabin, specifically as follows:

step S1, the driver performs an operation of the physical object in the virtual cabin and designs the operation time of the driver;

step S2, the tracking system firstly tracks the operation of the driver in the cockpit, and then transmits the physical object operation to the virtual object operation in the virtual scene on the premise of ensuring that the requirements of different data types in real time or reliability can be met by using different communication protocols, so that the user can observe and experience the physical world in the natural world operation mode;

adjusting the interaction speed of the physical object operation of the driver and the response of the virtual object in the virtual scene to enable the response of the virtual object in the virtual scene to be synchronously matched with the physical operation state of the driver;

and fourthly, displaying the response motion of the virtual object in the virtual scene on a display end in a real-time visual mode, so that the physical operation of a driver in the virtual cabin is visually visible and tactually real.

Further, the first step: on the basis that the calibration of the tracking system is completed, the whole physical object of the virtual cabin is in one-to-one correspondence with the virtual object in the coordinate system of the virtual world corresponding to the tracking system.

Further, the first step: the rapid calibration of the virtual world and the physical world comprises the following steps: one-point positioning, two-point positioning and multi-point positioning.

Further, the one-point positioning is used for an operation object which only needs to confirm the position, does not change the direction, or does not influence the experience due to the direction change: the device comprises an instrument panel, a steering wheel, a brake pedal, an accelerator pedal, a parking brake, a combined indicator light and an indication panel.

Further, the two-point positioning is used for an operation object which needs to confirm the position and the rotation, changes the direction and affects the experience: virtual cabin basic platform, indicator light switch, gear shift lever, headlight switch, marker light switch, wiper switch.

Further, the multi-point positioning is used for calibration of the whole virtual cabin base platform, and the object itself may be relatively changed and may affect the experienced operation object: a seat and a safety belt.

Further, the method is used for a cockpit system using VR helmet for driving simulation, and can be compatible with a use scene of surrounding stereo projection or window screen transformation.

Further, the method uses a game engine to perform dynamic visual calculation, and can also be compatible with an industrial visualization engine and other engines through a VRPN protocol.

(III) advantageous technical effects

Compared with the prior art, the invention has the following beneficial technical effects:

the method comprises the steps of establishing a one-to-one correspondence relationship between the whole physical object of the virtual cabin and the virtual object in a coordinate system of the virtual world corresponding to a tracking system by calibrating the position angle relationship between the whole system and the virtual world;

the physical operation of a driver in the cockpit is tracked, so that a user can observe and experience the physical world in an operation mode in nature;

because all operations are physical objects and the virtual objects are driven to move in the same way, different communication protocols can be used to ensure that the requirements of different data types on real-time performance or reliability can be met;

the interaction speed of the physical object operation of the driver and the response of the virtual object in the virtual scene is adjusted, so that the response of the virtual object in the virtual scene is synchronously matched with the physical operation state of the driver;

therefore, the consistency of all operation objects of the virtual cockpit and the physical cockpit in vision and body feeling can be improved, the physical operation is visible in vision and real in touch, and the beneficial technical effect of more real immersive feeling is realized.

Drawings

Fig. 1 is a flowchart of the steps of a virtual cockpit virtual-real interaction method;

FIG. 2 is a diagram of the physical location of a virtual cockpit base platform;

fig. 3 is a diagram of the physical location of the steering wheel.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The invention does not relate to a method for positioning and tracking the physical world and calibrating the corresponding virtual world coordinates, but directly uses the specific result of the method for tracking;

the method does not relate to the manufacturing process of the virtual vehicle, but assumes that a digital model and a corresponding scene with the completion degree capable of meeting the visualization requirement are provided, and provides the mapping relation between the physical operation and the virtual scene on the basis;

a virtual and real interaction method for a virtual cockpit, as shown in fig. 1, includes the following steps:

the calibration in this step is performed on the basis that the calibration of the tracking system is already completed, and the purpose is to establish a one-to-one correspondence relationship between the whole physical object of the virtual cockpit and the virtual object in the coordinate system of the virtual world corresponding to the tracking system;

the physical object composition of the virtual cockpit comprises: the system comprises a virtual cabin base platform, seats (comprising seat operation buttons), an indication panel, a steering wheel, a steering lamp switch, a wiper switch, a gear lever, a headlight switch, an outline marker lamp switch, a brake pedal, an accelerator pedal and a safety belt;

the virtual cabin base platform can adjust the position and the angle, and is integrally adjusted when the whole cabin is deployed;

as shown in fig. 2, assuming that AC is the direction of the head of the virtual cockpit base platform, four points of an ABCD with symmetrical positions use tracked objects with completely identical structures, or the same tracked object is used to sample at four positions respectively, to obtain information of the four points, wherein intersection points of the AD and the BC are aligned to the origin of coordinates of the tracking system, a connecting line of a midpoint of the AC and a midpoint of the BD is aligned to an X axis of the tracking system, a positive direction of the X axis points to the midpoint of the AC from the origin, a positive direction of the Z axis is opposite to a direction of gravity, a connecting line of the midpoint of the AB and the midpoint of the CD is aligned to a Y axis, and a positive direction of the Y axis points to the midpoint of the AB from the origin;

to ensure that the entire physical object system is more easily adjusted and calculated, the points obtained in the tracking system by the physical location of the virtual cabin base platform can be offset according to the virtual world coordinate system; virtual world coordinate [ A ] in the tracking system through the acquired four points _x 、A _y 、A _z ]，[B _x 、B _y 、B _z ]，[C _x 、C _y 、C _z ]，[D _x 、D _y 、D _z ]The coordinate of the intersection point of AD and BD in the virtual world can be calculated as [ (A) _x +B _x +C _x +D _x )/4，(A _y +B _y +C _y +D _y )/4，(A _z +B _z +C _z +D _z )/4]Then, the virtual world coordinate is shifted to the point in place; if the tracking system has no self offset, the offset value is suggested to be used as offset data of all subsequent tracking points;

the device is characterized in that the adjustable objects of a seat (comprising a seat operation button) are the whole position of the seat and a seat back, the tracked position of a cushion is used as the position information of the whole seat in a virtual world, the angle of the backrest rotates the backrest by taking a hinge point of the backrest and the seat as an axis, and the device needs two trackers, wherein the tracker A is used for tracking the cushion, and the tracker B is used for tracking the backrest;

the positional information of the seat is [ A ] _x 、A _y 、A _z ]The rotation of the tracker B around the hinge point is [ B ] _rotX ，B _rotY ，B _rotZ ]In which B is _rotX And B _rotZ Should approach 0;

the indicating panel comprises a combined indicating lamp and an instrument panel;

combination pilot lamp (left indicator, right indicator, parking braking warning light, far-reaching headlamp and power indicator) need not physical structure, adjusts overall position through the tracking, when using the position of tracking point acquisition panel board, should make its long limit be on a parallel with tracker's Y axle, can obtain the position coordinate of combination pilot lamp in the virtual world and be: [ A ] _x ，A _y ，A _z ]In an attitude of [ A _rotX ，A _rotY ，A _rotZ ]A acquired by the tracking system should be adjusted during calibration _rotY Approaching to 0 as much as possible;

the instrument panel (speedometer, tachometer, fuel gauge, water temperature gauge) does not need the physical structure, adjusts the overall position through tracking, when using the tracking point to obtain the position of instrument panel, should make the long limit in instrument panel plane be on a parallel with tracking system's Y axle, can obtain with the position coordinate of instrument panel in the virtual world: [ A ] _x ，A _y ，A _z ]In an attitude of [ A _rotX ，A _rotY ，A _rotZ ]，A _rotX And A _rotZ Should approach 0;

the steering wheel is a replaceable steering wheel and can be adjusted in position and angle, as shown in figure 3, the y coordinate of the point A is used, the x and z coordinates of the point B are aligned with the mass center of the steering wheel ring, namely the position coordinate of the steering wheel in the virtual world is [ B _x ，A _y ，B _z ]In an attitude of [ B _rotX ，A _rotY ，B _rotZ ]；

The turn light switch, the wiper switch and the gear lever can be replaced and the position and the angle can be adjusted;

the device is hinged to the steering wheel integral control device, so that the position information of the steering wheel can be directly used for positioning to obtain accurate positioning, angles in various states only need to be calibrated, and angle information in information obtained by optical or other tracking information with position positioning can be calibrated, and angle information given by an angle sensor can also be used;

the headlight switch, the clearance lamp switch, the brake pedal and the accelerator pedal can be replaced and the position and the angle can be adjusted; the tracking device is installed at the pedal position by using a set tool, tracks two limit positions and is removed after calibration;

the safety belt is replaceable and adjustable in position, and serves as a unique flexible structure of a cabin position, the calibration of the safety belt can be simply simulated as two-point calibration and two-point tracking, two fixed ends of the safety belt calibrate positions, and tracking devices are arranged on a left shoulder stress point and a chest stress point of the safety belt to acquire the state of the safety belt during driving action of a person;

the above device can perform fast calibration through vector points with directions, the specific method depends on the working environment of the whole system, and the implementation scheme is as follows:

the following methods all assume that when an adopted positioning system calibrates the virtual and physical world spaces, the origin (X is 0, Y is 0, and Z is 0) point of a coordinate system can be shifted and rotated according to a certain rule, if the original positioning system does not include a related method, the coordinate system needs to be converted, and a method for calculating coordinate shift can be used as a beneficial supplement of the method;

optical tracking marker point scheme: the core point of the scheme is that an optical motion capture system is used for tracking a specifically designed rigid body so as to complete the alignment of the whole system and the tracking of operation;

LightHouse protocol: the core point of the scheme is that a specially optimized head-mounted virtual reality display device is customized and developed for a Vive system by using HTC, and the self-consistency of the system is high;

sensor angle calculation scheme: the core point of the scheme is that various sensor devices are utilized to track the operation of the whole system;

the multi-technology mixing scheme comprises the following steps: the multi-technology mixing is the scheme with the most cost performance and the most development potential in the current technical level;

in fact, according to the characteristics of the object and the characteristics of the operation, the best experience effect can be achieved by using hybrid tracking;

for example, a Tracker of an HTC five system is used as a positioning point tool to position a virtual cabin platform to match the relationship between physical coordinates and virtual coordinates of the whole system, the Tracker is used for providing the position and the direction of a palm, a plurality of leapmotions are combined to cover an operation area to track gestures, a power-on switch is used for simulating a headlight switch and a wiper switch, and an angle sensor is used for tracking a pedal;

although the technology of inertial sensor hybrid optical or laser positioning device positioning can be involved in the implementation case, the innovation point of the method does not relate to the specific calculation method thereof, and only the product of the method or the self-calculation result of the customized device is used as a providing source of the positioning information;

according to the steps required by calibration positioning, the method can be divided into one-point positioning, two-point positioning and multi-point positioning to carry out the rapid calibration of the virtual world and the physical world:

one-point positioning is used for an operation object which only needs to confirm the position, does not change the direction, or does not influence the experience by the direction change: the device comprises an instrument panel, a steering wheel, a brake pedal, an accelerator pedal, a parking brake, a combined indicator light and an indication panel; one point positioning only takes the position information of the tracking device, and the position and the direction in the virtual world can be accurately positioned by taking the coordinate direction of the virtual cabin basic platform as reference;

two-point positioning is used for operation objects that require confirmation of position and rotation, change in direction, and affect experience: the virtual cockpit base platform, a turn light switch, a gear lever, a headlight switch, an outline marker light switch and a wiper switch; the two-point positioning comprises the position and direction information of the tracking device, and can track the dynamic change of the object

Multipoint positioning is used for calibration of the entire virtual cockpit base platform, and the object itself may change relatively and affect the experienced operation object: seats, safety belts; the multipoint positioning comprises position and direction information, at least comprises a group of positioning components of three tracking devices, and can define rigid body state change;

the point in the calibration positioning refers to a vector with a position and a direction in a space, and is not a point with only position information in a three-dimensional space, and a specific acquisition method has a difference according to a used positioning technical scheme;

the three calibration methods correspond to devices with multiple possibilities (HTCvive handle, tracker with laser emitter, optical rigid body with electronic compass);

or a level meter (bubble type or electronic type) is matched, and a coordinate system for contrasting the physical world and the virtual world is generated through the device;

the method can greatly reduce the difficulty of calibration between the physical object and the virtual object in the system under the condition of ensuring that the relative error precision of the physical operation object and the virtual object is acceptable;

calibration operations to achieve the best results, one should establish 1: 1, a model;

for devices that require only one-time adjustment (e.g., cabin base platform, electronic dashboard, seat), only track their position and issue a warning when the positional offset exceeds a threshold, asking the user to confirm whether the entire system needs to be recalibrated;

for devices with assistance (such as a headlight switch, a windshield wiper and high-low light control), only the trigger of the action is tracked, and an adjustable animation control mode is provided;

for devices requiring precise control (such as an accelerator pedal, a brake pedal, and a steering wheel), the variation thereof is precisely recorded, and calibration is required before the start of use;

or a positioning device is arranged on the palm to track the position and direction of the palm, and the palm can be matched with a gesture tracking system (such as an infrared optical gesture tracking device LeapMotion, an inertial sensor motion capture glove Hi5, a fiber stretching deflection sensor motion capture glove hand assembly, a resistance bending sensor and a flexible optical fiber sensor) to simulate the motion of the hand and use the sensor system to obtain the technical scheme of operation;

regulating the interaction speed of the physical object operation of the driver and the response of the virtual object in the virtual scene to enable the response of the virtual object in the virtual scene to be synchronously matched with the physical operation state of the driver;

step four, displaying the virtual object in the virtual scene on a display end (a stereo display device or a spectator display) in a real-time visual mode in response to the motion, so that the physical operation of a driver in the virtual cabin is visually visible and tactually real;

in consideration of cost and immersion, the scheme is mainly directed to a cabin system using a VR helmet for driving simulation, and can be compatible with a use scene of surrounding stereo projection or vehicle window screen transformation;

the method recommends using a game engine to perform dynamic visual calculation, and can also be compatible with an industrial visualization engine and other engines through a VRPN protocol;

combination of virtual vision and physical haptics: in the scheme, a cabin simulation device with replaceable parts and dynamically adjusted layout is recommended to be used, and the virtual driving simulation states of real vehicle modification, six-axis seat platforms and the like are compatible;

although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims

1. A virtual and real interaction method for a virtual cockpit is characterized by comprising the following steps:

the physical object constitution of the virtual cockpit comprises a virtual cockpit base platform, the position and the angle of the virtual cockpit base platform are integrally adjusted when the whole cockpit is deployed, and the specific method comprises the following steps:

setting AC as the direction of a head of a virtual cockpit basic platform, using a tracked object with the same structure at four ABCD points with symmetrical positions, or using the same tracked object to sample at four positions respectively to obtain information of the four points, aligning the intersection point of AD and BC in the four points to a coordinate origin of a tracking system, aligning a connecting line of a midpoint of AC and a midpoint of BD to an X axis of the tracking system, pointing the positive direction of the X axis to the midpoint of the AC from the origin, aligning the connecting line of the midpoint of AB and the midpoint of CD to a Y axis, and pointing the positive direction of the Y axis to the midpoint of AB from the origin;

shifting points obtained by the physical position of the virtual cabin basic platform in the tracking system according to a virtual world coordinate system; through the obtained virtual world coordinates [ Ax, Ay, Az ], [ Bx, By, Bz ], [ Cx, Cy, Cz ], [ Dx, Dy, Dz ] of the four points in the tracking system, the coordinate of the intersection point of AD and BC in the virtual world can be calculated to be [ (Ax + Bx + Cx + Dx)/4, (Ay + By + Dy)/4, (Az + Bz + Cz + Dz)/4], and then the origin of the virtual world coordinates is shifted to the point;

2. The virtual cockpit virtual-real interaction method of claim 1, wherein said first step: on the basis that the calibration of the tracking system is completed, the whole physical object of the virtual cabin is in one-to-one correspondence with the virtual object in the coordinate system of the virtual world corresponding to the tracking system.

3. The virtual cockpit virtual-real interaction method of claim 2, wherein said first step: the rapid calibration of the virtual world and the physical world comprises the following steps: one-point positioning, two-point positioning, and multi-point positioning.

4. The virtual cockpit virtual-real interaction method of claim 3, wherein said one-point positioning is used for an operation object that only needs to confirm a position, does not change a direction, or does not affect an experience: the device comprises an instrument panel, a steering wheel, a brake pedal, an accelerator pedal, a parking brake, a combined indicator light and an indication panel.

5. The virtual cockpit virtual-real interaction method of claim 3, wherein the two-point positioning is used for operation objects which need to confirm position and rotation, change direction and affect experience: virtual cabin basic platform, indicator light switch, gear shift lever, headlight switch, marker light switch, wiper switch.

6. The virtual cockpit virtual-real interaction method of claim 3, wherein the multi-point positioning is used for calibration of the entire virtual cockpit base platform, and the object itself may be relatively changed and may affect the experienced operation object: seats, safety belts.

7. The virtual cockpit virtual-real interaction method of any one of claims 1-6, wherein the method is compatible with a use scene using surround stereo projection or window screening for a cockpit system using VR helmet for driving simulation.

8. The virtual cockpit virtuality and reality interaction method of any one of claims 1-6, wherein the method uses a game engine to perform dynamic visual computation, and is also compatible with an industrial visualization engine and other engines through a VRPN protocol.