CN112289123A - Mixed reality scene generation method and system for automobile driving simulator - Google Patents
Mixed reality scene generation method and system for automobile driving simulator Download PDFInfo
- Publication number
- CN112289123A CN112289123A CN202011211091.5A CN202011211091A CN112289123A CN 112289123 A CN112289123 A CN 112289123A CN 202011211091 A CN202011211091 A CN 202011211091A CN 112289123 A CN112289123 A CN 112289123A
- Authority
- CN
- China
- Prior art keywords
- video
- scene
- automobile
- mixed reality
- simulator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B9/00—Simulators for teaching or training purposes
- G09B9/02—Simulators for teaching or training purposes for teaching control of vehicles or other craft
- G09B9/04—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Aviation & Aerospace Engineering (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- General Physics & Mathematics (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a mixed reality scene generation method and a mixed reality scene generation system for an automobile driving simulator, wherein the mixed reality scene generation method acquires a real-time high-definition video of a driver operation area through a camera arranged at the top of the automobile driving simulator and completes the correction of the video; loading and scheduling a virtual driving scene; registering and fusing the real video and the virtual scene to form a mixed scene of the real video and the virtual scene; and rendering and outputting the mixed scene for the students to watch. The mixed reality scene generation method and the mixed reality scene generation system for the automobile driving simulator can solve the problem of fusion of the real world and the virtual scene, so that a student can observe the actual steering wheel, gear and own operation process during driving training in the virtual scene, the simulator training is closer to the driving training process of an actual vehicle, and the training effect is greatly improved.
Description
Technical Field
The invention relates to the technical field of virtual driving, in particular to a method and a system for generating a mixed reality scene for an automobile driving simulator.
Background
The existing automobile driving simulator usually adopts a display and a projection screen as display equipment, has limited visual angle range and weak immersion feeling, can not realize 360-degree panoramic vision, and influences the driving training effect. The problem of 360-degree field of vision has been solved as the automobile driving simulator of display device with Virtual Reality (VR) glasses. However, after the VR glasses are worn, real world and virtual scenes are split, and students cannot see real objects such as a steering wheel and an instrument desk and cannot see own operation actions, which is an important obstacle of applying VR technology to driving training.
Therefore, how to realize the real-time fusion of the virtual scene and the real operation scene of the automobile driving simulator is a technical problem which needs to be solved urgently.
Disclosure of Invention
In order to solve the above problems, a first aspect of the present invention provides a mixed reality scene generation method for an automobile driving simulator, the mixed reality scene generation method comprising the steps of:
fixing a camera on an automobile simulator, and acquiring internal and external parameters of the camera;
establishing a three-dimensional model VM of the simulated automobile, and designing a mask model PM according to camera parameters;
selecting a training course;
loading a road scene three-dimensional model TM, placing an automobile three-dimensional model VM with a mask model PM in a road scene, and combining to form a virtual driving scene SM;
acquiring a real-time video V of an operation area obtained by a camera, and correcting the video V to obtain a video V';
adjusting the video V 'according to the model parameters in the virtual driving scene SM to obtain a video V';
mapping the video V' to the PM according to the preset vertex video texture coordinates VVC of the mask model PM;
and rendering the SM subjected to video mapping in a left-right parallax mode by adopting a graphic engine according to the output resolution of a head-mounted display (HMD) device, and outputting the SM to the HMD for a student to watch.
Preferably, the camera is fixed at the top of the automobile driving simulator so as to ensure that a steering wheel, an instrument desk, a gear and a hand brake part on the simulator can be completely shot.
Preferably, the mask model PM is a geometric body set, and includes vertex space coordinates VC, normal vectors VN of vertex positions, video texture coordinates VVC of vertices, and topological relations VT between vertices; and the mask model PM is placed in the corresponding position of the automobile three-dimensional model to replace the three-dimensional models of the four parts of a steering wheel, an instrument desk and a gear hand brake.
Preferably, the position and the posture of the three-dimensional model VM in the scene are determined in real time according to a dynamic simulation calculation result.
In a second aspect of the present invention, a mixed reality scene generation system for an automobile driving simulator is provided, the mixed reality scene generation system comprising:
the video acquisition module: and obtaining a real-time high-definition video V of a driver operation area from a camera arranged at the top of the automobile driving simulator, and finishing the correction of the video V.
A scene scheduling module: loading and scheduling the virtual driving scene.
A video fusion module: and registering and fusing the video V and the virtual scene S according to preset parameters to form a mixed scene MM of the actual video and the virtual scene.
A rendering output module: the hybrid scene MM is rendered and output for viewing by a student according to technical parameters of a Head Mounted Display (HMD) device.
The invention has the beneficial effects that: by the aid of the method and the system for generating the mixed reality scene for the automobile driving simulator, real-time mixing of the live video and the virtual scene picture can be achieved, the problem of fusion of the real world and the virtual scene is solved, a student can observe an actual steering wheel, gear and own operation process during driving training in the virtual scene, training of the simulator is closer to the driving training process of an actual vehicle, and training effect is improved to a great extent.
Drawings
FIG. 1 is a schematic structural diagram of a mixed reality scene generation method for an automobile driving simulator according to the present invention;
FIG. 2 is a schematic diagram of a video modification method provided by the present invention;
fig. 3 is a schematic structural diagram of a driving simulator of an automobile according to the present invention.
Detailed Description
For the purpose of describing the embodiments of the present invention in detail, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention mainly aims to provide a method and a system for generating a mixed reality scene for an automobile driving simulator, and aims to solve the technical problems that after VR glasses are worn by the existing automobile driving simulator, the real world and the virtual scene are split, and a student cannot see real objects such as a steering wheel, an instrument desk and the like and cannot see own operation actions.
In order to achieve the purpose, the invention provides a mixed reality scene generation method and a mixed reality scene generation system for an automobile driving simulator, wherein the mixed reality scene generation method acquires a real-time high-definition video of a driver operation area through a camera arranged at the top of the automobile driving simulator and completes the correction of the video; loading and scheduling a virtual driving scene; registering and fusing the real video and the virtual scene to form a mixed scene of the real video and the virtual scene; and rendering and outputting the mixed scene for the students to watch. The mixed reality scene generation method and the mixed reality scene generation system for the automobile driving simulator can solve the problem of fusion of the real world and the virtual scene, so that a student can observe the actual steering wheel, gear and own operation process during driving training in the virtual scene, the simulator training is closer to the driving training process of an actual vehicle, and the training effect is greatly improved.
The invention provides an embodiment, as shown in fig. 1, fig. 1 is a schematic structural diagram of a mixed reality scene generation method for an automobile driving simulator provided by the invention, and the method comprises video acquisition, scene scheduling, video fusion and rendering output, and comprises the following specific steps:
furthermore, a camera is fixed on the automobile simulator, so that the camera can shoot a steering wheel, an instrument desk, a gear and a hand brake part on a simulator real object from the top; and calibrating the camera to obtain the internal and external parameters of the camera.
Further, establishing a three-dimensional model VM of the simulated automobile, wherein the three-dimensional model VM comprises elements such as the internal structure and the appearance of the automobile; designing a mask model PM according to the internal and external parameters of the camera obtained in the step 1, wherein the mask model PM is a simple geometric body set, and the main data of the mask model PM comprises: vertex space coordinates VC, a normal vector VN of a vertex position, video texture coordinates VVC of the vertex, topological relation VT between the vertices and the like. The mask model is put into the corresponding position of the automobile three-dimensional model to replace the three-dimensional models of four parts, namely a steering wheel, an instrument desk, a gear and a hand brake.
Further, a training course is selected, and simulated driving training is started.
And further, loading a corresponding road scene three-dimensional model TM according to the training course, and putting the automobile three-dimensional model VM with the mask model PM established in the step 2 into the road scene. And determining the position and the posture of the VM in the scene in real time according to the dynamic simulation calculation result. The TM and VM combine to form a virtual driving scenario SM.
Further, acquiring a real-time video V of an operation area obtained by a camera, mapping the video V onto a mask model PM, and correcting the video V to obtain a video V';
further, adjusting the video V 'according to the illumination model parameters such as the environment light intensity in the virtual driving scene SM to obtain a video V';
further, mapping the video V' to the PM according to preset vertex video texture coordinates VVC of the mask model PM;
further, rendering the SM subjected to video mapping in a left-right parallax mode by using a graphic engine according to the output resolution of a head-mounted display (HMD) device, and outputting the SM to the HMD for a student to watch.
Another embodiment of the present invention provides a video correction method, as shown in fig. 2, fig. 2 is a schematic view of the video correction method provided by the present invention.
Specifically, an arbitrary point P after correction is obtained1(u, v) original image coordinates P corresponding to the original image selection area (quadrilateral ABCD)0。
Further, firstly, the method
And
respectively obtain 1/u position points Q0And Q1Then on to
The 1/v position point is obtained as P0。
Further, in FIG. 2, a point P after the correction is known1Coordinates (u, v), and its corresponding point P before correction0The coordinates in the frame are:
as shown in fig. 3, fig. 3 is a schematic structural diagram of the driving simulator of the vehicle according to the present invention, which includes a bottom frame (1), a vehicle body (2), a housing (3), a driver seat (4), a safety belt and a buckle (5), an electric cylinder and a hinge mechanism (6), an electric cylinder controller (7), an instrument console (8), an instrument (9), an electric power feedback steering wheel and controller (10), a pedal assembly (11), a head-mounted display device (12), a gear and hand brake box (13), a gear shift gear (14), a hand brake (15), a data acquisition controller (16), a device mounting frame (17), a camera (18), a computer host (19), and a touch display screen (20).
Furthermore, the bottom frame adopts a metal structure and is a supporting body of the whole equipment on the ground. The bottom of underframe is installed detachable wheel, the transportation of being convenient for.
Furthermore, the car body adopts metal frameworks such as steel pipes or aluminum profiles and the like as a main body structure, a floor made of non-metal flame-retardant materials is laid, and a mounting frame of the shell is provided.
Furthermore, the two electric cylinders are installed and connected with the vehicle body and the bottom frame through a hinge, a universal joint and other mechanisms, so that the vehicle body can do posture-controlled pitching and rolling two-degree-of-freedom motion under the condition that the bottom frame is not moved. The driver seat, the instrument center console, the pedal combination, the gear and the hand brake box are all arranged on the vehicle body structure, and the relative position relation of the driver seat, the instrument center console, the pedal combination, the gear and the hand brake box is consistent with that of a real vehicle.
Furthermore, the electrodynamic force feedback steering wheel, the steering wheel pipe column and the combination instrument are arranged in the center of the instrument desk, and the positions of the electrodynamic force feedback steering wheel, the steering wheel pipe column and the combination instrument are consistent with those of a real vehicle. The gear shift and the hand brake are arranged on the gear shift and the hand brake box.
Furthermore, sensors are arranged on the steering wheel, the combination instrument, the speed change gear, the hand brake, the pedal combination and the safety belt buckle, and the sensors are connected into the data acquisition controller. The equipment mounting frame is fixed on the vehicle body and synchronously moves with the vehicle body.
Further, the camera is installed on the equipment fixing frame, and its installation and angular position guarantee that the camera can shoot steering wheel, gear and manual brake.
Furthermore, the computer host is arranged on the mounting seat of the bottom frame and hidden at the lower part of the vehicle body.
Further, the computer host is connected with the data acquisition controller, the head-mounted display device, the touch display screen, the camera and the electric cylinder controller.
Further, the touch display screen is installed on the equipment installation frame and used for equipment management. When the teaching aid is used, a trainee wears the head-mounted display equipment, and when the teaching aid is not used, the head-mounted display equipment is hung on the equipment mounting frame.
It should be noted that the above-mentioned preferred embodiments are only illustrative and should not be construed as limiting the scope of the invention, and that modifications and variations can be made by those skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (6)
1. A mixed reality scene generation method for an automobile driving simulator is characterized by comprising the following steps:
fixing a camera on an automobile simulator, and acquiring internal and external parameters of the camera;
establishing a three-dimensional model VM of the simulated automobile, and designing a mask model PM according to camera parameters;
selecting a training course;
loading a road scene three-dimensional model TM, placing an automobile three-dimensional model VM with a mask model PM in a road scene, and combining to form a virtual driving scene SM;
acquiring a real-time video V of an operation area obtained by a camera, and correcting the video V to obtain a video V';
adjusting the video V 'according to the model parameters in the virtual driving scene SM to obtain a video V';
mapping the video V' to the PM according to the preset vertex video texture coordinates VVC of the mask model PM;
and rendering the SM subjected to video mapping in a left-right parallax mode by adopting a graphic engine according to the output resolution of the head-mounted display device HMD, and outputting the SM to the HMD for a student to watch.
2. The method as claimed in claim 1, wherein the camera is fixed at a top position of the driving simulator to ensure complete shooting of the steering wheel, the instrument desk, the gear and the hand brake part of the simulator.
3. The mixed reality scene generation method for the automobile driving simulator as claimed in claim 1, wherein the mask model PM is a set of geometric objects, including vertex space coordinates VC, normal vectors VN of vertex positions, video texture coordinates VVC of vertices, topological relations VT between vertices; the mask model PM is placed in the corresponding position of the automobile three-dimensional model to replace the three-dimensional models of the steering wheel, the instrument desk and the gear hand brake.
4. The mixed reality scene generation method for the automobile driving simulator as recited in claim 1, wherein the position and the posture of the automobile three-dimensional model VM in the scene are determined in real time according to the dynamic simulation calculation result.
5. The method as claimed in claim 1, wherein the video modification method is to find any point P after correction1(u, v) corresponding original image coordinate P in the original image selected quadrilateral ABCD area0(ii) a Firstly, firstly
And
respectively obtain 1/u position points Q0And Q1Then on to
Get 1/v position pointIs P0(ii) a Some point after the known correction
Coordinates (u, v), and its corresponding point P before correction0The coordinates in the frame are:
6. a mixed reality scene generation system for an automotive driving simulator, the mixed reality scene generation system comprising:
the video acquisition module: obtaining a real-time high-definition video V of a driver operation area from a camera arranged at the top of the automobile driving simulator, and finishing the correction of the video V;
a scene scheduling module: loading and scheduling a virtual driving scene;
a video fusion module: registering and fusing the video V and the virtual scene S according to preset parameters to form a mixed scene MM of the actual video and the virtual scene;
a rendering output module: and rendering and outputting the mixed scene MM according to the technical parameters of the head-mounted display device HMD for the students to watch.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011211091.5A CN112289123A (en) | 2020-11-03 | 2020-11-03 | Mixed reality scene generation method and system for automobile driving simulator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011211091.5A CN112289123A (en) | 2020-11-03 | 2020-11-03 | Mixed reality scene generation method and system for automobile driving simulator |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112289123A true CN112289123A (en) | 2021-01-29 |
Family
ID=74350440
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011211091.5A Pending CN112289123A (en) | 2020-11-03 | 2020-11-03 | Mixed reality scene generation method and system for automobile driving simulator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112289123A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113419632A (en) * | 2021-07-06 | 2021-09-21 | 广州市旗鱼软件科技有限公司 | Mixed reality simulation driving scene display method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102903092A (en) * | 2012-09-07 | 2013-01-30 | 珠海一多监测科技有限公司 | Four-point transformation based image adaptive-correction method |
| CN107134194A (en) * | 2017-05-18 | 2017-09-05 | 河北中科恒运软件科技股份有限公司 | Immersion vehicle simulator |
| CN111696216A (en) * | 2020-06-16 | 2020-09-22 | 浙江大华技术股份有限公司 | Three-dimensional augmented reality panorama fusion method and system |
| CN111766940A (en) * | 2020-05-14 | 2020-10-13 | 四川科华天府科技有限公司 | Wearable interactive teaching virtual reality fuses system for equipment |
-
2020
- 2020-11-03 CN CN202011211091.5A patent/CN112289123A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102903092A (en) * | 2012-09-07 | 2013-01-30 | 珠海一多监测科技有限公司 | Four-point transformation based image adaptive-correction method |
| CN107134194A (en) * | 2017-05-18 | 2017-09-05 | 河北中科恒运软件科技股份有限公司 | Immersion vehicle simulator |
| CN111766940A (en) * | 2020-05-14 | 2020-10-13 | 四川科华天府科技有限公司 | Wearable interactive teaching virtual reality fuses system for equipment |
| CN111696216A (en) * | 2020-06-16 | 2020-09-22 | 浙江大华技术股份有限公司 | Three-dimensional augmented reality panorama fusion method and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113419632A (en) * | 2021-07-06 | 2021-09-21 | 广州市旗鱼软件科技有限公司 | Mixed reality simulation driving scene display method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110097799B (en) | Virtual driving system based on real scene modeling | |
| US20230226445A1 (en) | Reality vs virtual reality racing | |
| US5796991A (en) | Image synthesis and display apparatus and simulation system using same | |
| JPH05174129A (en) | Modeling apparatus for imaging three-dimensional model | |
| CN103050028B (en) | Driving simulator with stereoscopic vision follow-up function | |
| CN110109552A (en) | Virtual driving scene modeling method based on true environment | |
| CN103718226A (en) | Device for spatially moving persons | |
| CN112150885B (en) | Cockpit system based on mixed reality and scene construction method | |
| CN107134194A (en) | Immersion vehicle simulator | |
| CN108735052A (en) | A kind of augmented reality experiment with falling objects method based on SLAM | |
| CN107154197A (en) | Immersion flight simulator | |
| CN111062869B (en) | Multi-channel correction splicing method for curved curtain | |
| EP0715280B1 (en) | Method and apparatus for processing image | |
| KR20120044461A (en) | The simulated training apparatus and method using mixed reality | |
| CN207883156U (en) | A kind of scenic spot simulated flight experience apparatus | |
| CN112289123A (en) | Mixed reality scene generation method and system for automobile driving simulator | |
| CN211124495U (en) | Driving emergency disposal training device based on MR | |
| CN214504686U (en) | Automobile driving simulator based on mixed reality technology | |
| Segura et al. | Interaction and ergonomics issues in the development of a mixed reality construction machinery simulator for safety training | |
| CN109427094B (en) | Method and system for acquiring mixed reality scene | |
| CN113516733B (en) | Method and system for filling blind areas at bottom of vehicle | |
| CN108983963A (en) | A kind of vehicle virtual reality system method for establishing model and system | |
| JP7318265B2 (en) | Image generation device and image generation method | |
| CN115311133A (en) | Image processing method and device, electronic equipment and storage medium | |
| JP6503098B1 (en) | Image processing apparatus, image processing program and image processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |