EP1913764A2

EP1913764A2 - Device for visualising a virtual passenger compartment of a vehicle in a real situation

Info

Publication number: EP1913764A2
Application number: EP06794315A
Authority: EP
Inventors: Valentin Lefevre; Jean-Marie Vaidie
Original assignee: Total Immersion
Current assignee: Qualcomm Inc
Priority date: 2005-08-09
Filing date: 2006-08-09
Publication date: 2008-04-23
Also published as: WO2007017596A3; FR2889753A1; WO2007017596A2

Abstract

The invention relates to a device for visualising a virtual passenger compartment of a vehicle. Said device comprises mobile video acquisition and visualisation means, at least one visualisation means, at least one video capturing means for capturing the field of vision, a movement capturing device, and at least one processing means which is connected to said mobile video acquisition and visualisation means and adapted in such a way as to receive a video of at least one video capturing means, to insert the virtual passenger compartment respectively into the video received, according to the information received from the movement capturing device, and to transmit the received video into which the virtual passenger compartment has been inserted, to the at least one visualisation means, in real-time. The inventive device also comprises synchronisation means arranged between the at least one processing means and the mobile video acquisition and visualisation means.

Description

DEVICE FOR VISUALIZING A VIRTUAL COMPARTMENT OF A VEHICLE IN A REAL SITUATION

The present invention relates to the testing and validation of the ergonomics of driving environments such as vehicle cockpits and more particularly a method and a device for testing and validating the ergonomics of a virtual passenger compartment in a real situation.

It finds a general application in visualizing a virtual cockpit of a vehicle in the real world and the possibility of driving the vehicle.

Thus, the invention more precisely relates to a viewing device in which a virtual cockpit model is inserted in real time in a video captured itself in real time.

To limit development costs, the test and validation of the ergonomics of a cockpit of a vehicle have evolved from the real world to a virtual world. The testing and validation of a real cockpit requires the manufacture of this cockpit and its mounting within the vehicle for which it is intended. In addition to the time and cost required for this solution, the cockpit can only be tested in a limited number of situations, due in particular to security issues. The use of a virtual model offers many advantages. In particular, each modification does not require the realization of a new model, reducing design time and cost. The testing and validation of a virtual cockpit is often done through the use of video projectors and screens forming an immersive room. Simulation of the use of the virtual cockpit is simulated in a pure synthesis image environment. In such a configuration, the user is seated on a mobile platform that is driven by electric or hydraulic cylinders, also called cabin movement. Although the computer generated image to test and validate the cockpit is as close to reality as possible, the actual interface of the tester with the cockpit is not reproduced. Indeed, on the one hand, the external environment, that is to say the decor, is in synthetic images so it is of a rough realism, the modeling of decors being long to achieve. On the other hand, the cabin movement is an approximate movement. Indeed, the mobile platform operates at low frequency, namely about 4 Hz, because of the slowness of the cylinders. In addition, a reaction time is introduced of about 300 milliseconds or more. In addition, an acceleration of the cabin is limited in time because the movement of the platform arrives very quickly abut. Finally, the dynamic equations of motion are approximate because of the real-time calculation. Thus, such a solution is particularly lacking realism of movement. There is therefore a need to optimize the testing and validation of the ergonomics of driving environments for vehicles.

The invention solves at least one of the problems discussed above.

The invention thus relates to a device for visualizing a virtual passenger compartment of a vehicle, characterized in that it comprises:

mobile video acquisition and display means including:

at least one visualization means;

at least one video capture means capable of capturing the field of view; and,

a motion capture device;

at least one processing means connected to said mobile video acquisition and display means, said at least one processing means being adapted to,

- receive a video of at least one video capture means; and,

inserting the virtual passenger compartment respectively into said received video, according to the information received from said motion capture device; and,

transmitting in real time to said at least one visualization means the received video in which the virtual passenger compartment has been inserted,

synchronization means between said at least one processing means and said mobile video acquisition and display means. The device according to the invention makes it possible to test and validate the ergonomics of a virtual passenger compartment, especially in a real driving situation.

Indeed, by means of this device, a pilot is able to drive a real vehicle whose cockpit is virtual while the pilot leads in a real context. Thus the pilot is immersed in a virtual cockpit within a real vehicle.

The actual vehicle may be a simplified vehicle formed for example, only the chassis, the engine, the steering wheel and a plexiglass bubble. Such a vehicle makes it possible to have a real cabin as transparent as possible, then to visualize a virtual cockpit without this cockpit being constrained by elements of the real cockpit.

In addition, according to this device, it is possible to obtain in real time the view of the actual driving context (including the road), that is to say, a real scenery and virtual cockpit.

This is made possible in particular by the synchronization means between the processing means and the mobile video acquisition and display means. Indeed, these means make it possible to optimize the transmission time of the images and to remove the artifacts during the movements of the user.

It should also be noted that this device makes it possible to test a passenger compartment in real driving conditions of the vehicle.

According to a particular embodiment, the virtual cockpit is a virtual cockpit of a previously modeled vehicle.

In a particular embodiment, the scanning frequency of said at least one video capture means is substantially equal to the scanning frequency of said at least one viewing means.

According to a particular characteristic, said at least one visualization means is a virtual headset.

In another particular embodiment, the mobile video acquisition and display means comprise,

at least one visualization means;

at least two video capture means; and, a motion capture device; the device further comprising at least two processing means connected to said mobile video acquisition and display means, each of said processing means being adapted to,

receiving a video of at least one of said video capture means; and,

synchronization means between said processing means and said mobile video acquisition and display means.

Other advantages, aims and features of the present invention will emerge from the detailed description which follows, given by way of non-limiting example, with reference to the accompanying drawings in which:

- Figure 1 shows schematically the test environment and validation of the ergonomics of a virtual cabin;

FIG. 2 illustrates the virtual reality helmet worn by the user during the test and validation of the ergonomics of a virtual passenger compartment;

- Figure 3 schematically illustrates the device of the present invention.

The device according to the invention makes it possible to mix in real time the images of a virtual cockpit with a real video so as to test and validate new cockpits in a real situation.

To do this, a driver is installed in a real vehicle. This can for safety reasons be assisted by a co-pilot who is also able to control the vehicle. Thus, the co-pilot has the same control means as the pilot, including means to brake, disengage, accelerate. According to the invention, the pilot has a device comprising mobile video acquisition and visualization means, in particular, in the form of a virtual reality helmet also called headmachine (HMD or "Head Mounted Display" in terminology Anglo-Saxon). These means are equipped, for example, with at least one visualization means, at least one video capture means, in particular a camera, capable of capturing the pilot's field of vision and a motion capture device rigidly linked to the (x ) video capture medium (s).

By means of this device, the pilot is able to visualize, in real time, a virtual cockpit, in particular a cockpit. To do this, at the initialization of the device, a virtual cockpit is chosen from a plurality.

This virtual cockpit can also consist of virtual enrichments animated by a virtual or real cabin, it can include indicators of speed, gauges and navigation system.

The exterior of the passenger compartment is the real world coming from the capture means of the pilot's field of vision.

Thus, in accordance with the invention, it is possible to test and validate the ergonomics of a virtual cockpit or an enrichment of a real or virtual cockpit in a real driving situation without actually realizing this new cabin.

In addition, the pilot is totally immersed in a virtual cockpit.

In accordance with the invention, the pilot has in real time a real image of the context in which he is piloting, notably via the means of capturing his field of vision and visualization means rendering this field of view captured.

Figure 1 schematically shows the environment for testing and validation of the ergonomics of a virtual passenger compartment in a real driving context.

The driver 12 in charge of performing the test and validation is installed in the real vehicle 14. The vehicle is for example a real car capable of being driven on a road. The pilot 12 is wearing a virtual reality helmet 16 equipped with a display screen, two cameras capable of capturing the pilot's field of vision and a motion capture device rigidly linked to the cameras, for example a branded sensor. "Laser Bird".

At the rear of the vehicle, computer and electronic equipment is embarked.

FIG. 2 illustrates the virtual reality helmet 16 worn on the head 20 of the pilot 12. The virtual reality helmet 16 comprises a motion capture device 22, or motion sensor, making it possible to determine in real time the position and the orientation of the virtual reality headset 16. The motion sensor may be, for example, of the "Laser Bird" type. According to this example, the motion sensor comprises a movable part 22 and a fixed part (not shown), the measured movement being the relative movement of the movable part with respect to the fixed part.

The virtual reality headset 16 may be monoscopic or stereoscopic. If the virtual reality headset 16 is monoscopic, it includes a single camera. If it is stereoscopic, it includes two cameras, one associated with the left eye and the other associated with the right eye, in order to restore the reliefs to the pilot.

The example of virtual reality headset 16 shown in FIG. 2 is stereoscopic and comprises two cameras 24 and 25.

The moving part of the motion capture device is integral with the cameras so that the displayed summary images are well calibrated with respect to the video images, the computer images being a virtual passenger compartment.

The virtual reality headset also comprises two display screens 26 and 27 for viewing images transmitted from a computer via a link 28, wired or not.

A display screen is placed in front of the right eye of the user (screen 26), the other being placed in front of his left eye (screen 27). The connections to the display screens and the display screens may be, for example, of the Super eXtended Graphics Array (SXGA) type. The principle, according to the invention, is to place a virtual cockpit, for example a virtual cockpit, in the real vehicle driven by the pilot.

Figure 3 schematically illustrates the device of the present invention. This device comprises the virtual reality headset 16, the motion sensor having a fixed portion 22b and a movable portion 22a integral with the cameras 24 and 25 attached to the virtual reality headset and two computers

31 and 32 of the individual computer type (or Personal Computer, PC).

One of the computers is associated with one of the virtual reality headset cameras while the other computer is associated with the other camera. Thus, the computer connected to the camera associated with the left eye manages the image to be presented to the left eye, the other manages the image to be presented to the right eye. If the virtual reality headset includes only one camera, only one computer is needed to manage the image that will be presented to both eyes.

The computers 31 and 32 each comprise calculation means and input and output peripherals. Preferably, calculators 31 and

32 each include,

a video input (311 and 321) connected to one of the cameras (24 and 25) of the virtual reality headset 16;

a video output (312 and 322), for example of the SXGA type, connected to one of the display screens (26 and 27) of the virtual reality headset 16;

- A connection (313 and 323) to the other computer, for example an Ethernet port, and the computers are interconnected by a network cable; and,

a synchronization connection (314 and 324), for example of the Genlock type, connected to the cameras (24 and 25) of the virtual reality headset 16 and to the synchronization connection of the other computer.

In addition, one of the computers 31 or 32 preferably comprises an input 315, for example an RS232 port or a USB port, connected to the motion capture device 22b for the position and orientation of the cameras 24 and 25. This information may be transmitted to the second computer via the connection between the two computers, for example via Ethernet.

Each computer is responsible for displaying in real time the video image from the corresponding camera, mixed with the summary images, on the corresponding display screen. Preferably, the images generated by the computers are at the resolution of the virtual reality headset 16.

It should be noted that the co-pilot may also have an LCD screen to visualize the view of the left eye and the right eye of the pilot.

To do this, the hardware architecture described in FIG. 3 adds a VGA type splitter for each PC and a switch ("switch" in English terminology) in order to be able to display on the screen either the view of the left eye is the view of the right eye.

By way of illustration, each computer may be a personal computer having the following characteristics,

- Pentium IV processor, 3GHz;

- 1 GB RAM or RAM and 80 GB hard drive;

- Microsoft Windows XP Professional operating system;

- nVidia GeForceFX Quadro 4400G card;

- Genlock synchronization card (G-SYNC); and,

- PAL or NTSC video capture card that uses the PCI bus, or that is connected to a USB2 type port, or a firewire port, or a cameralink port.

In one embodiment, the virtual reality headset imposes the refresh rate, for example 60 Hz. The video output of the computers must have a scanning frequency compatible with that of the virtual reality headset.

Ideally, video cameras have a scanning frequency close to that of the virtual reality headset for example 50Hz in Europe and 60Hz in the United States. Ideally, the video cameras and the video outputs of the computers are "genlockable" to synchronize the video cameras and the video outputs of the computers. The synchronization of the video cameras and the video outputs of the computers makes it possible to optimize the transmission time of the images (transport delay) and to obtain a stereoscopic restitution without artefact, even during the movements of the user.

The cameras can be SD resolution, for example Sony XC555 NTSC cameras with a Genlock sync input. Cameras can also be progressive (non-interlaced) with a USB2, firewire or cameralink output, for example a UEYE 2220C camera. The field of view should be as close as possible to the field of view of the virtual reality headset for each eye, for example a 3.5mm NF-mount lens with a field of view greater than 100 degrees and a VT sensor. thumb.

The cameras used can also be HD resolution, using HD-SDI or cameralink output. The video capture cards can have an HD-SDI input, for example on PCI bus, PCI-X or PCI-Express, such as the Decklink HD card, or a cameralink input. It is also possible to acquire non-interlaced signals via the USB2 and cameralink standards, for example.

The mixing of the images from the video cameras with the synthetic images representing the virtual passenger compartment is preferably carried out by the FUSION software of the company TOTAL IMMERSION designed for augmented reality applications. In a particular embodiment, each computer is equipped with a runtime version of the FUSION technology having the following features, at the launch of the software, FUSION loads the plurality of virtual driving environments that can be tested when the driving session; acquisition and display of real-time video streams with high performance; real-time processing of video streams to correct the radial optical distortions of video cameras, thus allowing the video from the cameras to be perfectly matched with the virtual cockpit; de-interlacing video images before the plot in the rendering loop if the cameras are interlaced; real-time processing of motion information (eg the "motion capture" flow from the "Laser Bird" sensor) by extrapolations and interpolations to dynamically synchronize video streams from cameras and computer graphics the virtual cockpit, by recalculating the positions and orientations of the motion sensor on the dates of reception of the video images; real-time calculation of the extrinsic parameters, that is to say the positions and orientations with respect to an absolute reference, of the two video cameras for real-time matching the images from the cameras with the virtual passenger compartment, even when the user moves the position and orientation of his head; choose to display a virtual cockpit among the plurality of N virtual compartments loaded in memory; "offline" calibration of the system, o calibration of the radial optical distortions of the two cameras (intrinsic parameters) by image analysis (distortion patterns) and o calibration of the extrinsic parameters of the cameras in the movement sensor mark according to the so-called method of camera pose extraction using a location tool.

The following table summarizes the performance of the solution according to the invention in the case of a virtual reality headset having a scanning frequency of 60 Hz and SXGA video inputs,

The invention makes it possible to validate the design and ergonomics of a passenger compartment by visualizing a new environment in a real driving context, without constraint of angle of view (the pilot can move his head according to the six degrees of freedom corresponding to changes of position and orientation).

Naturally, to meet specific needs, a person skilled in the field of image processing and augmented reality may apply modifications in the foregoing description to meet his needs.

Claims

1. Device for visualizing a virtual passenger compartment of a vehicle, characterized in that it comprises:

mobile video acquisition and display means including:

at least one visualization means;

at least one video capture means capable of capturing the field of view; and,

a motion capture device;

- receive a video of at least one video capture means; and,

synchronization means between said at least one processing means and said mobile video acquisition and display means.

2. Display device according to claim 1, characterized in that a virtual cockpit is a virtual cockpit of a previously modeled vehicle.

3. Display device according to claim 1 or claim 2, characterized in that the scanning frequency of said at least one video capture means is substantially equal to the scanning frequency of said at least one display means.

4. Display device according to any one of the preceding claims, characterized in that said at least one visualization means is a virtual headset.

5. Viewing device according to any one of the preceding claims, characterized in that the mobile video acquisition and display means comprise,

at least one visualization means;

at least two video capture means; and,

a motion capture device; the device further comprising at least two processing means connected to said mobile video acquisition and display means, each of said processing means being adapted to,

receiving a video of at least one of said video capture means; and,