WO2006016295A1

WO2006016295A1 - Transport device simulator

Info

Publication number: WO2006016295A1
Application number: PCT/IB2005/052469
Authority: WO
Inventors: Daniel W. E. Schobben; Bartel M. Van De Sluis; Ronaldus M. Aarts; Evert J. Van Loenen; Martin Ouwerkerk; Natasha Kravtsova
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2004-08-04
Filing date: 2005-07-22
Publication date: 2006-02-16

Abstract

A transport device (10) comprises a video camera (16) for recording an environment of an actual route traversed by the transport device (10) and means (17) for, while capturing the images, determining actual positions of the transport device (10). Each actual position corresponds to one of the captured images. The transport device (10) further comprises a user interface (15), a transport device simulator (11) and a display (18) for providing a simulation of a route of a simulated transport device to a user. The transport device simulator (11) comprises a processor (13) for constructing the simulation based on the captured images, the corresponding actual positions and user commands from the user interface (15). The simulator (11) enables the user to interactively interfere in a simulation of a route actually driven by the real transport device.

Description

Transport device simulator

The invention relates to a transport device comprising a transport device simulator for providing a simulation of a route of a simulated transport device, a user interface for enabling a user to control the simulated transport device and a display for providing the simulation to the user. The invention further relates to a transport device simulator for providing a simulation of a route of a simulated transport device.

The invention further relates to a method for providing a simulation of a route of a simulated transport device.

The invention further relates to a computer program product. In the car, children are often bored and hinder the driver directly or indirectly by bothering other children. They often have an interest in pretending to drive the car. However, letting them drive the car is intrinsically impossible as this would lead to unsafe situations. Some modern cars have displays integrated in the headrests of the front chairs and a DVD player/ game computer on board for enabling the children to play video games while travelling to entertain the children. One of the possible games for the children to play is a driving simulator. With a driving simulator the children's desire to actually drive a car can be met.

A video recorder car for a car simulator is known from the German patent DE 4105963. The car simulator described therein uses video images, recorded by one or more video cameras mounted on the video recorder car. The video images are stored for later use by the car simulator. The car simulator uses the recorded images to provide a user a realistic environment to drive the simulated car through. The video recorder car may use a plurality of cameras to enable the user to drive the simulated car over the whole width of the road. Additionally the viewpoint of the cameras may be changed during the video recording and/or the video recorder car may drive various routes for enlarging the environment the simulated car can be driven through. A drawback of the car simulator according to the German patent DE 4105963 is that the user can only drive the simulated car through the environment that is pre-recorded by the video recorder car. It is an object of the invention to provide a transport device with a transport device simulator for enabling a user to interactively interfere in a simulation of a route actually driven by the transport device. According to the invention, this object is achieved by providing a transport device comprising a video camera for capturing images of an environment of an actual route traversed by the transport device, means for, while capturing the images, determining actual positions of the transport device, each actual position corresponding to one of the captured images, a user interface, a transport device simulator for providing a simulation of a route of a simulated transport device and a display for providing the simulation to a user. The transport device simulator comprises an input for receiving the captured images, the corresponding actual positions and, from the user interface, user commands for controlling the simulated transport device, a processor for constructing the simulation based on the captured images, the corresponding actual positions and the user commands, and an output for providing the simulation for rendering.

The simulator according to the invention enables the user to drive a simulated transport device, such as a car or a boat, via a user interface. The processor constructs the simulation based on the user commands, the environment of the actual route of the real transport device and the position of the real transport device in said environment. The images are captured by the real transport device and comprise the environment that can be observed from the viewpoint of the driver of the real transport device. The viewpoint of the driver of the real transport device is known from the means for determining the actual position of the real transport device. The user commands control the position of the simulated transport device on the route. Based on the user commands, the images of the environment and the viewpoint of the driver of the real transport device, the simulation is constructed. By incorporating the images in the simulation, the user of the simulator will be able to drive the simulated car through an environment resembling the environment of the real car and will enjoy an experience similar to actually driving the real transport device.

According to one embodiment of the invention the processor is operative to compare a position of the simulated transport device to the actual position of the transport device and to construct the simulation based on the outcome of the position comparison. According to another embodiment of the invention the processor is also operative to use the outcome of the position comparison to construct the simulation by zooming in or out on the captured images and/or scrolling through the captured images. Zooming and scrolling are the simplest techniques for constructing the simulation. For example, if the position of the simulated transport device is in front of the actual position of the real transport device, zooming in on the recorded video image may provide the user the sensation of being closer to objects in the video image than the real transport device is. If the user hits the brakes and the position of the simulated transport device gets behind the actual position of the real transport device, zooming out may provide a sensation of being further away from the objects in the video image. If just part of the recorded video images is shown on the display, steering the simulated transport device to the left may be simulated by scrolling through the video images to the left. According to one embodiment of the invention the processor is operative to analyze the images for detecting objects in the images and to synthesize graphics objects representing the objects, for constructing the simulation.

Synthesized graphics can substantially enhance the simulation. The video images only comprise information that is detectable from the viewpoint of the real transport device. When looking from another viewpoint, new information appears which may not be directly available from the recorded information. Synthesized graphics can fill in the gaps that occur when the viewpoint of the simulated transport device differs from the viewpoint of the real transport device. If, for example, the back and the left side of a truck are visible from the viewpoint of a real car and a simulated car is situated at the right side of that truck, the processor may detect the truck, synthesize a graphics object of that truck and show its right side to the user. When using synthesized graphics objects it is also possible to zoom in on, for example, a truck in front of the simulated car, without zooming in on distant background objects, instead of zooming in on the whole image. This will make the simulation more realistic. According to another embodiment of the invention the processor is operative to provide the simulation with a synthesized graphics object representing the transport device at a position corresponding to the actual position. According to yet another embodiment of the invention the processor is operative to keep a distance from the simulated transport device to the transport device within predetermined limits. The indicator enables the user to follow the route of the real transport device.

The simulator may compute a score, indicating the success of the user in following the route of the real transport device. When the user tries to steer the simulated transport device beyond the predetermined limits the position of the simulated transport device may, for example, instantaneously be changed to coincide with the position of the real transport device. Alternatively the control of the user over the simulated transport device may be limited to navigating the simulated transport device towards the position of the real transport device. If, for example, the distance between the real and the simulated transport device, positioned at the left of the real transport device, equals the predetermined limit, steering to the left will not result in a change of the distance. If the user steers to the right, more than the driver of the real transport device does, the distance will decrease.

According to another embodiment of the invention the transport device further comprises status recording means for recording status information describing a status of the transport device and the processor is operative to construct the simulation also based on the status information.

The user may, for example, be informed about the gear the real transport device is in or the speed at which the real transport device is travelling. A score of the user may, among others, be based on a comparison of the status of the real transport device and the simulated transport device. According to another embodiment of the invention the status information also describes driver input for control of the transport device by the driver and the driver input is used for at least partly controlling the simulated transport device.

For example, in an 'easy-to-play' mode, the user will only have to steer the simulated transport device and gas, brakes and gear changes are controlled by a real user of the real transport device.

According to another embodiment of the invention, a transport device simulator is provided for providing a simulation of a route of a simulated transport device.

Such a transport device simulator may be used, for example, at home when a driver is driving in a real transport device. Means in the real transport device may capture the video images and determine its actual positions. This information may be communicated, preferably by wireless communication means, to a distant transport device simulator. This will enable a user of the simulator to drive along with a real driver in a real transport device.

These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. Brief description of the drawings In the drawings: Figure 1 shows a schematic representation of the transport device according to the invention,

Figure 2 shows an exemplary 3D representation of the simulation, Figure 3 shows an exemplary 2D representation of the simulation, Figure 4a and 4b show exemplary representations of the simulation comprising score and status information,

Figure 5 shows an exemplary arrangement of two status information panels and a score indicator, and

Figure 6 shows a flow diagram of the method according to the invention. Figure 7 shows a flow diagram of a preferred method according to the invention.

Figure 1 shows a schematic representation of the transport device 10 according to the invention. The real transport device 10 comprises a user interface 15, a video camera 16, means 17 for determining an actual position of the real transport device, a display 18 and a transport device simulator 11. The simulator 11 comprises an input 12 for receiving user commands from the user interface 15, video images from the video camera 16 and the actual position of the real transport device 10 from the means 17 for determining the position. The received data is processed by a processor 13, which processor 13 is operative to construct a simulation. An output 14 provides the simulation to the user via a display 18, speakers (not shown) or other types of output devices.

The transport device 10 may, for example, be a car, a bicycle, a boat or an airplane. The actual position of the real transport device 10 may, for example, be determined using GPS sensors, a compass, measurements of the speed of the real transport device 10, measurements of the speed of the real transport device 10, and/or measurements of the positions of a steering wheel. Techniques and means 17 for position measurements of cars are often used in car navigation systems. The position of the real transport device may also be determined by analyzing the recorded video images. From the video images the position of the camera 16 relative to the environment shown in the recorded images can be derived. Additional to the camera 16, microphones or object detection systems using laser or radar may be used for gathering information about the environment. Knowing the position of the real transport device 10, information about the environment may be obtained from on board navigation systems or databases describing the environment, or from a dedicated wireless network or a general purpose network, such as the Internet.

For the real transport device according to the invention, a keyboard, a mouse, a joystick or any other type of user interface 15 may be used for providing user commands 15 to the first input 12a of the simulator 11. Many different types of user interfaces are known for use with computers and computer games. Preferably, a user interface 15 comprising a steer, pedals and a gear lever is used for providing a realistic simulation. The user commands, the environment and the actual position of the real transport device are used by the processor for 13 constructing the simulation. The output 14 may be coupled to all sorts of output devices, such as a display 18 and a speaker.

The real car 10 may comprise the video camera 16, the means 17 for determining the actual position, the simulator 11 and the user interface 15. The display 18 and the speaker may be integrated in the back of a head rest of a front seat. The display 18 preferably is a LCD display, but other display types may be used as well. In this event the user, sitting in the real car 10, can use the simulator and enjoy the experience of actually driving the real car 10 while the video camera 16 records the route of the real car 10. Alternatively, during the recording of the route, the information about the environment is .stored on any storage means and is to be used later for constructing a simulation. In yet another embodiment, only the video camera 16 and the means 17 for determining actual positions are included in the real car 10 and the recorded information is sent to a distant simulator 11 via wireless communication means. This will, for example, enable a child to, using a personal computer or other suitable computer devices such as a handheld, follow his father on his route to work. The recorded information may also be stored on removable storage means, such as a removable hard disc, a solid state memory, or a recordable CD or DVD. Such a removable storage medium may later be used by any simulator arranged for retrieving the information from the storage medium.

Figure 2 shows an exemplary 3D representation of the simulation. Figure 2 shows part of a tree trunk 21, a traffic sign 26, a car 24 and a road 25. The representation of these objects may directly be obtained from the recorded video images or may partly or completely be synthesized graphics objects. Figure 2 also shows a representation of the nose 22 of the simulated car. An indicator 23 is provided, indicating the position of the real car 10 and enabling the user to follow the real car 10. Preferably, in the simulation, the simulated car takes the place of the real car 10 in the real world. The user may try to let the representation of the nose 22 of the simulated car coincide with the indicator 23 indicating the position of the real car 10. Alternatively, the simulated car and the real car 10 are both represented on the simulated road 25. In this event the user should try to keep at a safe distance from the indicator 23.

The environment shown in this 3D representation is described by the recorded video images. The viewpoint from which the driver of the simulated car observes the environment is defined by the user commands. Based on the recorded video images and the user commands, the processor 13 constructs the simulation. The simulation may be constructed using simple techniques like zooming and scrolling. For example, if the position of the simulated transport device is in front of the actual position of the real transport device, zooming in on the recorded video image may provide the user the sensation of being closer to objects in the video image than the real transport device is. If the user hits the brakes and the position of the simulated transport device gets behind the actual position of the real transport device, zooming out may provide a sensation of being further away from the objects in the video image. If just part of the recorded video images is shown on the display, steering the simulated transport device to the left may be simulated by scrolling through the video images to the left.

Preferably, synthesized graphics supplement the recorded video images when the viewpoint of the simulated and the real car 10 are not equal. The video images only comprise the information that can be observed from the viewpoint of the camera 16 of the real car 10. Some information can be observed from the viewpoint of the simulated car and not from the viewpoint of the camera 16. Such information can be included in the simulation by detecting specific real objects in the recorded video images and synthesizing graphics objects representing the complete real objects. For synthesizing graphics objects also recorded information from other recording means may be used. For example, if an object detection system detects a car that is five meters in front of the simulated car, a graphics object representing the car can be included in the simulation. Similarly, if a rain sensor detects rain, rain could be included in simulation.

The simulation may be enhanced by using two or more video cameras instead of one and by using captured video images from multiple cameras for constructing the simulation. Video images captured by multiple 2D video cameras, recorded from different viewpoints, may be combined into one 3D representation of the environment. Alternatively 2D to 3D conversion techniques may be applied to video images recorded by only one 2D video camera. Such conversion techniques may use relative movements of objects in the 2D video images to construct a 3D representation and are well known from the art of simulated 3D television.

Figure 3 shows an exemplary 2D representation of the simulation. Such a bird-eye view cannot easily be recorded by the camera 16 of the real car 10. However, such a view could easily be constructed using synthesized graphics when the objects and their positions are known. These positions can be derived from the video images and other recorded information. The 2D view shown in Figure 3 shows the same objects as the 3D view in Figure 2.

Figure 4a and 4b show exemplary representations of the simulation comprising score 41 and status information. A score 41 may indicate how well the user succeeds in following the real car 10. Preferably, the processor 13 is operative to use object recognition for recognizing predetermined objects, such as traffic signs and traffic lights, and extra points are awarded for obeying the traffic regulations. A competition element can be introduced by comparing the scores of two players. For example two children may simultaneously try to follow the route their mother is driving in a real car 10. Alternatively, the driver of the real car 10 may have a score 41 too, depending on how well the driver obeys the traffic rules.

Additional recording means may record status information describing the status of the real transport device. For example, the speed at which the car drives, the gear the car is in, the lights that are turned on or the angle over which the steering wheel has been turned may be monitored. The status information may be included in the simulation, for example in a first status information panel 43 for the real car 10. The status information for the simulated car may also be included in the simulation, for example, in a second status information panel 42. Providing both the status information of the real 10 car and of the simulated car will enable the user to compare his or hers control decisions to those of the driver of the real car 10.

In an embodiment of the simulator 11 according to the invention, status information describing the status of the real transport device 10 is used for controlling the simulated transport device. In this event, the simulated transport device is at least partly controlled by driver input for controlling the real transport device 10. For example, in an 'easy-to-play' mode, the user will only have to steer the simulated transport device and gas, brakes and gear changes of the simulated transport device are controlled by a real user of the real transport device. Especially the status information describing the status of the steering wheel of the real car 10 may be used to enhance the simulation. When the real car makes a turn, the viewpoint of the camera 16 changes and the simulated car will follow the road although the user does not have to steer the simulated car. The user will only be able to switch lanes and does not have to make turns. Use of the status information describing the status of the steering wheel may enhance the simulation. For example, when the driver of the real car 10 steers to the right, the simulation may simultaneously steer the simulated car to the left. The user of the simulator 11 will then have to steer the simulated car to the right in order to keep the simulated car on the road. In this way the user of the simulator 11, like the driver of the real car 10, will have to make turns where the road curves.

Figure 5 shows an exemplary arrangement of two status information panels 42, 43 and a relative score indicator 54. Preferably a first status information panel 42 shows status information describing the simulated transport device and a second status 43 information panel shows status information describing the real car 10 or another simulated car, which is controlled by another user. A status information panel 42, 43 may, for example, comprise a speed indicator 51, a gear selection indicator 52 and a series of on/off- indicators 53. The indicators on both status information panels may look different to allow an easy distinction between a user's own status information and the status information for another user or the driver of the real car 10. When in competition mode, the simulation may also ' comprise a relative score indicator 54 for showing the ratio of the scores of the competitors. Figure 6 shows a flow diagram of the method according to the invention. In a receiving step RIM, video data is received from a camera, the data comprising visual information about the environment. Then in an optional conversion step CON the video data is converted to a graphical representation of the environment. User commands are received from a user interface in a process RUC and the actual position of the real transport device 10 is received in another receiving process RAP. Then in a simulation step SIM, all received information is used for constructing a simulation of a route of a simulated transport device. Then an output provides the simulation to the user in a providing step OUT. As the receiving of user commands, the actual position of the real transport device 10 and the video data occurs continuously during the use of the method, the processing of this information and the providing of the simulation do so too.

Figure 7 shows a flow diagram of a preferred method according to the invention. In this method, additional status information is received in a status receiving process RSI. This information may be used, for example, for comparing the driving speed, gear and rpm of the real and the simulated transport devices. Additionally the simulation step SIM may comprise a comparison step COMP, for comparing the actual position AP of the real transport device 10 with the simulated position SP of the simulated transport device. As a result of this comparison a score SC may be determined or the freedom of movement for the simulated car may be restricted if the distance between actual and simulated car is above a predetermined limit. For example, the control of the user over the simulated transport device may be limited to navigating the simulated transport device towards the position AP of the real transport device. If, for example, the distance between the real and the simulated transport device, positioned at the left of the real transport device, equals the predetermined limit, steering to the left will not result in a change of the distance. If the user steers to the right, more than the driver of the real transport device does, the distance will decrease.

The methods shown in Figure 6 and 7 may be performed by the transport device simulator 11 shown in Figure 1, using software for instructing the processor 13 to perform the required steps. Alternatively dedicated hardware modules may be used for performing the steps of the methods shown in Figure 6 and 7. It will be appreciated that the invention also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice. The program may be in the form of source code, object code, a code intermediate source and object code such as partially compiled form, or in any other form suitable for use in the implementation of the method according to the invention. The carrier may be any entity or device capable of carrying the program. For example, the carrier may include a storage medium, such as a ROM, for example a CD ROM or a semiconductor ROM, or a magnetic recording medium, for example a floppy disc or hard disk. Further the carrier may be a transmissible carrier such as an electrical or optical signal, which may be conveyed via electrical or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being adapted for performing, or for use in the performance of, the relevant method

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

CLAIMS:

1. A transport device (10) comprising: a video camera (16) for capturing images of an environment of an actual route traversed by the transport device (10), means (17) for, while capturing the images, determining actual positions of the transport device (10), each actual position corresponding to one of the captured images, a user interface (15), a transport device simulator (11) for providing a simulation of a route of a simulated transport device, the transport device simulator (11) comprising: an input (12) for receiving the captured images, the corresponding actual positions and, from the user interface (15), user commands for controlling the simulated transport device, a processor (13) for constructing the simulation based on the captured images, the corresponding actual positions and the user commands, and an output (14) for providing the simulation for rendering, and - a display (18) for providing the simulation to a user.

2. A transport (10) device according to claim 1, wherein the processor (13) is operative to compare a position of the simulated transport device to the actual position of the transport device (10) and to construct the simulation based on an outcome of the position comparison.

3. A transport device (10) according to claim 2, wherein the processor (13) is operative to use the outcome of the position comparison to construct the simulation by zooming in or out on the captured images and/or scrolling through the captured images.

4. A transport device (10) according to one of the preceding claims, wherein the processor (13) is operative to analyze the images for detecting objects in the images and to synthesize graphics objects representing the objects, for constructing the simulation.

5. A transport device (10) as claimed in claim 4, wherein the processor (13) is operative to compare a behaviour of the simulated transport device to a predetermined behaviour corresponding to the detected objects and to provide the output with an outcome of the behaviour comparison.

6. A transport device (10) as claimed in claim 4, wherein the processor (13) is operative to provide the simulation with a synthesized graphics object representing the transport device (10) at a position corresponding to the actual position.

7. A transport device (10) as claimed in claim 2, wherein the processor (13) is operative to keep a distance from the simulated transport device to the transport device (10) within predetermined limits.

8. A transport device (10) as claimed in claim 1, further comprising status recording means for recording status information describing a status of the transport device (10) and wherein the processor (13) is operative to construct the simulation also based on the status information.

9. A transport device (10) as claimed in claim 8, wherein the status information also describes driver input for control of the transport device (10) by the driver and wherein the driver input is used for at least partly controlling the simulated transport device.

10. A transport device simulator (11) for providing a simulation of a route of a simulated transport device, comprising: - an input (12) for receiving images of an environment of an actual route traversed by a transport device

(10), actual positions of the transport device (10), each actual position corresponding to one of the images, and - from a user interface (15), user commands for controlling the simulated transport device, a processor (13) for constructing the simulation based on the images, the corresponding actual positions and the user commands, and an output (14) for providing the simulation for rendering.

11. A method for providing a simulation of a route of a simulated transport device, the method comprising: receiving images of an environment of an actual route traversed by a transport device, receiving user commands, from a user interface, for controlling the simulated transport device, receiving actual positions of the transport device, each actual position corresponding to one of the images, - constructing the simulation based on the images, the corresponding actual positions and the user commands, and providing the simulation to an output for rendering.

12. A method as claimed in claim 11, further comprising a step of comparing a position of the simulated transport device to the actual position of the transport device and wherein the step of constructing the simulation is based on an outcome of the positional comparison.

13. A method as claimed in claim 11, further comprising the step of receiving status information describing the status of the transport device and wherein the constructing is also based on the status information.

14. A computer program product which program is operative to cause a processor to perform the method as claimed in claim 11, 12 or 13.