WO2024070608A1

WO2024070608A1 - Information processing device, information processing method, and program

Info

Publication number: WO2024070608A1
Application number: PCT/JP2023/032950
Authority: WO
Inventors: 達也山崎; 巨成高橋; 辰志梨子田
Original assignee: ソニーグループ株式会社
Priority date: 2022-09-27
Filing date: 2023-09-11
Publication date: 2024-04-04

Abstract

The present disclosure pertains to an information processing device, an information processing method, and a program that make it possible to achieve a driving simulation that uses the vehicle to be used during normal driving.　The present invention controls operation of the vehicle by obtaining driving operation data, which expresses a driving operation amount for multiple types of operation systems for operating a vehicle, and supplying a control signal to multiple types of drive systems for driving the vehicle. The present invention controls operation of the vehicle using control data for controlling travel of the vehicle according to driving operation data when in travel mode, and controls operation of the vehicle using control data for controlling the behavior of the vehicle so as to reproduce the desired behavior of the vehicle in a virtual space according to the driving operation data, on the basis of a vehicle settings parameter obtained via communication from another information processing device when in a simulation mode. When an operation indicating a switch is to be made from the travel mode to the simulation mode, whether or not to transition to the simulation mode is determined according to whether the vehicle is stopped.

Description

Information processing device, information processing method, and program

The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that are capable of implementing a driving simulation using a vehicle used in normal driving.

Traditionally, vehicle driving simulations have not used vehicles used in normal driving, but have instead been performed using simulation operation systems (steering wheels, accelerator pedals, brake pedals, etc.).

For example, Patent Document 1 discloses a racing game device in which a racing game is played using an operating unit that has a steering wheel, accelerator, and brake.

JP 2000-229177 A

However, there was a need to perform driving simulations using vehicles used in normal driving, rather than using the simulation operation system described above.

This disclosure has been made in light of these circumstances, and makes it possible to realize a driving simulation using a vehicle used in normal driving.

The information processing device according to the first aspect of the present disclosure includes a communication unit that communicates with other information processing devices, an operation data acquisition unit that acquires driving operation data indicating the amount of operation of driving operations for a plurality of types of operation systems that operate the vehicle, a vehicle operation control unit that supplies control signals to a plurality of types of drive systems that drive the vehicle to control the operation of the vehicle, a driving mode control unit that supplies control data to the vehicle operation control unit for controlling the driving of the vehicle according to the driving operation data when in a driving mode in which the vehicle is used to drive, a simulation mode control unit that supplies control data to the vehicle operation control unit for controlling the behavior of the vehicle so as to reproduce the behavior of the vehicle in a virtual space determined according to the driving operation data based on vehicle setting parameters acquired from the other information processing device via the communication unit when in a simulation mode in which a driving simulation is performed using the vehicle, and a stopped state determination unit that determines whether to transition to the simulation mode according to the stopped state of the vehicle when an operation instructing switching from the driving mode to the simulation mode is performed.

The information processing method or program of the first aspect of the present disclosure includes communicating with another information processing device, acquiring driving operation data indicating the amount of driving operation for a plurality of types of operating systems that operate a vehicle, supplying control signals to a plurality of types of drive systems that drive the vehicle to control the operation of the vehicle, controlling the operation of the vehicle with control data that controls the running of the vehicle according to the driving operation data when in a driving mode in which the vehicle is used to run, controlling the operation of the vehicle with control data that controls the behavior of the vehicle so as to reproduce the behavior of the vehicle in a virtual space determined according to the driving operation data based on vehicle setting parameters acquired by communication from the other information processing device when in a simulation mode in which a driving simulation is performed using the vehicle, and determining whether to transition to the simulation mode according to the stopped state of the vehicle when an operation instructing a switch from the driving mode to the simulation mode is performed.

In a first aspect of the present disclosure, communication is performed with another information processing device, driving operation data indicating the amount of driving operation for multiple types of operating systems that operate the vehicle is obtained, control signals are supplied to multiple types of drive systems that drive the vehicle to control the operation of the vehicle, and in a driving mode in which the vehicle is used to drive, the operation of the vehicle is controlled by control data that controls the driving of the vehicle in accordance with the driving operation data, and in a simulation mode in which a driving simulation is performed using the vehicle, the operation of the vehicle is controlled by control data that controls the behavior of the vehicle so as to reproduce the behavior of the vehicle in a virtual space determined in accordance with the driving operation data based on vehicle setting parameters obtained by communication from the other information processing device, and when an operation is performed to instruct switching from the driving mode to the simulation mode, it is determined whether or not to transition to the simulation mode in accordance with the stopped state of the vehicle.

The information processing device according to the second aspect of the present disclosure includes a communication unit that communicates with other information processing devices mounted on the vehicle, a virtual space generation unit that performs a driving simulation in which the vehicle virtually travels in a virtual space according to driving operation data acquired from the vehicle via the communication unit, and generates a simulation image by capturing images of the virtual space in all directions centered on the vehicle with a virtual camera arranged in the virtual space so as to correspond to a predetermined position of the vehicle, and an image conversion processing unit that performs image conversion processing on the simulation image based on vehicle position and attitude data indicating the position and attitude of the vehicle and occupant position and attitude data including at least the viewpoint positions of the occupants of the vehicle.

The information processing method or program of the second aspect of the present disclosure includes communicating with another information processing device mounted on the vehicle, performing a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle through the communication, capturing images of the virtual space in all directions from the vehicle with a virtual camera disposed in the virtual space so as to correspond to a predetermined position of the vehicle, and generating a simulation image, and performing an image conversion process on the simulation image based on vehicle position and attitude data indicating the position and attitude of the vehicle and occupant position and attitude data including at least the viewpoint positions of the occupants of the vehicle.

In a second aspect of the present disclosure, communication is performed with another information processing device mounted on the vehicle, and a driving simulation is performed in which the vehicle is virtually driven in a virtual space according to driving operation data acquired from the vehicle through communication, and a simulation image is generated by capturing images of the virtual space in all directions centered on the vehicle with a virtual camera placed in the virtual space corresponding to a predetermined position of the vehicle, and an image conversion process is performed on the simulation image based on vehicle position and attitude data indicating the position and attitude of the vehicle and occupant position and attitude data including at least the viewpoint positions of the vehicle occupants.

1 is a block diagram showing a configuration example of an embodiment of a driving simulation system to which the present technology is applied; FIG. 1 is a diagram illustrating a configuration example of a vehicle that employs a by-wire system. FIG. 2 is a diagram illustrating an example of the configuration of a garage device. FIG. 2 is a diagram illustrating an example of the internal configuration of a vehicle. 11A and 11B are diagrams illustrating an example of a method for identifying the viewpoint of an occupant. FIG. 2 is a block diagram showing a configuration example of a garage system control device. 11 is a diagram illustrating an example in which a virtual plane viewed from a viewpoint position is displayed on a garage display. FIG. 2 is a block diagram showing an example of the configuration of an operation system and a drive system of the in-vehicle control device; 2 is a block diagram showing a configuration example of a video system and an audio system of the in-vehicle control device. FIG. FIG. 4 is a diagram showing an example of a data structure of vehicle parameters and vehicle setting parameters. FIG. 13 is a diagram showing the behavior of a vehicle in a spin state. FIG. 4 is a diagram showing an example of vehicle parameters and vehicle setting parameters. FIG. 4 is a diagram illustrating suspension setting data. FIG. 4 is a diagram illustrating sheet setting data. 11 is a flowchart illustrating an initial setup process on the garage system side. 4 is a flowchart illustrating an initial setup process on the vehicle side. 11 is a flowchart illustrating a driving simulation operation process on the garage system side. 10 is a flowchart illustrating a driving simulation operation process on the vehicle side. 11 is a flowchart illustrating a vehicle position and attitude recognition process. 5 is a flowchart illustrating an occupant position and attitude recognition process. 11 is a flowchart illustrating a reflection control process. FIG. 1 is a block diagram showing an example of the configuration of a training system. 1 is a flowchart illustrating unsupervised training. 1 is a flowchart illustrating supervised training. 13 is a flowchart illustrating learning. 1 is a flowchart illustrating training conducted one-on-one between a teacher and a student, the teacher being a human. 13 is a flowchart illustrating training in which the teacher is a human and the teacher and students are one-to-many. This is a flowchart that explains training in which the teacher is an AI and the teacher and students are trained one-to-one or one-to-many. 11 is a flowchart illustrating personal optimization of a vehicle. 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied.

Below, specific embodiments of the application of this technology will be described in detail with reference to the drawings.

<Garage system configuration example>
FIG. 1 is a block diagram showing an example of the configuration of an embodiment of a driving simulation system to which the present technology is applied.

The driving simulation system 11 shown in FIG. 1 is composed of a garage system 12 and a vehicle 13, and can realize a driving simulation using the vehicle 13 that the user uses during normal driving.

The garage system 12 is composed of a garage device 14, a charging device 15, a garage system control device 16, and a home server 17.

The garage device 14 has the function of displaying images used in a driving simulation (hereinafter referred to as simulation images) surrounding the vehicle 13, in addition to the function of a general garage for storing the vehicle 13, as described below with reference to FIG. 3. In this embodiment, the garage device 14 is described as being installed inside a garage. Alternatively, the garage device 14 may be installed in a parking lot, parking space, etc.

The charging device 15 charges the vehicle 13 by connecting a charging and communication cable to the vehicle 13, and enables high-speed IP (Internet Protocol) communication between the garage system 12 and the vehicle 13.

The garage system control device 16 controls the garage system 12 when performing a driving simulation, using data acquired via an external network and data stored in the home server 17. The detailed configuration of the garage system control device 16 will be described later with reference to FIG. 6.

The home server 17 obtains data necessary for the garage system control device 16 to control the garage system 12 via an external network and provides it to the garage system control device 16.

As described below with reference to FIG. 2, the vehicle 13 employs a by-wire system that is configured to detect driving operations on the operating system using sensors and electrically transmit signals corresponding to the driving operations to the drive system via signal lines, rather than physically transmitting operations on the operating system to the drive system. The vehicle 13 is also configured with an in-vehicle control device 18 and a power source 19.

The in-vehicle control device 18 communicates with the garage system control device 16 and controls the vehicle 13 when performing a driving simulation. The detailed configuration of the in-vehicle control device 18 will be described later with reference to Figures 8 and 9.

For example, if the vehicle 13 is an EV (Electric Vehicle), the power source 19 is an electric motor that uses electricity as an energy source. If the vehicle 13 is a gasoline-powered vehicle, the power source 19 is an internal combustion engine that uses gasoline as fuel, and if the vehicle 13 is a hybrid vehicle, a combination of an internal combustion engine and an electric motor is used.

The driving simulation system 11 is configured in this manner, and when the vehicle 13 is stored in the garage device 14, a driving mode in which normal driving is performed using the vehicle 13 is switched to a simulation mode in which a driving simulation is performed using the vehicle 13, and a driving simulation can be performed.

With reference to Figure 2, an example configuration of a vehicle 13 that employs a by-wire system will be described.

As shown in FIG. 2, the vehicle 13 is configured with a steering wheel 21, accelerator pedal 22, brake pedal 23, axle steering unit 24, throttle motor 25, brake 26, steering wheel rotation sensor 27, accelerator position sensor 28, brake position sensor 29, axle drive mechanism 30, throttle drive mechanism 31, and brake drive mechanism 32. The vehicle 13 is configured such that the operating system, such as the steering wheel 21, accelerator pedal 22, and brake pedal 23, and the drive system, such as the axle steering unit 24, throttle motor 25, and brake 26, are electrically connected to the in-vehicle control device 18 via signal lines, and are physically separated from each other.

The steering wheel rotation sensor 27 detects the rotation of the steering wheel 21 in response to the rotation operation by the driver of the vehicle 13, and outputs a steering wheel rotation signal indicating the amount of rotation of the steering wheel 21 to the in-vehicle control device 18.

The accelerator position sensor 28 detects the position of the accelerator pedal 22 according to the depression operation by the driver of the vehicle 13, and outputs an accelerator position signal indicating the amount of depression of the accelerator pedal 22 to the in-vehicle control device 18.

The brake position sensor 29 detects the position of the brake pedal 23 according to the depression operation by the driver of the vehicle 13, and outputs a brake position signal indicating the amount of depression of the brake pedal 23 to the in-vehicle control device 18.

The axle-direction drive mechanism 30 drives the axle-direction steering unit 24 according to an axle-direction control signal supplied from the vehicle control device 18 based on the steering wheel rotation signal output from the steering wheel rotation sensor 27. This allows the axle-direction steering unit 24 to steer the tires of the vehicle 13 to an angle that corresponds to the rotation operation of the steering wheel 21 by the driver of the vehicle 13.

The throttle drive mechanism 31 drives the throttle motor 25 in accordance with a throttle control signal supplied from the in-vehicle control device 18 based on the accelerator position signal output from the accelerator position sensor 28. This allows the throttle motor 25 to drive the vehicle 13 so as to accelerate in response to the driver of the vehicle 13 depressing the accelerator pedal 22.

The brake drive mechanism 32 drives the brake 26 in accordance with a brake control signal supplied from the in-vehicle control device 18 based on the brake position signal output from the brake position sensor 29. This allows the brake 26 to drive the vehicle 13 so as to decelerate in response to the driver of the vehicle 13 depressing the brake pedal 23.

When in a driving mode, the vehicle 13 configured in this manner can travel by operating the drive system, such as the axle steering unit 24, throttle motor 25, and brake 26, according to the control of the in-vehicle control device 18 in response to the operation of the steering wheel 21, accelerator pedal 22, brake pedal 23, and other operating systems by the driver of the vehicle 13.

On the other hand, in the simulation mode, the vehicle 13 supplies driving operation data indicating the operation by the driver of the vehicle 13 of the operating systems such as the steering wheel 21, accelerator pedal 22, and brake pedal 23 to the garage system control device 16 to execute a driving simulation. At this time, the drive system such as the axle steering unit 24, throttle motor 25, and brake 26 do not operate and remain stopped. In other words, the operating system and drive system of the vehicle 13 are connected by a by-wire system, and the vehicle is configured such that the drive system does not operate even if the operating system is operated in the simulation mode. It is sufficient that at least one corresponding operating system and drive system is connected by the by-wire system. For example, if the steering wheel 21 and the axle steering unit 24 are connected by the by-wire system, tire wear caused by steering the tires while the vehicle 13 is stopped can be avoided.

Furthermore, the vehicle 13 is configured with a handle reaction force drive mechanism 33, an accelerator reaction force drive mechanism 34, a brake reaction force drive mechanism 35, a suspension drive mechanism 36, and a seat drive mechanism 37.

The steering wheel reaction force drive mechanism 33 generates a steering wheel reaction force in the steering wheel 21 that resists the arm force of the driver who is rotating the steering wheel 21 in accordance with a steering wheel reaction force control signal supplied from the in-vehicle control device 18.

The accelerator reaction force drive mechanism 34 generates an accelerator reaction force in the accelerator pedal 22 that resists the leg force of the driver who is depressing the accelerator pedal 22 in accordance with an accelerator reaction force control signal supplied from the in-vehicle control device 18.

The brake reaction force drive mechanism 35 generates a brake reaction force in the brake pedal 23 that resists the leg force of the driver who is depressing the brake pedal 23 in accordance with a brake reaction force control signal supplied from the in-vehicle control device 18.

The suspension drive mechanism 36 drives a hydro-pneumatic suspension (not shown) that can adjust the height of the vehicle 13 by adjusting air pressure or hydraulic pressure, and actively controls the height of each of the four wheels of the vehicle 13 independently.

The seat drive mechanism 37 drives actuators (not shown) built into the back and seat surface of each seat of the vehicle 13 to perform various adjustment operations on the seat surface and back at a certain speed along the time axis.

With reference to Figure 3, an example configuration of the garage device 14 will be described.

A of FIG. 3 shows a schematic configuration example of the garage device 14 viewed from the side with the vehicle 13 stored, and B of FIG. 3 shows a schematic configuration example of the garage device 14 viewed from the top with the vehicle 13 stored. In this way, a driving simulation can be performed with the vehicle 13 stored in the garage device 14.

As shown in FIG. 3, the garage device 14 is configured with a garage display 41-1 for the ceiling, a garage display 41-2 for the front, a garage display 41-3 for the right side, a garage display 41-4 for the left side, a garage display 41-5 for the rear, a garage display 41-6 for the floor, a garage camera 42, and garage sensors 43-1 and 43-2.

The garage display 41-1 for the ceiling surface is installed so as to cover the entire ceiling wall surface of the garage device 14, and displays a simulation image for the ceiling surface supplied from the garage system control device 16 when a driving simulation is performed.

The front garage display 41-2 is installed so as to cover the entire front wall of the garage device 14, and displays the front simulation image supplied from the garage system control device 16 when the driving simulation is being performed.

The garage display 41-3 for the right side is installed so as to cover the entire right side wall of the garage device 14, and displays the simulation image for the right side supplied from the garage system control device 16 when the driving simulation is performed.

The garage display 41-4 for the left side is installed so as to cover the entire left side wall of the garage device 14, and displays the simulation image for the left side supplied from the garage system control device 16 when the driving simulation is being performed.

The rear garage display 41-5 is installed to cover the entire rear wall of the garage device 14, and displays the rear simulation image supplied from the garage system control device 16 when the driving simulation is being performed.

The garage floor display 41-6 is installed so as to cover the entire floor and wall surface of the garage device 14, and displays a simulation image for the floor supplied from the garage system control device 16 when a driving simulation is being performed.

In the following, when there is no need to distinguish between the garage display 41-1 for the ceiling, the garage display 41-2 for the front, the garage display 41-3 for the right side, the garage display 41-4 for the left side, the garage display 41-5 for the rear, and the garage display 41-6 for the floor, they will be referred to as garage display 41. For example, the garage display 41 uses a display unit that employs a light-emitting method such as LED (Light Emitting Diode) or OLED (Organic Light Emitting Diode), and multiple display units can be tiled for use.

The garage camera 42 photographs the vehicle 13 stored in the garage device 14 to recognize the position and posture of the vehicle 13, and supplies the image data to the garage system control device 16.

In order to recognize the position and attitude of the vehicle 13, the garage sensors 43-1 and 43-2 supply sensor data obtained by detecting the positions of multiple markers (e.g., retroreflective materials) attached to the body of the vehicle 13 to the garage system control device 16. Note that, when there is no need to distinguish between the garage sensors 43-1 and 43-2, they will be referred to simply as garage sensor 43 below. Also, in the example shown in FIG. 3, two garage sensors 43-1 and 43-2 are shown, but the position and attitude of the vehicle 13 may be detected by one garage sensor 43 or three or more garage sensors 43. Alternatively, the relative relationship between the position and attitude of the vehicle 13 and the garage device 14 may be detected by an external sensor (not shown) provided on the vehicle 13.

The driving simulation system 11 may recognize the position and attitude of the vehicle 13 by using a positioning sensor (e.g., a piezoelectric element) embedded in the floor of the garage device 14 to detect the four points where the tires of the vehicle 13 are in contact. Alternatively, the driving simulation system 11 may recognize the position and attitude of the vehicle 13 by acquiring shape data such as a point cloud or mesh of the vehicle 13 using a measuring device that combines an RGB camera, a depth sensor, etc.

The garage device 14 configured in this manner can display a simulation image seen from the viewpoint position P of a passenger inside the vehicle 13 on a garage display 41 arranged to surround the periphery of the vehicle 13 when a driving simulation is being performed.

An example of the internal configuration of the vehicle 13 will be described with reference to Figure 4.

FIG. 4A shows an example of the internal configuration of vehicle 13 as viewed forward from the driver's seat and passenger seat, and FIG. 4B shows an example of the internal configuration of vehicle 13 as viewed forward from the rear seat.

As shown in FIG. 4, the interior of the vehicle 13 is provided with a navigation display 51, side mirror displays 52L and 52R, an instrument panel display 53, a rearview mirror display 54, rear seat displays 55L and 55R, an in-vehicle camera 56, in-vehicle sensors 57-1 and 57-2, and speakers 58-1 and 58-2.

The navigation display 51 displays a navigation image based on the video stream for navigation supplied from the in-vehicle control device 18, for example, by placing a vehicle mark indicating the driving position of the vehicle 13 on a map image and displaying a navigation image in which the vehicle mark moves on the map image. In addition, in the driving mode, the navigation display 51 displays an actual navigation image according to the position information of the vehicle 13, and in the simulation mode, it displays a virtual navigation image according to the driving position in a virtual space based on a driving simulation.

Side mirror displays

52L and 52R display, for example, side mirror images taken toward the rear of vehicle 13 by side mirror cameras installed on the left and right sides of vehicle 13, based on the side mirror image streams supplied from in-vehicle control device 18. In addition, in driving mode, side mirror displays 52L and 52R display actual side mirror images taken by side mirror cameras installed on vehicle 13, and in simulation mode, display virtual side mirror images taken by side mirror cameras installed in a virtual space based on a driving simulation.

The instrument panel display 53 displays an instrument panel image showing various instrument data such as the vehicle 13's driving speed and engine RPM, based on the video stream for the instrument panel supplied from the in-vehicle control device 18. In addition, in the driving mode, the instrument panel display 53 displays an actual instrument panel image based on the driving of the vehicle 13, and in the simulation mode, displays a virtual instrument panel image based on a driving simulation.

The rearview mirror display 54 displays, for example, a rearview mirror image taken toward the rear of the vehicle 13 by a rearview mirror camera installed on the rear of the vehicle 13, based on the rearview mirror image stream supplied from the in-vehicle control device 18. In addition, in the driving mode, the rearview mirror display 54 displays an actual rearview mirror image taken by a rearview mirror camera installed on the vehicle 13, and in the simulation mode, it displays a virtual rearview mirror image taken by a rearview mirror camera installed in a virtual space based on the driving simulation.

The rear seat displays 55L and 55R display rear seat viewpoint images, which are images captured by an external camera installed outside the vehicle 13 and are assumed to be seen through the rear seat displays 55L and 55R from the viewpoint of a passenger in the rear seat, based on the rear seat video stream supplied from the in-vehicle control device 18. In addition, in the driving mode, the rear seat displays 55L and 55R display actual rear seat viewpoint images captured by an external camera installed in the vehicle 13, and in the simulation mode, display virtual rear seat viewpoint images captured by an external camera installed in a virtual space based on the driving simulation.

The in-vehicle camera 56 photographs the occupants of the vehicle 13 to recognize their positions and postures, and supplies the image data to the in-vehicle control device 18.

In order to recognize the position and posture of the occupants of the vehicle 13, the in-vehicle sensors 57-1 and 57-2 detect the distance (depth) to the occupants and supply the sensor data obtained to the in-vehicle control device 18. Note that, when there is no need to distinguish between the in-vehicle sensors 57-1 and 57-2, hereinafter they will be referred to simply as the in-vehicle sensor 57. Also, in the example shown in FIG. 3, two in-vehicle sensors 57-1 and 57-2 are illustrated, but the position and posture of the occupants may be detected by one in-vehicle sensor 57 or three or more in-vehicle sensors 57.

Speakers 58-1 and 58-2 output audio based on an audio signal supplied from in-vehicle control device 18. For example, in driving mode, speakers 58-1 and 58-2 output audio played by an audio system installed in vehicle 13, and in simulation mode, output object audio emitted from objects in a virtual space based on a driving simulation.

With reference to Figure 5, an example of a method for identifying the occupant's viewpoint position P will be described.

For example, the garage system control device 16 recognizes the position and orientation of the vehicle 13 relative to each garage display 41 of the garage device 14 based on the image data supplied from the garage camera 42 and the sensor data supplied from the garage sensor 43. This allows the garage system control device 16 to obtain position and orientation data of the vehicle 13 represented by a rectangular parallelepiped space C inscribed by the vehicle 13, relative to the front garage display 41-2, for example, as shown in A of FIG. 5.

The in-vehicle control device 18 also recognizes the position and posture of the occupant relative to the vehicle 13, as well as the occupant's viewpoint position P (viewpoint position and line of sight direction) based on the video data supplied from the in-vehicle camera 56 and the sensor data supplied from the in-vehicle sensor 57. The in-vehicle control device 18 then acquires occupant position and posture data including information indicating the occupant's viewpoint position and line of sight direction in addition to information indicating the occupant's position and posture, and supplies this data to the garage system control device 16. This allows the garage system control device 16 to recognize the position, posture, viewpoint position, and line of sight direction of the occupant relative to the rectangular parallelepiped space C inscribed by the vehicle 13, as shown in FIG. 5B.

Therefore, the garage system control device 16 can, for example, determine the occupant's viewpoint position P relative to the front garage display 41-2 based on the rectangular space C relative to the front garage display 41-2 and the occupant's viewpoint position P relative to the rectangular space C.

It should be noted that the position and orientation data of the vehicle 13 and the position and orientation data of the occupants may be identified by methods other than those described here. For example, the position and orientation data of the vehicle 13 may be expressed in coordinates of the vehicle 13 centered on a part of the vehicle 13 (such as the position where the front right tire touches the ground). In this case, the position and orientation data of the vehicle 13 includes position information of the part of the vehicle 13 that corresponds to the center of the vehicle coordinate system.

<Configuration example of garage system control device>
An example of the configuration of the garage system control device 16 will be described with reference to FIGS.

FIG. 6 is a block diagram showing an example configuration of the garage system control device 16.

As shown in FIG. 6, the garage system control device 16 is configured with a sensor data acquisition unit 61, a video data acquisition unit 62, a vehicle position and attitude recognition unit 63, a communication unit 64, a virtual space storage unit 65, a 3DCG storage unit 66, a 3DCG generation unit 67, an object audio storage unit 68, an object audio generation unit 69, a user setting acquisition unit 70, a virtual space generation unit 71, a signal multiplexing unit 72, a video conversion processing unit 73, a ceiling surface video transmission unit 74, a front surface video transmission unit 75, a right side surface video transmission unit 76, a left side surface video transmission unit 77, a rear surface video transmission unit 78, and a floor surface video transmission unit 79.

The sensor data acquisition unit 61 acquires the sensor data supplied from the garage sensor 43 and supplies it to the vehicle position and attitude recognition unit 63.

The video data acquisition unit 62 acquires the video data provided by the garage camera 42 and provides it to the vehicle position and attitude recognition unit 63.

The vehicle position and attitude recognition unit 63 recognizes the position and attitude of the vehicle 13 stored in the garage device 14 based on the sensor data supplied from the sensor data acquisition unit 61 and the video data supplied from the video data acquisition unit 62, and acquires position and attitude data of the vehicle 13. For example, the vehicle position and attitude recognition unit 63 can acquire position and attitude data of the vehicle 13 represented by the rectangular parallelepiped space C in which the vehicle 13 is inscribed as described above with reference to FIG. 5. The vehicle position and attitude recognition unit 63 then supplies the position and attitude data of the vehicle 13 to the video conversion processing unit 73.

The communication unit 64 communicates with the in-vehicle control device 18 and an external network. For example, the communication unit 64 receives vehicle parameters transmitted from the in-vehicle control device 18, supplies position and posture data of occupants in the vehicle 13 contained in the vehicle parameters to the video conversion processing unit 73, and supplies driving operation data contained in the vehicle parameters to the virtual space generation unit 71. The communication unit 64 also transmits to the vehicle 13 the vehicle setting parameters supplied from the virtual space generation unit 71 and the video and audio stream for the vehicle 13 supplied from the signal multiplexing unit 72.

The virtual space storage unit 65 stores virtual space data consisting of the shapes and textures of roads, buildings, etc. that constitute the virtual space in which the vehicle 13 virtually travels during the driving simulation.

The 3DCG memory unit 66 stores 3DCG (3-Dimensional Computer Graphics) data consisting of shapes and textures representing various three-dimensional objects (e.g., other vehicles, pedestrians, traffic lights, etc.) that are placed in the virtual space in which the vehicle 13 virtually travels during the driving simulation.

The 3DCG generation unit 67 reads 3DCG data of objects to be placed near the vehicle 13 during the driving simulation from the 3DCG storage unit 66, generates the objects, and supplies them to the virtual space generation unit 71.

The object audio storage unit 68 stores audio data representing audio (e.g., the sounds of other vehicles traveling, pedestrian footsteps, traffic light melodies, etc.) emitted from various three-dimensional objects placed in the virtual space in which the vehicle 13 travels during the driving simulation.

The object audio generation unit 69 reads audio data corresponding to objects placed near the vehicle 13 in the driving simulation from the object audio storage unit 68, generates the audio, and supplies it to the virtual space generation unit 71.

When a user using the driving simulation system 11 performs various operational inputs to the garage system control device 16, the user setting acquisition unit 70 acquires and stores user setting values corresponding to the user's operational input, and supplies the user setting values to the virtual space generation unit 71. For example, when an operational input is performed to instruct the start of a driving simulation, the user setting acquisition unit 70 acquires and stores the user setting value that instructs the start of a driving simulation, and supplies the user setting value to the virtual space generation unit 71.

The virtual space generation unit 71 reads the virtual space data from the virtual space storage unit 65 to generate a virtual space, places the objects supplied from the 3DCG generation unit 67 within the virtual space, and sets the audio source supplied from the object audio generation unit 69 in correspondence with the position of each object.

Then, the virtual space generation unit 71 performs a driving simulation in which the vehicle 13 virtually runs in the virtual space according to the driving operation data supplied from the communication unit 64. As a result, the virtual space generation unit 71 determines the behavior of the vehicle 13 in the virtual space, generates vehicle setting parameters (suspension setting data, seat setting data, and reaction force setting data) for controlling the behavior of the vehicle 13 so as to reproduce that behavior, and supplies these to the communication unit 64.

Furthermore, the virtual space generation unit 71 generates a video stream for the vehicle 13, an audio stream for the vehicle 13, and a video stream for the garage according to the driving position of the vehicle 13 according to the driving operation data.

For example, the virtual space generation unit 71 generates a video stream for the vehicle 13 corresponding to the navigation image displayed on the navigation display 51 by placing a vehicle mark indicating the driving position of the vehicle 13 on a map image in the virtual space and moving the vehicle mark on the map image in accordance with the driving operation data. The virtual space generation unit 71 also generates a video stream for the vehicle 13 corresponding to the side mirror images displayed on the side mirror displays 52L and 52R by placing virtual cameras in the virtual space corresponding to the side mirror cameras installed on the left and right sides of the vehicle 13 and photographing the virtual space with each virtual camera. The virtual space generation unit 71 also generates a video stream for the vehicle 13 corresponding to the instrument panel image displayed on the instrument panel display 53 in accordance with various instrument data such as driving speed and engine speed in accordance with the driving operation data.

The virtual space generation unit 71 also generates a video stream for the vehicle 13, which serves as the rear-mirror image displayed on the rear-mirror display 54, by placing a virtual camera in the virtual space corresponding to the rear-mirror camera installed on the rear of the vehicle 13 and capturing an image of the virtual space with the virtual camera. The virtual space generation unit 71 also generates a video stream for the vehicle 13, which serves as the rear-seat viewpoint image displayed on the rear-seat displays 55L and 55R, by placing a virtual camera in the virtual space corresponding to the viewpoint of the rear-seat passengers and capturing an image of the virtual space seen through the rear-seat displays 55L and 55R with the virtual camera. The virtual space generation unit 71 then supplies the video stream for the vehicle 13 generated in this manner to the signal multiplexing unit 72.

The virtual space generation unit 71 also generates an audio stream for the vehicle 13 so that object audio is emitted from each object, with the positions of objects placed near the vehicle 13 traveling in the virtual space in accordance with the driving operation data being set as audio source positions, and supplies the audio stream to the video conversion processing unit 73.

The virtual space generation unit 71 also places a virtual camera in the virtual space so as to correspond to a predetermined position of the vehicle 13 (for example, the center position of the vehicle 13), and uses the virtual camera to capture images of the virtual space in all directions centered on the vehicle 13, thereby generating a video stream for the garage, and supplies this to the video conversion processing unit 73.

The signal multiplexing unit 72 multiplexes the video stream for the vehicle 13 and the audio stream for the vehicle 13 supplied from the virtual space generating unit 71 to generate a video/audio stream for the vehicle 13 and supplies it to the communication unit 64.

The image conversion processing unit 73 performs image conversion processing on the image stream for the garage supplied from the virtual space generation unit 71, based on the position and orientation data of the vehicle 13 supplied from the vehicle position and orientation recognition unit 63 and the position and orientation data of the occupants in the vehicle 13 supplied from the communication unit 64. For example, the image conversion processing unit 73 has a viewpoint detection unit 81 and a geometric transformation unit 82.

The viewpoint detection unit 81, as described above with reference to FIG. 5, sets a rectangular parallelepiped space C in which the vehicle 13 is inscribed at the position of the vehicle 13 relative to the garage display 41 based on the position and orientation data of the vehicle 13. The viewpoint detection unit 81 then sets the position, orientation, viewpoint, and line of sight of the occupant in the rectangular parallelepiped space C based on the position and orientation data of the occupant inside the vehicle 13, thereby being able to determine the position, orientation, viewpoint, and line of sight of the occupant relative to the garage display 41. This allows the viewpoint detection unit 81 to supply information about the viewpoint and line of sight of the occupant relative to the garage display 41 to the geometric transformation unit 82.

The geometric transformation unit 82 performs a geometric transformation on the garage video stream supplied from the virtual space generation unit 71 based on the position of each garage display 41 and the occupant's viewpoint and line of sight relative to the garage display 41. That is, the geometric transformation unit 82 performs a geometric transformation on the garage video stream so as to project the garage video stream using each garage display 41 as a projection surface, based on the occupant's viewpoint and line of sight identified by the viewpoint detection unit 81, thereby generating a video stream for each garage display 41. In this way, a spherical garage video stream centered on the vehicle 13 is projected onto, for example, the ceiling garage display 41-1, thereby generating a planar ceiling video stream. Similarly, planar video streams are generated for each of the other garage displays 41.

For example, as shown in FIG. 7, if you want to represent a virtual plane viewed from viewpoint position P on the garage display 41, you can draw a straight line equivalent to a ray that passes from viewpoint position P through a point on the virtual plane you want to represent, and the intersections on the garage display 41 of the lines that pass through each point on the virtual plane become perspective projection points, making it possible to identify the pixels of the garage display 41. In this case, since the viewpoint position P and the position of the garage display 41 are found by calculation, their respective positions and orientations need to be precisely quantified, such as coordinate values based on a common coordinate system. For example, these processes can easily generate an image to be displayed on the garage display 41 placed according to the position and orientation of the camera taking the image by utilizing mechanisms that are already implemented in game engines.

The ceiling surface video transmission unit 74 transmits the ceiling surface video stream supplied from the video conversion processing unit 73 to the ceiling surface garage display 41-1, causing the ceiling surface garage display 41-1 to display the ceiling surface simulation video.

The front image transmission unit 75 transmits the front image stream supplied from the image conversion processing unit 73 to the front garage display 41-2, causing the front garage display 41-2 to display the front simulation image.

The right side image transmission unit 76 transmits the right side image stream supplied from the image conversion processing unit 73 to the right side garage display 41-3, causing the right side garage display 41-3 to display the right side simulation image.

The left side image transmission unit 77 transmits the left side image stream supplied from the image conversion processing unit 73 to the left side garage display 41-4, causing the left side garage display 41-4 to display the left side simulation image.

The rear view video transmission unit 78 transmits the rear view video stream supplied from the video conversion processing unit 73 to the rear view garage display 41-5, causing the rear view garage display 41-5 to display the rear view simulation video.

The floor surface image transmission unit 79 transmits the floor surface image stream supplied from the image conversion processing unit 73 to the floor surface garage display 41-6, causing the floor surface garage display 41-6 to display the floor surface simulation image.

<Configuration example of in-vehicle control device>
An example of the configuration of the in-vehicle control device 18 will be described with reference to FIGS.

FIG. 8 is a block diagram showing an example of the configuration of the operation system and drive system of the in-vehicle control device 18.

As shown in FIG. 8, the in-vehicle control device 18 is configured to include a sensor data acquisition unit 91, a video data acquisition unit 92, a sensor data acquisition unit 93, a video data acquisition unit 94, a vehicle interior environment recognition unit 95, a vehicle exterior environment recognition unit 96, a communication unit 97, a user setting acquisition unit 98, a vehicle stop state determination unit 99, a brake position data acquisition unit 100, a steering wheel rotation data acquisition unit 101, an accelerator position data acquisition unit 102, a driving operation detection unit 103, a driving mode control unit 104, a simulation mode control unit 105, an axle direction control unit 106, a throttle control unit 107, a brake control unit 108, a steering wheel reaction force control unit 109, a suspension control unit 110, a brake reaction force control unit 111, an accelerator reaction force control unit 112, and a seat control unit 113.

The sensor data acquisition unit 91 acquires sensor data provided from the in-vehicle sensor 57 and provides it to the vehicle interior environment recognition unit 95.

The video data acquisition unit 92 acquires video data provided by the in-vehicle camera 56 and provides it to the vehicle interior environment recognition unit 95.

The sensor data acquisition unit 93 acquires sensor data provided from an external sensor (not shown) installed outside the vehicle 13 and provides it to the vehicle external environment recognition unit 96.

The video data acquisition unit 94 acquires video data provided from an external camera (not shown) installed outside the vehicle 13 and provides it to the vehicle external environment recognition unit 96.

The vehicle interior environment recognition unit 95 generates position and orientation data of the occupants in the vehicle 13 based on the sensor data supplied from the sensor data acquisition unit 91 and the video data supplied from the video data acquisition unit 92, and supplies the data to the communication unit 97. For example, the position and orientation data of the occupants in the vehicle 13 includes information indicating the viewpoint position and line of sight direction of the occupants in the vehicle 13 relative to the rectangular parallelepiped space C inscribed by the vehicle 13, as described above with reference to FIG. 5.

The vehicle external environment recognition unit 96 generates vehicle external environment data based on the sensor data supplied from the sensor data acquisition unit 93 and the video data supplied from the video data acquisition unit 94, and supplies the data to the stopped state determination unit 99 and the driving mode control unit 104. For example, the vehicle external environment data includes information indicating the distance to objects around the vehicle 13, and when the vehicle 13 is stored in the garage device 14, includes information indicating the distance to the garage display 41.

The communication unit 97 communicates with the garage system control device 16 and the external network. For example, the communication unit 97 transmits a stopped state notification provided from the stopped state determination unit 99 to the garage system control device 16, and receives a garage state notification transmitted from the garage system control device 16 and supplies it to the stopped state determination unit 99. The communication unit 97 also transmits vehicle parameters to the garage system control device 16, including position and posture data of the occupants in the vehicle 13 provided from the vehicle internal environment recognition unit 95 and driving operation data provided from the driving operation detection unit 103. The communication unit 97 then receives vehicle setting parameters transmitted from the garage system control device 16 and supplies them to the simulation mode control unit 105.

When a user using the driving simulation system 11 performs various operational inputs to the in-vehicle control device 18, the user setting acquisition unit 98 acquires and stores a user setting value corresponding to the operational input, and supplies the user setting value to the stopped state determination unit 99. For example, when an operational input is performed to instruct switching from the driving mode to the simulation mode, the user setting acquisition unit 98 acquires and stores a user setting value instructing switching from the driving mode to the simulation mode, and supplies the user setting value to the stopped state determination unit 99.

When the user setting value instructing switching from the driving mode to the simulation mode is supplied from the user setting acquisition unit 98, the stopped state determination unit 99 determines whether or not to transition to the simulation mode according to the vehicle external environment data supplied from the vehicle external environment recognition unit 96. The stopped state determination unit 99 also transmits a stopped state notification to the garage system control device 16 via the communication unit 97, and receives a garage state notification transmitted from the garage system control device 16 via the communication unit 97. The stopped state determination unit 99 also supplies a stopped state signal to the driving mode control unit 104 and the simulation mode control unit 105.

For example, when the vehicle 13 is stopped and the user performs an operation to instruct switching from the driving mode to the simulation mode, the stopped state determination unit 99 determines that the vehicle will transition to the simulation mode. The stopped state determination unit 99 may also determine that the vehicle will transition to the simulation mode when the vehicle 13 is stopped and the gear selector of the vehicle 13 is in the parking range.

Furthermore, the stopped state determination unit 99 may make a determination based on the position information of the vehicle 13 using, for example, GPS (Global Positioning System) or the like. In this case, the stopped state determination unit 99 stores in advance the position information of a location where a driving simulation may be performed (for example, the installation location of the garage device 14), and can determine to transition to simulation mode when the vehicle 13 is stopped at a location where a driving simulation may be performed based on the position information of the vehicle 13.

The stopped state determination unit 99 may also make a determination using, for example, the distance or relative position from the garage display 41. In this case, the garage state notification includes the distance or relative position of the vehicle 13 from the garage display 41. The stopped state determination unit 99 can then determine to transition to simulation mode when the distance or relative position of the vehicle 13 from the garage display 41 is equal to or less than a predetermined threshold, that is, when the vehicle 13 is stopped at a predetermined position within the garage device 14.

In this way, by having the vehicle stop state determination unit 99 determine whether or not to transition to simulation mode, it is possible to avoid a situation in which it becomes impossible to drive the vehicle 13 in an unexpected location, for example. For example, even if an operation to start the simulation mode is mistakenly performed while the vehicle is temporarily stopped during a drive, it is possible to avoid a situation in which it becomes impossible to suddenly operate the vehicle 13.

In addition, if it is not permitted to switch to simulation mode, a message stating "Cannot switch to simulation mode" may be displayed on the display inside the vehicle 13 based on the result of the judgment by the stopped state judgment unit 99. At that time, the reason for this may also be displayed. For example, the reason may be that the vehicle 13 is not stopped, the gear selector of the vehicle 13 is not in the parking range, the position of the vehicle 13 is not in an appropriate location (such as a garage, a parking lot, a parking space, or a predetermined position relative to the driving simulation system 11), etc.

The brake position data acquisition unit 100 acquires brake position data indicating the amount of depression of the brake pedal 23 by the driver of the vehicle 13 based on the brake position signal supplied from the brake position sensor 29 in FIG. 2, and supplies the data to the driving operation detection unit 103.

The steering wheel rotation data acquisition unit 101 acquires steering wheel rotation data indicating the amount of rotation of the steering wheel 21 by the driver of the vehicle 13 based on the steering wheel rotation signal supplied from the steering wheel rotation sensor 27 in FIG. 2, and supplies the data to the driving operation detection unit 103.

The accelerator position data acquisition unit 102 acquires accelerator position data indicating the amount of depression of the accelerator pedal 22 by the driver of the vehicle 13 based on the accelerator position signal supplied from the accelerator position sensor 28 in FIG. 2, and supplies this data to the driving operation detection unit 103.

The driving operation detection unit 103 detects the driver's driving operation of the vehicle 13 based on the brake position data supplied from the brake position data acquisition unit 100, the steering wheel rotation data supplied from the steering wheel rotation data acquisition unit 101, and the accelerator position data supplied from the accelerator position data acquisition unit 102. The driving operation detection unit 103 then supplies driving operation data indicating the content of the driving operation (brake position data, steering wheel rotation data, and accelerator position data) to the driving mode control unit 104 and the communication unit 97.

When in the driving mode, the driving mode control unit 104 controls the driving of the vehicle 13 according to the driving operation data supplied from the driving operation detection unit 103. For example, the driving mode control unit 104 generates axle direction control data for controlling the axle direction of the tires of the vehicle 13 according to the steering wheel rotation data, and supplies the data to the axle direction control unit 106. The driving mode control unit 104 also generates throttle control data for controlling the acceleration of the vehicle 13 according to the accelerator position data, and supplies the data to the throttle control unit 107. The driving mode control unit 104 also generates brake control data for controlling the deceleration of the vehicle 13 according to the brake position data, and supplies the data to the brake control unit 108.

Furthermore, the driving mode control unit 104 generates steering reaction force control data, suspension control data, brake reaction force control data, accelerator reaction force control data, and seat control data according to the driving operation data supplied from the driving operation detection unit 103 and the vehicle external environment data supplied from the vehicle external environment recognition unit 96. The driving mode control unit 104 then supplies each of the control data to the steering reaction force control unit 109, the suspension control unit 110, the brake reaction force control unit 111, the accelerator reaction force control unit 112, and the seat control unit 113.

In simulation mode, the simulation mode control unit 105 controls the behavior of the vehicle 13 in a virtual space according to the vehicle setting parameters supplied from the communication unit 97.

For example, the vehicle setting parameters include suspension setting data, seat setting data, and reaction force setting data (data values for steering reaction force, brake reaction force, and accelerator reaction force). Therefore, the simulation mode control unit 105 generates suspension control data that controls the height of the suspension of each of the four wheels of the vehicle 13 according to the suspension setting data, and supplies it to the suspension control unit 110. The simulation mode control unit 105 also generates seat control data that controls the inclination of the seat cushion and backrest according to the seat setting data, and supplies it to the seat control unit 113.

The simulation mode control unit 105 also generates steering reaction force control data for controlling the steering reaction force generated in the steering wheel 21 according to the steering reaction force data value in the reaction force setting data, and supplies this to the steering reaction force control unit 109. The simulation mode control unit 105 also generates brake reaction force control data for controlling the brake reaction force generated in the brake pedal 23 according to the brake reaction force data value in the reaction force setting data, and supplies this to the brake reaction force control unit 111. The simulation mode control unit 105 also generates accelerator reaction force control data for controlling the accelerator reaction force generated in the accelerator pedal 22 according to the accelerator reaction force data value in the reaction force setting data, and supplies this to the accelerator reaction force control unit 112.

The axle direction control unit 106 generates an axle direction control signal according to the axle direction control data supplied from the driving mode control unit 104, and supplies it to the axle direction drive mechanism 30 in FIG. 3.

The throttle control unit 107 generates a throttle control signal according to the throttle control data supplied from the driving mode control unit 104, and supplies it to the throttle drive mechanism 31 in FIG. 3.

The brake control unit 108 generates a brake control signal according to the brake control data supplied from the driving mode control unit 104, and supplies it to the brake drive mechanism 32 in FIG. 3.

The steering reaction force control unit 109 generates a steering reaction force control signal according to steering reaction force control data supplied from the driving mode control unit 104 or the simulation mode control unit 105, and supplies it to the steering reaction force drive mechanism 33 in FIG. 3.

The suspension control unit 110 generates a suspension control signal according to the suspension control data supplied from the driving mode control unit 104 or the simulation mode control unit 105, and supplies it to the suspension drive mechanism 36 in FIG. 3.

The brake reaction force control unit 111 generates a brake reaction force control signal according to the brake reaction force control data supplied from the driving mode control unit 104 or the simulation mode control unit 105, and supplies it to the brake reaction force drive mechanism 35 in FIG. 3.

The accelerator reaction force control unit 112 generates an accelerator reaction force control signal according to the accelerator reaction force control data supplied from the driving mode control unit 104 or the simulation mode control unit 105, and supplies it to the accelerator reaction force drive mechanism 34 in FIG. 3.

The seat control unit 113 generates a seat control signal according to the seat control data supplied from the driving mode control unit 104 or the simulation mode control unit 105, and supplies it to the seat drive mechanism 37 in FIG. 3.

FIG. 9 shows a block diagram illustrating an example of the configuration of the video and audio systems of the in-vehicle control device 18.

As shown in FIG. 9, the in-vehicle control device 18 is configured with a user setting acquisition unit 121, a communication unit 122, a video acquisition unit 123, an audio acquisition unit 124, a signal separation unit 125, a video input switching unit 126, an audio input switching unit 127, a video signal processing unit 128, a navigation video transmission unit 129, a side mirror video transmission unit 130, an instrument panel video transmission unit 131, a rearview mirror video transmission unit 132, a rear seat video transmission unit 133, an audio signal processing unit 134, and an analog amplification unit 135.

When a user using the driving simulation system 11 performs various operational inputs to the in-vehicle control device 18, the user setting acquisition unit 121 acquires and stores a user setting value corresponding to the user's operational input, and supplies the user setting value to the communication unit 122, the video input switching unit 126, and the audio input switching unit 127. For example, when an operational input is performed to instruct switching from the driving mode to the simulation mode, the user setting acquisition unit 121 acquires and stores a user setting value instructing switching from the driving mode to the simulation mode, and supplies the user setting value to the communication unit 122, the video input switching unit 126, and the audio input switching unit 127.

The communication unit 122 communicates with the garage system control device 16 and an external network. For example, the communication unit 122 transmits a user setting value supplied from the user setting acquisition unit 121 to the garage system control device 16, the user setting value instructing switching from the driving mode to the simulation mode. The communication unit 122 also receives a video and audio stream for the vehicle 13 transmitted from the garage system control device 16, and supplies the video and audio stream to the signal separation unit 125.

In driving mode, the image acquisition unit 123 acquires images to be displayed on the navigation display 51, the side mirror displays 52L and 52R, the instrument panel display 53, the rearview mirror display 54, and the rear seat displays 55L and 55R, and supplies these image streams to the image input switching unit 126. For example, the image acquisition unit 123 acquires navigation images according to the position information of the vehicle 13, side mirror images captured by side mirror cameras installed on the left and right sides of the vehicle 13, instrument panel images based on the vehicle 13's traveling speed and engine RPM, rearview mirror images captured by the rearview mirror camera of the vehicle 13, rear seat viewpoint images based on images captured by an external camera of the vehicle 13, etc.

The audio acquisition unit 124 acquires audio played on the audio system installed in the vehicle 13 during driving mode, and supplies the audio stream to the audio input switching unit 127.

The signal separation unit 125 separates the video and audio stream for vehicle 13 supplied from the communication unit 122 into a video stream for vehicle 13 and an audio stream for vehicle 13. The signal separation unit 125 then supplies the video stream for vehicle 13 to the video input switching unit 126, and supplies the audio stream for vehicle 13 to the audio input switching unit 127.

The video input switching unit 126 switches the video stream supplied to the video signal processing unit 128 in accordance with the user setting value supplied from the user setting acquisition unit 121. For example, when a user setting value instructing switching from the driving mode to the simulation mode is supplied to the video input switching unit 126, the video input switching unit 126 switches so that the video stream for the vehicle 13 supplied from the signal separation unit 125 is supplied to the video signal processing unit 128. On the other hand, when a user setting value instructing switching from the simulation mode to the driving mode is supplied to the video input switching unit 126, the video input switching unit 126 switches so that the video stream supplied from the video acquisition unit 123 is supplied to the video signal processing unit 128.

The audio input switching unit 127 switches the audio stream supplied to the audio signal processing unit 134 in accordance with the user setting value supplied from the user setting acquisition unit 121. For example, when a user setting value instructing switching from the driving mode to the simulation mode is supplied to the audio input switching unit 127, the audio input switching unit 127 switches so that the audio stream for the vehicle 13 supplied from the signal separation unit 125 is supplied to the audio signal processing unit 134. On the other hand, when a user setting value instructing switching from the simulation mode to the driving mode is supplied to the audio input switching unit 127, the audio input switching unit 127 switches so that the audio stream supplied from the audio acquisition unit 124 is supplied to the audio signal processing unit 134.

The video signal processing unit 128 performs signal processing on the video streams supplied from the video input switching unit 126 to obtain a video stream for navigation, a video stream for side mirrors, a video stream for the instrument panel, a video stream for the rearview mirror, and a video stream for the rear seats. The video signal processing unit 128 then supplies the video stream for navigation to the navigation video transmission unit 129, the video stream for the side mirrors to the side mirror video transmission unit 130, the video stream for the instrument panel to the instrument panel video transmission unit 131, the video stream for the rearview mirror to the rearview mirror video transmission unit 132, and the video stream for the rear seats to the rear seat video transmission unit 133.

The navigation video transmission unit 129 transmits the video stream for navigation provided by the video signal processing unit 128 to the navigation display 51, causing the navigation display 51 to display the navigation video.

The side mirror image transmission unit 130 transmits the side mirror image stream supplied from the image signal processing unit 128 to the side mirror displays 52L and 52R, and displays the side mirror image on the side mirror displays 52L and 52R.

The instrument panel image transmission unit 131 transmits the image stream for the instrument panel supplied from the image signal processing unit 128 to the instrument panel display 53, causing the instrument panel display 53 to display the instrument panel image.

The rearview mirror image transmission unit 132 transmits the rearview mirror image stream supplied from the image signal processing unit 128 to the rearview mirror display 54, causing the rearview mirror display 54 to display the rearview mirror image.

The rear seat video transmission unit 133 transmits the rear seat video stream supplied from the video signal processing unit 128 to the rear seat displays 55L and 55R, and displays the rear seat viewpoint video on the rear seat displays 55L and 55R.

The audio signal processing unit 134 performs signal processing on the audio stream supplied from the audio input switching unit 127 to obtain an audio signal and supply it to the analog amplification unit 135.

The analog amplifier 135 amplifies the audio signal supplied from the audio signal processor 134 and supplies it to the speakers 58-1 and 58-2, causing the speakers 58-1 and 58-2 to output audio.

<Examples of vehicle parameters and vehicle setting parameters>
The vehicle parameters and the vehicle setting parameters will be described with reference to FIGS.

A in FIG. 10 shows an example of the data structure of vehicle parameters transmitted from the in-vehicle control device 18 to the garage system control device 16. B in FIG. 10 shows an example of the data structure of vehicle setting parameters transmitted from the garage system control device 16 to the in-vehicle control device 18.

As shown in A of FIG. 10, the vehicle parameters are configured by sequentially arranging steering wheel rotation data, brake position data, accelerator position data, and other data following a header section in which all common data for the vehicle parameters is stored. Furthermore, each of the steering wheel rotation data, brake position data, and accelerator position data has a header section and a data storage section. Each data storage section stores one or more data values associated with each time, and in the example shown in A of FIG. 10, x data values (for example, data value 1a to data value xa at time a) are stored associated with each time from time a to time y.

As shown in FIG. 10B, the vehicle setting parameters are configured by sequentially arranging suspension setting data, seat setting data, reaction force setting data, and other data following a header section in which all common data for the vehicle setting parameters is stored. Furthermore, the suspension setting data, seat setting data, and reaction force setting data each have a header section and a data storage section. Each data storage section stores one or more data values associated with each time. In the example shown in FIG. 10B, x data values (for example, data value 1a to data value xa at time a) are stored in association with each time from time a to time y.

In this example, FIG. 12 shows an example of vehicle parameters and vehicle setting parameters that are generated at times T1 to T4 for a vehicle 13 that has experienced a spinning behavior due to excessive steering as shown in FIG. 11.

As shown in FIG. 12, the vehicle parameters input to the garage system control device 16 store one data value each for the accelerator position data, brake position data, and steering wheel rotation data for each time T.

Furthermore, in the vehicle setting parameters used by the garage system control device 16 to control the vehicle 13, four data values (Front RIGHT, Front LEFT, Rear RIGHT, Rear LEFT) are stored for the suspension setting data for each time T. Similarly, three data values (Seat Control 1, Seat Control 2, Seat Control 3) are stored for the seat setting data, and two data values (steering reaction force, brake reaction force) are stored for the reaction force setting data.

As shown in A of FIG. 13, the data value Front RIGHT is suspension setting data for controlling the suspension height of the right front wheel, the data value Front LEFT is suspension setting data for controlling the suspension height of the left front wheel, the data value Rear RIGHT is suspension setting data for controlling the suspension height of the right rear wheel, and the data value Rear LEFT is suspension setting data for controlling the suspension height of the left rear wheel. Then, as shown in B of FIG. 13, the suspension setting data is set so that the right suspension is lowered and the left suspension is raised during left cornering. Also, as shown in C of FIG. 13, the suspension setting data is set so that the front suspension is lowered and the rear suspension is raised during braking. Also, as shown in D of FIG. 13, the suspension setting data is set so that the rear suspension is lowered and the front suspension is raised during acceleration.

As shown in A of FIG. 14, the data value Seat Control 1 is seat setting data for controlling the height of the front end of the seat cushion, the data value Seat Control 2 is seat setting data for controlling the height of the rear end of the seat cushion, and the data value Seat Control 3 is seat setting data for controlling the forward or rearward tilt of the seat back. Then, as shown in B of FIG. 14, the seat setting data is set so that no control is performed on the seat cushion and the seat back during constant speed driving. Also, as shown in C of FIG. 14, the seat setting data is set so that the height of the front end of the seat cushion is lowered, the height of the rear end of the seat cushion is raised, and the seat back is tilted forward during braking.

<Examples of using driving simulation>
Here, a first and a second application example of the driving simulation will be described.

As a first example of the use of driving simulation, the driving simulation system 11 can recreate the riding conditions of an actual vehicle 13 using the actual vehicle 13, and can therefore be used to learn correct driving techniques under specific conditions.

For example, when the occupants are made to simulate driving in a heavy rain situation where the road surface is extremely slippery, the garage display 41 in the garage device 14 presents an image that mimics the scenery outside the vehicle 13. Specifically, an image of the scenery moving in a certain direction when driving at 40 km/h is displayed on the garage display 41-1 for the ceiling surface, the garage display 41-2 for the front, the garage display 41-3 for the right side, the garage display 41-4 for the left side, and the garage display 41-5 for the rear, thereby creating a visual state in which the occupants of the vehicle 13 visually feel as if they are driving in that scene.

At that time, the side mirror displays 52L and 52R and the rearview mirror display 54, which correspond to the rearview mirror and side mirrors mounted on the vehicle 13, also display images of the vehicle as if it were being driven, in the same way as the garage display 41. The occupants of the vehicle 13 can then follow the instructions of the driving simulation and actually operate the steering wheel, accelerator, and brakes in a given scene, for example, in their own actual vehicle 13. At this time, because the vehicle 13 itself employs a by-wire system, the actual rudder does not move even if the steering wheel is operated, and no braking operation or brake pad piston movement occurs, and driving operation data indicating the amount of each operation is sent to the garage system control device 16.

The garage system control device 16 generates and displays an image of the occupants of the vehicle 13 as if they were driving on not only the garage display 41 but also the side mirror displays 52L and 52R and the rearview mirror display 54 mounted on the vehicle, according to the amount of operation obtained from the driving operation data. At this time, the garage system control device 16 creates a scene in which the driving is taking place in a virtual space, places a virtual camera in the virtual space at a position corresponding to the line of sight of the occupants, and can appropriately display the image captured there on the garage display 41, the side mirror displays 52L and 52R, and the rearview mirror display 54. At this time, in order to generate an appropriate display image, position and orientation data of the vehicle 13 and the occupants in the vehicle 13 with respect to the position of the garage display 41 are required.

Furthermore, the vehicle 13 is equipped with a suspension function that allows the height of each of the four wheels to be changed independently, so that the height of each of the four wheels can be changed according to information from the driving simulation. This makes it possible to reproduce rolling when going around a curve, or diving when braking rapidly, and to provide the occupants with a more realistic situation, for example, by expressing a state of slipping on a rainy day. In addition, by equipping the seats in which the occupants sit with various adjustment mechanisms, it is possible to move them appropriately according to the situation, and by working in coordination with the control of the suspension height function, it is possible to make the occupants feel the acceleration.

In this case, the garage system control device 16 can record various operations performed by the occupant as an operation log, and by analyzing these on a server or the like, it is possible to perform a detailed analysis of the problem with the occupant's driving technique. This can also be obtained during actual driving, but in that case, it is necessary to accurately recognize the state of the driving scene, and it is expected that there may be high hurdles to accurately judge the driving technique.

In contrast, the driving simulation system 11 can envision in advance the scenes to be presented in the driving simulation, and can obtain data on how the occupants will drive in those scenes or in specific conditions, making it possible to make more accurate judgments. Of course, here too, by using the vehicle 13 that the driver owns, rather than a general driving simulator that uses a separate device, more accurate data can be collected.

The second use case for driving simulation is to help people learn ideal driving techniques for specific situations, such as avoiding danger.

For example, the steering wheel 21, accelerator pedal 22, brake pedal 23, etc. of the vehicle 13 are by-wire, and at the same time, each of them can generate a reaction force for each operation using an actuator, etc., which can be effectively used to learn driving techniques.

Specifically, in a certain situation, when an occupant operates the steering wheel 21, accelerator pedal 22, or brake pedal 23 in a manner that is far removed from the teaching data that indicates correct driving, a large reaction force can be applied to the steering wheel 21, for example, to allow the occupant to experience correct steering operation. Similarly, by applying a reaction force to braking and accelerator operation, the occupant can experience correct driving operation, and can learn correct driving operation in heavy rain.

Again, unlike general driving simulations, the experience can be carried out in one's own vehicle 13, so it is expected that extremely efficient learning of driving techniques can be achieved. In addition, the teacher data can be changed depending on the skill level of the occupant's driving operation technique.

In other words, when learning how to avoid danger on slippery roads, such as on rainy days, if the driver's skill is extremely high and this is determined during the driving simulation as in the first use case described above, it is expected that the driver will learn how to counteract handling, braking, and accelerating operations when avoiding danger by experiencing the reaction forces of each. On the other hand, if it is determined that the driver's skill is relatively low, it is expected that the driver will learn how to perform minimal steering, braking, and accelerating operations by experiencing the reaction forces of each.

<Example of driving simulation processing>
15 to 21, a process example of the driving simulation executed in the driving simulation system 11 will be described.

Figure 15 is a flowchart explaining the initial setup process on the garage system 12 side.

For example, when a user performs an operation input to the user setting acquisition unit 70 of the garage system control device 16 to instruct the start of a driving simulation, the initial setup process on the garage system 12 side is started.

In step S11, the communication unit 64 of the garage system control device 16 determines whether communication is possible between the communication unit 97 and the communication unit 122 of the in-vehicle control device 18, and waits until it determines that communication is possible between the communication unit 97 and the communication unit 122 of the in-vehicle control device 18. Then, if the communication unit 64 determines that communication is possible between the communication unit 97 and the communication unit 122 of the in-vehicle control device 18, the process proceeds to step S12.

In step S12, the communication unit 64 of the garage system control device 16 starts communication between the communication unit 97 and the communication unit 122 of the in-vehicle control device 18.

In step S13, the vehicle position and attitude recognition unit 63 of the garage system control device 16 executes a vehicle position and attitude recognition process (see FIG. 19) to recognize the position and attitude of the vehicle 13, and obtains position and attitude data of the vehicle 13.

In step S14, the communication unit 64 of the garage system control device 16 receives the vehicle data (various data related to the vehicle 13 required to perform the driving simulation) sent from the communication unit 97 of the in-vehicle control device 18, and supplies it to the video conversion processing unit 73.

In step S15, the communication unit 64 of the garage system control device 16 receives the position and posture data of the occupants in the vehicle 13 transmitted from the communication unit 97 of the in-vehicle control device 18, and supplies it to the image conversion processing unit 73.

In step S16, the communication unit 64 of the garage system control device 16 transmits setup parameters for setting up the display system, audio system, etc. of the vehicle 13 to the in-vehicle control device 18.

In step S17, the communication unit 64 of the garage system control device 16 determines whether the setup of the vehicle 13 is complete, and waits until it is determined that the setup of the vehicle 13 is complete. For example, when the communication unit 64 receives a vehicle setup completion notification sent from the communication unit 122 of the in-vehicle control device 18 in step S32 of FIG. 16 described below, it determines that the setup of the vehicle 13 is complete, and the initial setup process on the garage system 12 side is terminated.

FIG. 16 is a flowchart explaining the initial setup process on the vehicle 13 side.

For example, when the in-vehicle control device 18 is started, the initial setup process on the vehicle 13 side is started, and at the start of the process, the vehicle 13 is set to a driving mode.

In step S21, the vehicle stop state determination unit 99 determines whether or not to switch from the driving mode to the simulation mode. For example, when a user setting value instructing the switching from the driving mode to the simulation mode is supplied from the user setting acquisition unit 98, the vehicle stop state determination unit 99 can determine that the switching from the driving mode to the simulation mode will be performed.

If the vehicle stop state determination unit 99 determines in step S21 that the driving mode should not be switched to the simulation mode, the process proceeds to step S22, and the vehicle 13 continues in the driving mode. After that, the process returns to step S21, and the same process is repeated.

On the other hand, if the stopped state determination unit 99 determines in step S21 that the mode should be switched from the driving mode to the simulation mode, the process proceeds to step S23.

In step S23, the vehicle stop state determination unit 99 determines whether the vehicle 13 is stopped based on the vehicle external environment data supplied from the vehicle external environment recognition unit 96, and waits until it determines that the vehicle 13 is stopped. Then, if the vehicle stop state determination unit 99 determines that the vehicle 13 is stopped, the process proceeds to step S24.

In step S24, the communication unit 97 and the communication unit 122 of the in-vehicle control device 18 determine whether communication with the communication unit 64 of the garage system control device 16 is possible, and waits until it is determined that communication with the communication unit 64 of the garage system control device 16 is possible. Then, if the communication unit 97 and the communication unit 122 determine that communication with the communication unit 64 of the garage system control device 16 is possible, the process proceeds to step S25.

In step S25, the communication unit 97 and the communication unit 122 of the in-vehicle control device 18 start communication with the communication unit 64 of the garage system control device 16.

In step S26, the communication unit 97 of the in-vehicle control device 18 transmits the vehicle data to the communication unit 97 of the in-vehicle control device 18.

In step S27, the vehicle interior environment recognition unit 95 executes an occupant position and posture recognition process (see FIG. 20) to recognize the positions and postures of occupants inside the vehicle 13, and obtains position and posture data of the occupants inside the vehicle 13.

In step S28, the vehicle interior environment recognition unit 95 supplies the position and attitude data of the occupant inside the vehicle 13 acquired in the occupant position and attitude recognition process in step S27 to the communication unit 97, and the communication unit 97 transmits the position and attitude data of the occupant inside the vehicle 13 to the garage system control device 16.

In step S29, the communication unit 122 of the in-vehicle control device 18 receives the setup parameters sent from the garage system control device 16 in step S16 of FIG. 15.

In step S30, the communication unit 122 of the in-vehicle control device 18 supplies the setup parameters received in step S29 to the display system, audio system, etc. of the vehicle 13, and executes their setup.

In step S31, switching to the by-wire system is performed, and control is shifted from the driving mode control unit 104 to the simulation mode control unit 105.

In step S32, the communication unit 97 and the communication unit 122 of the in-vehicle control device 18 send a vehicle setup completion notification to the garage system control device 16 to notify that the setup of the vehicle 13 has been completed, and the initial setup process on the vehicle 13 side is terminated.

FIG. 17 is a flowchart explaining the driving simulation operation processing on the garage system 12 side.

For example, when the initial setup process on the garage system 12 side in FIG. 15 is completed, the driving simulation operation process on the garage system 12 side is started.

In step S41, the communication unit 64 of the garage system control device 16 determines whether or not the vehicle parameters have been received, and waits to process until it is determined that the vehicle parameters have been received. For example, when the communication unit 64 receives the vehicle parameters transmitted from the communication unit 97 of the in-vehicle control device 18 in step S69 of FIG. 18 described below, it supplies the driving operation data contained in the vehicle parameters to the virtual space generation unit 71, and supplies the position and posture data of the occupants in the vehicle 13 contained in the vehicle parameters to the image conversion processing unit 73. The communication unit 64 then determines that the vehicle parameters have been received, and the process proceeds to step S42.

In step S42, the virtual space generation unit 71 generates suspension setting data as described above with reference to FIG. 13, based on the behavior of the vehicle 13 (cornering, acceleration, deceleration, etc.) determined by the driving simulation according to the driving operation data supplied in step S41.

In step S43, the virtual space generation unit 71 generates seat setting data as described above with reference to FIG. 14, based on the behavior of the vehicle 13 (when driving at a constant speed, when braking, etc.) determined by a driving simulation according to the driving operation data supplied in step S41.

In step S44, the virtual space generation unit 71 calculates the steering reaction force generated in the steering wheel 21, the accelerator reaction force generated in the accelerator pedal 22, and the brake reaction force generated in the brake pedal 23 based on the behavior of the vehicle 13 determined by a driving simulation in accordance with the driving operation data supplied in step S41. The virtual space generation unit 71 then generates reaction force setting data in which the steering reaction force, accelerator reaction force, and brake reaction force are stored.

In step S45, the virtual space generation unit 71 supplies the vehicle setting parameters, which are composed of the suspension setting data generated in step S42, the seat setting data generated in step S43, and the reaction force setting data generated in step S44, to the communication unit 64. The communication unit 64 then transmits the vehicle setting parameters to the in-vehicle control device 18.

In step S46, the virtual space generation unit 71 generates rendering parameters based on the steering wheel rotation data included in the driving operation data supplied from the communication unit 64 in step S41.

In step S47, the virtual space generation unit 71 generates rendering parameters based on the brake position data included in the driving operation data supplied from the communication unit 64 in step S41.

In step S48, the virtual space generation unit 71 generates rendering parameters based on the accelerator position data included in the driving operation data supplied from the communication unit 64 in step S41.

In step S49, the virtual space generation unit 71 reads the virtual space data from the virtual space storage unit 65 and places the objects supplied from the 3DCG generation unit 67 in the virtual space generated by the virtual space generation unit 71. The virtual space generation unit 71 then renders the virtual space based on the rendering parameters generated in steps S46 to S47, and generates a video stream for the vehicle 13 and a video stream for the garage.

In step S50, the virtual space generation unit 71 outputs the video stream for the vehicle 13 generated in step S49 to the signal multiplexing unit 72. The video stream for the vehicle 13 is then transmitted to the vehicle 13 via the signal multiplexing unit 72 and the communication unit 64.

In step S51, the virtual space generation unit 71 outputs the video stream for the garage generated in step S49 to the video conversion processing unit 73. The video conversion processing unit 73 then performs video conversion processing based on the position and orientation data of the vehicle 13 supplied from the vehicle position and orientation recognition unit 63 and the position and orientation data of the occupants in the vehicle 13 supplied from the communication unit 64 in step S41.

In step S52, the video conversion processing unit 73 outputs the video stream for the garage that has been subjected to the video conversion processing in step S51. That is, the video conversion processing unit 73 supplies the video stream for the ceiling surface to the video transmission unit 74 for the ceiling surface, and causes the simulation video for the ceiling surface to be displayed on the garage display 41-1 for the ceiling surface. Similarly, the video conversion processing unit 73 causes the video stream for the front surface to be displayed on the garage display 41-2 for the front surface, the simulation video for the right side surface to be displayed on the garage display 41-3 for the right side surface, the simulation video for the left side surface to be displayed on the garage display 41-4 for the left side surface, the simulation video for the rear surface to be displayed on the garage display 41-5 for the rear surface, and the simulation video for the floor surface to be displayed on the garage display 41-6 for the floor surface. After that, the process returns to step S41, and the same process is repeated.

FIG. 18 is a flowchart explaining the driving simulation operation process on the vehicle 13 side.

For example, when the initial setup process on the vehicle 13 side in FIG. 16 is completed, the driving simulation operation process on the vehicle 13 side is started.

In step S61, the steering wheel rotation data acquisition unit 101 determines whether or not the driver of the vehicle 13 has rotated the steering wheel 21, and waits until it determines that the driver of the vehicle 13 has rotated the steering wheel 21. Then, in step S61, if the steering wheel rotation data acquisition unit 101 determines that the driver of the vehicle 13 has rotated the steering wheel 21, the process proceeds to step S62.

In step S62, the steering wheel rotation data acquisition unit 101 acquires steering wheel rotation data in accordance with the rotation operation of the steering wheel 21 by the driver of the vehicle 13, and supplies the data to the driving operation detection unit 103.

In step S63, the brake position data acquisition unit 100 determines whether or not the driver of the vehicle 13 has depressed the brake pedal 23, and waits until it determines that the driver of the vehicle 13 has depressed the brake pedal 23. Then, in step S63, if the brake position data acquisition unit 100 determines that the driver of the vehicle 13 has depressed the brake pedal 23, the process proceeds to step S64.

In step S64, the brake position data acquisition unit 100 acquires brake position data according to the brake pedal 23 depression operation by the driver of the vehicle 13, and supplies the data to the driving operation detection unit 103.

In step S65, the accelerator position data acquisition unit 102 determines whether or not the driver of the vehicle 13 has depressed the accelerator pedal 22, and waits until it determines that the driver of the vehicle 13 has depressed the accelerator pedal 22. Then, in step S65, if the accelerator position data acquisition unit 102 determines that the driver of the vehicle 13 has depressed the accelerator pedal 22, the process proceeds to step S66.

In step S66, the accelerator position data acquisition unit 102 acquires accelerator position data according to the depression of the accelerator pedal 22 by the driver of the vehicle 13, and supplies the data to the driving operation detection unit 103.

In step S67, the vehicle interior environment recognition unit 95 determines whether or not it has recognized the position and posture of the occupant inside the vehicle 13, and waits until it determines that it has recognized the position and posture of the occupant inside the vehicle 13. Then, in step S67, if the vehicle interior environment recognition unit 95 determines that it has recognized the position and posture of the occupant inside the vehicle 13, the process proceeds to step S68.

In step S68, the vehicle interior environment recognition unit 95 generates position and posture data of the occupants inside the vehicle 13.

In step S69, the driving operation detection unit 103 supplies driving operation data (brake position data, steering wheel rotation data, and accelerator position data) indicating the content of the driving operation of the driver of the vehicle 13 to the communication unit 97. In addition, the vehicle internal environment recognition unit 95 supplies position and posture data of the occupants in the vehicle 13 to the communication unit 97. The communication unit 97 then transmits vehicle parameters including the driving operation data and the position and posture data of the occupants in the vehicle 13 to the garage system control device 16.

In step S70, the communication unit 97 determines whether the vehicle setting parameters have been received, and waits until it is determined that the vehicle setting parameters have been received. Then, when the communication unit 64 of the garage system control device 16 transmits the vehicle setting parameters in step S45 of FIG. 17 described above, the communication unit 97 supplies the vehicle setting parameters to the simulation mode control unit 105, determines that the vehicle setting parameters have been received, and the process proceeds to step S71.

In step S71, the simulation mode control unit 105 generates suspension control data in accordance with the suspension setting data included in the vehicle setting parameters supplied from the communication unit 97, and supplies the data to the suspension control unit 110. This enables the suspension control unit 110 to control the suspension height as described with reference to FIG. 13, and reproduce the behavior of the vehicle 13 in the virtual space.

In step S72, the simulation mode control unit 105 generates seat control data according to the seat setting data included in the vehicle setting parameters supplied from the communication unit 97, and supplies the generated data to the seat control unit 113. This enables the seat control unit 113 to control the seat cushion and the inclination of the seat cushion as described with reference to FIG. 14, and reproduce the behavior of the vehicle 13 in the virtual space.

In step S73, the simulation mode control unit 105 generates steering reaction force control data in accordance with the steering reaction force data value included in the vehicle setting parameters supplied from the communication unit 97, and supplies the generated steering reaction force control data to the steering reaction force control unit 109. This enables the steering reaction force control unit 109 to generate a steering reaction force in the steering wheel 21 that resists the arm force of the driver who is turning the steering wheel 21, thereby reproducing the behavior of the vehicle 13 in the virtual space.

In step S74, the simulation mode control unit 105 generates brake reaction force control data in accordance with the brake reaction force data value included in the vehicle setting parameters supplied from the communication unit 97, and supplies the generated data to the brake reaction force control unit 111. This enables the brake reaction force control unit 111 to generate a brake reaction force in the brake pedal 23 that resists the leg force of the driver who depresses the brake pedal 23, thereby reproducing the behavior of the vehicle 13 in the virtual space.

In step S75, the video signal processing unit 128 determines whether a video stream for the vehicle 13 has been supplied, and waits until it is determined that a video stream for the vehicle 13 has been supplied. Then, when a video stream for the vehicle 13 is transmitted from the garage system control device 16 in step S50 of FIG. 17 described above, the video stream for the vehicle 13 is supplied to the video signal processing unit 128 via the communication unit 122, the signal separation unit 125, and the video input switching unit 126. As a result, the video signal processing unit 128 determines that a video stream for the vehicle 13 has been supplied, and the process proceeds to step S76.

In step S76, the video signal processing unit 128 performs signal processing on the video stream for the vehicle 13, and supplies the corresponding video streams to the navigation video transmission unit 129, the side mirror video transmission unit 130, the instrument panel video transmission unit 131, the rearview mirror video transmission unit 132, and the rear seat video transmission unit 133. As a result, the corresponding images are displayed on the navigation display 51, the side mirror displays 52L and 52R, the instrument panel display 53, the rearview mirror display 54, and the rear seat displays 55L and 55R. Then, the process returns to step S61, and the same process is repeated.

FIG. 19 is a flowchart explaining the vehicle position and attitude recognition process performed in step S13 of FIG. 15.

In step S81, the garage sensor 43 and garage camera 42 are set up.

In step S82, the garage sensor 43 and garage camera 42 start sensing the position and attitude of the vehicle 13, and sensor data and video data are output.

In step S83, the sensor data acquisition unit 61 acquires the sensor data output from the garage sensor 43 and supplies it to the vehicle position and attitude recognition unit 63, and the video data acquisition unit 62 acquires the video data output from the garage camera 42 and supplies it to the vehicle position and attitude recognition unit 63.

In step S84, the vehicle position and attitude recognition unit 63 calculates the position and attitude of the vehicle 13 based on the sensor data and video data supplied in step S83. As a result, the vehicle position and attitude recognition unit 63 acquires the position and attitude data of the vehicle 13, and the vehicle position and attitude recognition process ends.

FIG. 20 is a flowchart explaining the occupant position and attitude recognition process performed in step S27 of FIG. 16.

In step S91, the in-vehicle sensor 57 and the in-vehicle camera 56 are set up.

In step S92, the in-vehicle sensor 57 and the in-vehicle camera 56 start sensing the positions and postures of the occupants in the vehicle 13, and the sensor data and video data are output.

In step S93, the sensor data acquisition unit 91 acquires sensor data output from the in-vehicle sensor 57 and supplies it to the vehicle interior environment recognition unit 95, and the video data acquisition unit 92 acquires video data output from the in-vehicle camera 56 and supplies it to the vehicle interior environment recognition unit 95.

In step S94, the vehicle interior environment recognition unit 95 calculates the positions and postures of the occupants in the vehicle 13 based on the sensor data and video data supplied in step S93. As a result, the vehicle interior environment recognition unit 95 acquires position and posture data of the occupants in the vehicle 13, and the occupant position and posture recognition process is terminated.

Figure 21 is a flowchart explaining the reflection control process that controls the reflection of the body of the vehicle 13.

In step S101, the virtual space generation unit 71 acquires the position and orientation data of the vehicle 13 supplied from the vehicle position and orientation recognition unit 63.

In step S102, the virtual space generation unit 71 places a virtual light source in the virtual space.

In step S103, the virtual space generation unit 71 places a 3DCG object corresponding to the vehicle 13 in the virtual space based on the position and orientation data acquired in step S101.

In step S104, the virtual space generation unit 71 sets the reflectance of the body surface of the 3DCG object corresponding to the vehicle 13 in the virtual space.

In step S105, the virtual space generation unit 71 performs ray tracing such that light is irradiated from the virtual light source placed in step S102 to the 3DCG object corresponding to the vehicle 13 placed in step S103, and the light is reflected on the body surface with the reflectance set in step S104.

In step S106, the virtual space generation unit 71 acquires the position and orientation data of the occupants in the vehicle 13 supplied from the communication unit 64.

In step S107, the virtual space generation unit 71 places a virtual camera at a position in the virtual space that corresponds to the viewpoint position P of the occupant in the vehicle 13.

In step S108, the virtual space generation unit 71 identifies light rays on the body of the vehicle 13 that can be observed by the occupants in the vehicle 13 in the virtual space from the ray tracing results using the virtual camera placed in step S107.

In step S109, the virtual space generation unit 71 identifies the body of the vehicle 13 in the virtual space where the light ray identified in step S108 is reflected, and the coordinates on that body.

In step S110, the vehicle position and attitude recognition unit 63 identifies the body of the vehicle 13 in the virtual space identified in step S109 and the coordinates on that body on the vehicle 13 in the real space based on the sensor data supplied from the sensor data acquisition unit 61 and the video data supplied from the video data acquisition unit 62.

In step S111, the vehicle position and attitude recognition unit 63 determines the position of the garage display 41 in the vertical direction relative to the coordinates determined in step S110.

In step S112, the virtual space generation unit 71 identifies within the virtual space the position and orientation of the garage display 41 in the real space identified in step S111.

In step S113, the virtual space generation unit 71 identifies rendering content for the garage display 41 identified in the virtual space in step S112.

In step S114, the virtual space generation unit 71 displays the rendering content identified in step S113 on the garage display 41 identified in step S112, and the reflection control process is terminated.

The above-described reflection control process makes it possible to reproduce the shine of the surface of the body of the vehicle 13.

<Training System>
A training system using the driving simulation system 11 of FIG. 1 will be described with reference to FIGS.

FIG. 22 is a block diagram showing an example configuration of a training system.

As shown in FIG. 22, the training system 201 is composed of a cloud system 212 constructed on a network 211, and multiple client garage systems 213 connected to the network 211. In the example shown in FIG. 22, N client garage systems 213-1 to 213-N are connected to the network 211. Note that, when there is no need to distinguish between the client garage systems 213-1 to 213-N, they will hereinafter be referred to as client garage systems 213.

The cloud system 212 is composed of a database 214 and a server device 215, and the server device 215 can use various data registered in the database 214 to provide training to multiple client garage systems 213.

As shown in the figure, the server device 215 is configured with a processor 221, an input/output interface 222, an operating system 223, and a memory 224, and application software 225 is stored in the memory 224. The processor 221 then reads out the application software 225 from the memory 224 and executes it to perform training.

Similarly, the client garage system 213 is configured with a processor 231, an input/output interface 232, an operating system 233, and a memory 234, and application software 235 is stored in the memory 234. The processor 231 reads and executes the application software 235 from the memory 234 to perform training. The client garage system 213 corresponds to the driving simulation system 11 in FIG. 1, and includes each of the blocks included in the driving simulation system 11.

For example, the training configurations implemented by the training system 201 are divided into unsupervised training and supervised training.

Figure 23 is a flowchart explaining unsupervised training.

In step S201, the client garage system 213 acquires the user's driving operations (steering wheel rotation data, accelerator position data, and brake position data) for the steering wheel 21, accelerator pedal 22, and brake pedal 23.

In step S202, the client garage system 213 executes a driving simulation (the driving simulation operation process in FIG. 17 and FIG. 18 described above) according to each driving operation of the user acquired in step S201.

In step S203, the client garage system 213 displays the simulation image generated in the driving simulation executed in step S202 on the garage display 41.

In step S204, the client garage system 213 controls the corresponding drive mechanisms according to each control data (steering reaction force control data, suspension control data, brake reaction force control data, accelerator reaction force control data, and seat control data) generated in the driving simulation executed in step S202.

In step S205, the client garage system 213 determines whether or not to end the driving simulation, for example, according to the user's operational input.

If in step S205 the client garage system 213 determines not to end the driving simulation, the process returns to step S201, and the same process is repeated thereafter. In this case, the user can perform each driving operation while being aware of the results of the previous driving simulation.

On the other hand, if the client garage system 213 determines that the driving simulation should be ended, the unsupervised training is terminated.

Figure 24 is a flowchart explaining supervised training.

In step S211, the client garage system 213 acquires the user's driving operations for the steering wheel 21, the accelerator pedal 22, and the brake pedal 23.

In step S212, the client garage system 213 executes a driving simulation according to each of the user's driving operations acquired in step S211. Then, the client garage system 213 generates a difference between the user's operations and the training data.

In step S213, the client garage system 213 displays the simulation image generated in the driving simulation executed in step S212 on the garage display 41. At this time, the client garage system 213 outputs the difference between the user's operation generated in step S212 and the teacher data.

In step S214, the client garage system 213 controls the corresponding drive mechanisms according to the control data generated in the driving simulation executed in step S212. At this time, the client garage system 213 notifies the difference between the user's operation generated in step S212 and the teacher data.

In step S215, the client garage system 213 determines whether or not to end the driving simulation, for example, according to the user's operational input.

If in step S215 the client garage system 213 determines not to end the driving simulation, the process returns to step S211, and the same process is repeated thereafter. In this case, the user can perform each driving operation while being aware of the results of the previous driving simulation.

On the other hand, if the client garage system 213 determines that the driving simulation should be terminated, the supervised training is terminated.

Figure 25 is a flowchart explaining learning.

In step S221, the client garage system 213 selects learning according to the user's operational input.

In step S222, the client garage system 213 starts learning and performs a driving simulation according to the ideal teacher driving pattern.

In step S223, the client garage system 213 displays the simulation video generated in the driving simulation executed in step S222 on the garage display 41, and outputs the audio generated together with the simulation video.

In step S224, the client garage system 213 controls the corresponding drive mechanisms according to the control data generated in the driving simulation executed in step S222, and then the learning process ends.

In this type of learning, students can experience the teacher's example, including AI, without actually driving. For example, students can learn ideal driving patterns, such as steering, braking, and accelerating, while sitting in the driver's seat of their own vehicle 13 and experiencing the reaction forces (operations by the teacher) given to them.

Next, in the training stage, the student performs the operation and receives operational assistance and advice from the teacher while undergoing training. For example, if the student's operation deviates from the ideal driving pattern, the optimal method of returning to the ideal driving pattern depending on the course, road surface, and speed conditions, and the method of training measures to prevent deviation will differ as described below with reference to Figures 26 to 28.

Figure 26 is a flowchart that explains training conducted one-on-one between a teacher and a student, where the teacher is a human.

In step S231, the client garage system 213 selects a training session according to the user's (student's) operational input.

In step S232, the client garage system 213 connects the operation systems and communication paths of the teacher and students via the network 211.

In step S233, the client garage system 213 executes a driving simulation according to the ideal teacher driving pattern.

In step S234, the client garage system 213 displays the simulation video generated in the driving simulation executed in step S233 on the garage display 41, and outputs the audio generated together with the simulation video.

In step S235, the client garage system 213 controls the corresponding drive mechanisms according to the control data generated in the driving simulation executed in step S233.

In step S236, if there is a deviation from the ideal teacher driving pattern, the teacher performs auxiliary operations via the network 211, and the client garage system 213 controls the corresponding drive mechanisms according to each control data based on the auxiliary operations.

In step S237, the client garage system 213 displays the simulation video generated in the driving simulation according to the teacher's assisted operations in step S236 on the garage display 41, and outputs the audio generated together with the simulation video.

In step S238, the client garage system 213 acquires the user's driving operations for the steering wheel 21, accelerator pedal 22, and brake pedal 23.

In step S239, the client garage system 213 determines whether the training is complete. If it determines that the training is not complete, the process returns to step S233, and the same process is repeated. On the other hand, if the client garage system 213 determines in step S239 that the training is complete, the process proceeds to step S240.

In step S240, the client garage system 213 records the evaluation and scoring of the operation log, linking it to the user ID.

In step S241, if there are any concerns, the client garage system 213 adjusts the software in response to the teacher's operation. At this time, the teacher may prepare a request for adjustments to be sent to the dealer if necessary.

In step S242, the client garage system 213 communicates with the teacher, and then the process ends.

In this way, training system 201 allows a human teacher and student to train one-on-one. That is, the operation systems and communication paths of the teacher and student are connected via network 211. Then, while the student is driving, the teacher can remotely provide assistance and advice on steering, braking, accelerator operation, and other driving-related actions via network 211. This allows the user to train individually in the driver's seat of their own vehicle 13.

Figure 27 is a flowchart that explains training in which the teacher is a human and the teacher and students are one-to-many.

In step S251, the client garage system 213 selects a training session according to the user's (student's) operational input.

In step S252, the client garage system 213 connects the operation systems and communication paths of the teacher and student via the network 211, and connects the communication paths between the students.

In step S253, the client garage system 213 executes a driving simulation according to the ideal teacher driving pattern.

In step S254, the client garage system 213 displays the simulation video generated in the driving simulation executed in step S253 on the garage display 41, and outputs the audio generated together with the simulation video.

In step S255, the client garage system 213 controls the corresponding drive mechanisms according to the control data generated in the driving simulation executed in step S253.

In step S256, if there is a deviation from the ideal teacher driving pattern, the teacher performs auxiliary operations as a measure to address the tendency of the entire group via the network 211, and the client garage system 213 controls the corresponding drive mechanisms according to each control data based on the auxiliary operations.

In step S257, the client garage system 213 displays the simulation video generated in the driving simulation following the teacher's assisted operations in step S256 for the entire group on the garage display 41, and outputs the audio generated together with the simulation video.

In step S258, if there is a deviation from the ideal teacher driving pattern, the teacher performs auxiliary operations as a countermeasure and tendency for individuals and groups via the network 211, and the client garage system 213 controls the corresponding drive mechanisms according to each control data based on the auxiliary operations.

In step S259, the client garage system 213 displays, individually and as a group, on the garage display 41 the simulation video generated in the driving simulation following the teacher's assisted operations in step S258, and outputs the audio generated together with the simulation video.

In step S260, the client garage system 213 acquires the user's driving operations for the steering wheel 21, accelerator pedal 22, and brake pedal 23.

In step S261, the client garage system 213 determines whether the training has been completed. If it is determined that the training has not been completed, the process returns to step S253, and the same process is repeated thereafter. On the other hand, if the client garage system 213 determines in step S261 that the training has been completed, the process proceeds to step S262.

In step S262, the client garage system 213 records the evaluation and scoring of each individual operation log, linking it to the user ID.

In step S263, if there are any concerns, the client garage system 213 adjusts each piece of software in response to the teacher's operation. At this time, the teacher may prepare an adjustment request form for the dealer if necessary. For example, concerns here may include the software settings being inappropriate for the user's driving skill, or the software being unable to adjust the settings to suit the user's driving skill, etc.

In step S264, the client garage system 213 communicates with the teacher and among the students, and then the process ends.

In this way, training system 201 can be used to allow one-to-many training between a human teacher and students. That is, the operation systems and communication paths of the teacher and many students are connected via network 211 (communication paths between students may also be connected). Then, while the students are driving, the teacher can remotely provide assistance and advice to the entire group via network 211 for actions related to driving, such as steering, braking, and accelerator operation, based on trends and countermeasures. In addition, the teacher can select individuals or groups that deviate greatly from the ideal driving pattern and provide assistance and advice to them individually, remotely. This allows users to train individually or together with a community, in the driver's seat of their own vehicle 13.

Figure 28 is a flowchart explaining training in which the teacher is an AI and the teacher and students are trained one-to-one or one-to-many.

In step S271, the client garage system 213 selects a training session according to the user's (student's) operational input.

In step S272, the client garage system 213 connects the operation systems and communication paths of the teacher AI and the students via the network 211, and if there are a large number of students, connects the communication paths between the students.

In step S273, the client garage system 213 executes a driving simulation according to the ideal teacher driving pattern.

In step S274, the client garage system 213 displays the simulation video generated in the driving simulation executed in step S273 on the garage display 41, and outputs the audio generated together with the simulation video.

In step S275, the client garage system 213 controls the corresponding drive mechanisms according to the control data generated in the driving simulation executed in step S273.

In step S276, if there is a deviation from the ideal teacher driving pattern, the teacher AI performs auxiliary operations individually via the network 211 as a measure to address the tendency, and the client garage system 213 performs control of the corresponding drive mechanism according to each control data based on the auxiliary operations.

In step S277, the client garage system 213 displays the simulation video generated in the driving simulation according to the individual auxiliary operations of the teacher AI in step S276 on the garage display 41, and outputs the audio generated together with the simulation video.

In step S278, the client garage system 213 generates a driving simulation and controls the corresponding drive mechanisms according to each control data with the support of the teacher AI.

In step S279, the client garage system 213 displays, for each individual and group, on the garage display 41 the simulation video generated by the driving simulation following the assisted operations of the teacher AI in step S278, and outputs the audio generated together with the simulation video.

In step S280, the client garage system 213 acquires the user's driving operations for the steering wheel 21, accelerator pedal 22, and brake pedal 23.

In step S281, the client garage system 213 determines whether the training has been completed. If it is determined that the training has not been completed, the process returns to step S273, and the same process is repeated thereafter. On the other hand, if the client garage system 213 determines in step S281 that the training has been completed, the process proceeds to step S282.

In step S282, the client garage system 213 records the evaluation and scoring of each individual operation log, linking it to the user ID.

In step S283, if there are any concerns, the client garage system 213 adjusts each piece of software according to the output of the teacher AI. At this time, the teacher AI may prepare an adjustment request form for the dealer if necessary.

In step S284, the client garage system 213 communicates between the students, and then the process ends.

In this way, the teacher AI and students can train one-to-one and one-to-many. That is, the operation systems and communication paths of the teacher AI and many students are connected via the network 211 (communication paths between students may also be connected). The teacher AI can then remotely provide auxiliary operations and advice in response to deviations from the ideal driving pattern. This allows the user to train individually or together with a community in the driver's seat of their own vehicle 13.

As described above, the training system 201 can improve the efficiency of management and training itself by using a combination of a human teacher and a teacher AI, such as entrusting repetitive training to the teacher AI depending on the training situation. Furthermore, the teacher AI may be executed on either the cloud system 212 or the client garage system 213.

For example, the ideal driving pattern differs depending on the purpose, such as safe driving or racing, but in the training system 201, it is derived by setting goals to be achieved through training, tests, game elements, etc., and by the accompanying professional driving, teacher examples, and various AI technologies. In the training system 201, the achievement of the set goals and all trained actions are recorded as metadata in the metaverse space, making fair evaluation and scoring possible. Scoring can also include actions related to driving operations, such as detection of actions such as checking for collisions through camera monitoring, and detection of shift, turn signal, hazard lamp, fog lamp, and defroster operation.

In this way, by using the training system 201, the user can understand the functions and performance of the vehicle 13 through learning and training based on the user's own actual experience with the vehicle 13, and can improve their driving skills by improving their vehicle sense and operation skills. For example, the training system 201 can not only reduce accidents by improving the user's driving skills, but also contribute to the granting and renewal of licenses by enabling fair evaluation and scoring.

Next, referring to FIG. 29, we will explain personal optimization of the vehicle 13 using the training system 201.

For example, as the trend towards electric vehicles for vehicles 13 leads to the consolidation of chassis platforms from each OEM (Original Equipment Manufacturing (Manufacturer)) into one or two, it is expected that some of the settings for the ride comfort and handling of vehicles 13 in the future will be controlled by software. Here, we will explain a method for personal optimization of vehicles 13 in light of this trend.

For example, through training, the driver will improve his/her driving skills to match the performance of his/her vehicle 13. At this time, it is expected that the response, weight, and stiffness of the steering, brakes, accelerator, suspension, etc., may be hindered by differences in each individual's abilities, or may be impossible to do in the first place. In such cases, it is possible to compare the training response with the data, communicate with the teacher, and adjust parts that can be partially controlled by software using software.

On the other hand, any parts that cannot be adjusted using software can be passed on to the dealer as adjustment items and can be adjusted.

In step S291, the client garage system 213 reads parameters linked to the personal ID of the user in the vehicle 13 that supports the personal optimization function, and acquires each driving operation of the user.

In step S292, the client garage system 213 performs a matching process on the cloud with the performance of the vehicle 13 in which the user will be riding, and then downloads the parameters to a vehicle 13 that supports the personal optimization function.

In step S293, the client garage system 213 performs parameter adjustment and the process ends.

These personalized parameters can be used in the following ways:

For example, if a driver can improve their driving skills during training and achieve optimization through compromise between the driver and the vehicle by adjusting the vehicle, the driver's own vehicle 13 will become unique to the driver. On the other hand, there is an issue that it will become difficult for the driver to ride in other vehicles 13.

Then, the vehicle adjustment parameters linked to one's ID are stored on the cloud, and when one drives one's partner's vehicle 13 or a shared car or rental vehicle 13, if the vehicle 13 one is driving supports personal optimization functions, the parameters are read from the cloud and a matching process is performed with the performance of the vehicle 13 one is driving, allowing the settings to be automatically adjusted to match those of one's own vehicle 13.

In addition, when replacing the vehicle 13, it is possible to start training from scratch when purchasing the vehicle 13, but if the replacement vehicle 13 supports the personal optimization function, it is possible to read the vehicle adjustment parameters linked to one's ID from the cloud and perform a matching process with the performance of the replacement vehicle 13 to bring it closer to the settings of one's own vehicle 13. For example, if it is known in advance that the replacement vehicle 13 supports the personal optimization function, it is possible to virtually reproduce the personally optimized settings after replacement in the current vehicle 13, and to carry out learning and training related to the replacement vehicle 13 in advance.

Furthermore, by being able to virtually experience driving the replacement vehicle 13 and learning and training about it in a state that has been individually optimized in advance, it is possible to provide an optimal experience to a larger number of people and reduce some of the physical test drives, which could lead to effective promotion of the car while also being environmentally friendly and reducing accidents.

As described above, a training system has been described as an example of a system that uses the driving simulation system 11 of FIG. 1. The driving simulation system 11 may be used as a system that provides a simulated driving experience. The driving simulation system 11 may also be used as a system that provides an experience related to new safety features of the vehicle 13, or safety features not yet installed in the vehicle 13 owned by the user.

<Example of computer configuration>
Next, the above-mentioned series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

FIG. 30 is a block diagram showing an example of the configuration of one embodiment of a computer on which a program that executes the series of processes described above is installed.

The program can be pre-recorded on the hard disk 305 or ROM 303 as a recording medium built into the computer.

Alternatively, the program can be stored (recorded) on a removable recording medium 311 driven by the drive 309. Such a removable recording medium 311 can be provided as so-called packaged software. Here, examples of the removable recording medium 311 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, a semiconductor memory, etc.

In addition to being installed on a computer from the removable recording medium 311 as described above, the program can also be downloaded to the computer via a communication network or broadcasting network and installed on the built-in hard disk 305. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer via a wired connection via a network such as a LAN (Local Area Network) or the Internet.

The computer has a built-in CPU (Central Processing Unit) 302, to which an input/output interface 310 is connected via a bus 301.

When a command is input by the user via the input/output interface 310 by operating the input unit 307, the CPU 302 executes a program stored in the ROM (Read Only Memory) 303 accordingly. Alternatively, the CPU 302 loads a program stored on the hard disk 305 into the RAM (Random Access Memory) 304 and executes it.

As a result, the CPU 302 performs processing according to the above-mentioned flowchart, or processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 302 outputs the processing results from the output unit 306 via the input/output interface 310, or transmits them from the communication unit 308, or even records them on the hard disk 305, as necessary.

The input unit 307 is composed of a keyboard, mouse, microphone, etc. The output unit 306 is composed of an LCD (Liquid Crystal Display), speaker, etc.

In this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing that is executed in parallel or individually (for example, parallel processing or processing by objects).

The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configuration other than that described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

Also, for example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

Furthermore, for example, the above-mentioned program can be executed in any device. In that case, it is sufficient that the device has the necessary functions (functional blocks, etc.) and is capable of obtaining the necessary information.

Furthermore, for example, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

In addition, the processing of the steps that describe a program executed by a computer may be executed chronologically in the order described in this specification, or may be executed in parallel, or individually at the required timing, such as when a call is made. In other words, as long as no contradictions arise, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps that describe this program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

Note that the multiple present technologies described in this specification can be implemented independently and individually, provided no contradictions arise. Of course, any multiple present technologies can also be implemented in combination. For example, some or all of the present technologies described in any embodiment can be implemented in combination with some or all of the present technologies described in other embodiments. Also, some or all of any of the present technologies described above can be implemented in combination with other technologies not described above.

<Examples of configuration combinations>
The present technology can also be configured as follows.
(1)
A communication unit that communicates with other information processing devices;
an operation data acquisition unit that acquires driving operation data indicating amounts of driving operations for a plurality of types of operation systems for operating a vehicle;
a vehicle operation control unit that supplies control signals to a plurality of types of drive systems that drive the vehicle to control the operation of the vehicle;
a driving mode control unit that supplies control data to the vehicle operation control unit to control the driving of the vehicle in accordance with the driving operation data when the vehicle is in a driving mode in which the vehicle is driven;
a simulation mode control unit that, in a simulation mode in which a driving simulation is performed using the vehicle, supplies to the vehicle operation control unit control data for controlling a behavior of the vehicle so as to reproduce a behavior of the vehicle in a virtual space determined according to the driving operation data, based on vehicle setting parameters acquired from the other information processing device via the communication unit;
and a stopped state determination unit that determines whether to transition to the simulation mode according to a stopped state of the vehicle when an operation is performed to instruct switching from the driving mode to the simulation mode.
(2)
At least one of the plurality of types of operation systems and at least one of the plurality of types of drive systems corresponding to the operation systems are connected by a by-wire system,
The information processing device described in (1) above is configured so that, in the simulation mode, even if a driving operation is performed on one of the operating systems, the drive system connected to that one of the operating systems by a by-wire system will not operate.
(3)
The information processing device described in (1) or (2) above, wherein the stopped state determination unit determines that the vehicle will transition to the simulation mode when an operation is performed to instruct switching from the driving mode to the simulation mode while the vehicle is stopped.
(4)
a vehicle interior environment recognition unit that recognizes the positions and postures of occupants of the vehicle based on sensor data and image data supplied from sensors and cameras provided in the vehicle, and acquires occupant position and posture data including at least the viewpoint positions of the occupants,
The information processing device according to any one of (1) to (3) above, wherein the communication unit transmits the occupant position and attitude data to the other information processing device.
(5)
The vehicle can perform a driving simulation while being stored in a garage device having a garage display that displays an image surrounding the vehicle,
The garage display performs a driving simulation in which the vehicle is virtually driven within the virtual space in accordance with the driving operation data, and displays a simulation image generated by photographing the virtual space in all directions centered on the vehicle with a virtual camera positioned within the virtual space to correspond to a predetermined position of the vehicle, the simulation image being geometrically transformed according to the occupant's viewpoint based on vehicle position and attitude data indicating the position and attitude of the vehicle and the occupant position and attitude data.The information processing device described in (4) above.
(6)
The vehicle is equipped with a side mirror display that displays a side mirror image captured toward the rear of the vehicle by a side mirror camera installed on a side of the vehicle,
The information processing device described in (5) above, in which, in the simulation mode, a virtual side mirror image generated by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on the side of the vehicle in the virtual space is displayed on the side mirror display.
(7)
The vehicle is equipped with a rearview mirror display that displays a rearview mirror image captured toward the rear of the vehicle by a rearview mirror camera installed on a rear surface of the vehicle,
The information processing device described in (5) or (6) above, in which, in the simulation mode, a virtual rearview mirror image generated by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on the rear of the vehicle in the virtual space is displayed on the rearview mirror display.
(8)
The vehicle includes a rear seat display provided in a rear seat of the vehicle,
In the simulation mode, a rear seat viewpoint image generated by capturing an image of the virtual space as seen through the rear seat display using a virtual camera positioned in the virtual space to correspond to the viewpoint of the rear seat occupant in accordance with the occupant position and attitude data is displayed on the rear seat display. An information processing device as described in any of (5) to (7) above.
(9)
An information processing device,
Communicating with other information processing devices;
Acquiring driving operation data indicating amounts of driving operations for a plurality of types of operation systems for operating a vehicle;
supplying control signals to a plurality of types of drive systems that drive the vehicle to control operation of the vehicle;
When the vehicle is in a driving mode, the vehicle is controlled by control data for controlling the driving of the vehicle in accordance with the driving operation data.
In a simulation mode in which a driving simulation is performed using the vehicle, control of the operation of the vehicle is performed using control data that controls the behavior of the vehicle so as to reproduce the behavior of the vehicle in the virtual space determined according to the driving operation data, based on vehicle setting parameters acquired by communication from the other information processing device.
and when an operation is performed to instruct switching from the driving mode to the simulation mode, determining whether to transition to the simulation mode according to a stopped state of the vehicle.
(10)
The computer of the information processing device
Communicating with other information processing devices;
Acquiring driving operation data indicating amounts of driving operations for a plurality of types of operation systems for operating a vehicle;
supplying control signals to a plurality of types of drive systems that drive the vehicle to control operation of the vehicle;
When the vehicle is in a driving mode, the vehicle is controlled by control data for controlling the driving of the vehicle in accordance with the driving operation data.
In a simulation mode in which a driving simulation is performed using the vehicle, control of the operation of the vehicle is performed using control data that controls the behavior of the vehicle so as to reproduce the behavior of the vehicle in the virtual space determined according to the driving operation data, based on vehicle setting parameters acquired by communication from the other information processing device.
and when an operation is performed to instruct switching from the driving mode to the simulation mode, determining whether to transition to the simulation mode according to a stopped state of the vehicle.
(11)
A communication unit that communicates with other information processing devices mounted in the vehicle;
a virtual space generation unit that performs a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle via the communication unit, and that generates a simulation image by capturing images of the virtual space in all directions centered on the vehicle using a virtual camera that is arranged in the virtual space so as to correspond to a predetermined position of the vehicle;
an image conversion processing unit that performs image conversion processing on the simulation image based on vehicle position and attitude data indicating a position and attitude of the vehicle and occupant position and attitude data including at least a viewpoint position of an occupant of the vehicle.
(12)
The information processing device described in (11) above, wherein the virtual space generation unit determines the behavior of the vehicle within the virtual space, generates vehicle setting parameters for controlling the behavior of the vehicle so as to reproduce that behavior, and transmits the vehicle setting parameters to the vehicle via the communication unit.
(13)
The vehicle can perform a driving simulation while being stored in a garage device having a garage display that displays an image surrounding the vehicle,
The information processing device according to (11) or (12) above, wherein the simulation image output from the image conversion processing unit is displayed on the garage display.
(14)
The video conversion processing unit includes:
a viewpoint detection unit that identifies a viewpoint position of the occupant relative to the garage display by setting a rectangular parallelepiped space in which the vehicle is inscribed at a position of the vehicle relative to the garage display based on the vehicle position and orientation data, and setting a viewpoint position of the occupant relative to the rectangular parallelepiped space based on the occupant position and orientation data;
The information processing device according to claim 13, further comprising: a geometric transformation unit that performs a geometric transformation such that the simulation image generated in the virtual space generation unit is projected onto the garage display as a projection surface, based on the viewpoint position of the occupant identified by the viewpoint detection unit.
(15)
The information processing device described in (14) above, wherein the virtual space generation unit generates a virtual side mirror image by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on the side of the vehicle within the virtual space, transmits the virtual side mirror image to the vehicle via the communication unit, and displays the image on a side mirror display of the vehicle.
(16)
The information processing device described in (14) or (15) above, wherein the virtual space generation unit generates a virtual rearview mirror image by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on the back of the vehicle within the virtual space, transmits the image to the vehicle via the communication unit, and displays the image on a rearview mirror display of the vehicle.
(17)
The virtual space generation unit generates a rear seat viewpoint image by capturing the virtual space as seen through a rear seat display provided in the rear seat of the vehicle using a virtual camera positioned in the virtual space to correspond to the viewpoint of an occupant in the rear seat of the vehicle in accordance with the occupant position and attitude data, transmits the image to the vehicle via the communication unit, and displays it on the rear seat display. An information processing device described in any of (14) to (16) above.
(18)
An information processing device,
Communicating with other information processing devices mounted in the vehicle;
performing a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle through the communication; and capturing images of the virtual space in all directions from the vehicle center with a virtual camera disposed in the virtual space so as to correspond to a predetermined position of the vehicle, thereby generating a simulation image;
and performing an image conversion process on the simulation image based on vehicle position and attitude data indicating a position and attitude of the vehicle and occupant position and attitude data including at least a viewpoint position of an occupant of the vehicle.
(19)
The computer of the information processing device
Communicating with other information processing devices mounted in the vehicle;
performing a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle through the communication; and capturing images of the virtual space in all directions from the vehicle center with a virtual camera disposed in the virtual space so as to correspond to a predetermined position of the vehicle, thereby generating a simulation image;
and performing image conversion processing on the simulation image based on vehicle position and attitude data indicating the position and attitude of the vehicle and occupant position and attitude data including at least the viewpoint positions of occupants of the vehicle.

Note that this embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of this disclosure. Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

11 driving simulation system, 12 garage system, 13 vehicle, 14 garage device, 15 charging device, 16 garage system control device, 17 home server, 18 in-vehicle control device, 19 power source, 21 steering wheel, 22 accelerator pedal, 23 brake pedal, 24 axle steering section, 25 throttle motor, 26 brake, 27 steering wheel rotation sensor, 28 accelerator position sensor, 29 brake position sensor, 30 axle drive mechanism, 31 throttle drive mechanism, 32 Brake drive mechanism, 33 Steering wheel reaction drive mechanism, 34 Accelerator reaction drive mechanism, 35 Brake reaction drive mechanism, 36 Suspension drive mechanism, 37 Seat drive mechanism, 41 Garage display, 42 Garage camera, 43 Garage sensor, 51 Navigation display, 52 Side mirror display, 53 Instrument panel display, 54 Rearview mirror display, 55 Rear seat display, 56 In-car camera, 57 In-car sensor, 58 Speaker

Claims

A communication unit that communicates with other information processing devices;
an operation data acquisition unit that acquires driving operation data indicating amounts of driving operations for a plurality of types of operation systems for operating a vehicle;
a vehicle operation control unit that supplies control signals to a plurality of types of drive systems that drive the vehicle to control the operation of the vehicle;
a driving mode control unit that supplies control data for controlling the driving of the vehicle to the vehicle operation control unit in accordance with the driving operation data when the vehicle is in a driving mode in which the vehicle is driven;
a simulation mode control unit that, in a simulation mode in which a driving simulation is performed using the vehicle, supplies to the vehicle operation control unit control data for controlling a behavior of the vehicle so as to reproduce a behavior of the vehicle in a virtual space determined according to the driving operation data, based on vehicle setting parameters acquired from the other information processing device via the communication unit;
and a stopped state determination unit that determines whether to transition to the simulation mode according to a stopped state of the vehicle when an operation is performed to instruct switching from the driving mode to the simulation mode.
At least one of the plurality of types of operation systems and at least one of the plurality of types of drive systems corresponding to the operation systems are connected by a by-wire system,
2. The information processing device according to claim 1, wherein, in the simulation mode, even if a driving operation is performed on one of the operation systems, the drive system connected to the one of the operation systems by a by-wire system will not operate.
The information processing device according to claim 1 , wherein the vehicle stop state determination unit determines that the mode will be transitioned to the simulation mode when an operation for instructing switching from the driving mode to the simulation mode is performed while the vehicle is stopped.
a vehicle interior environment recognition unit that recognizes the positions and postures of occupants of the vehicle based on sensor data and image data supplied from sensors and cameras provided in the vehicle, and acquires occupant position and posture data including at least the viewpoint positions of the occupants,
The information processing device according to claim 1 , wherein the communication unit transmits the occupant position and attitude data to the other information processing device.
The vehicle can perform a driving simulation while being stored in a garage device having a garage display that displays an image surrounding the vehicle,
5. The information processing device according to claim 4, wherein a driving simulation is performed on the garage display in which the vehicle is virtually driven within the virtual space in accordance with the driving operation data, and a simulation image is displayed which is generated by photographing the virtual space in all directions centered on the vehicle with a virtual camera arranged within the virtual space so as to correspond to a predetermined position of the vehicle, and which has been geometrically transformed according to the viewpoint of the occupant based on vehicle position and attitude data indicating the position and attitude of the vehicle and the occupant position and attitude data.
The vehicle is equipped with a side mirror display that displays a side mirror image captured toward the rear of the vehicle by a side mirror camera installed on a side of the vehicle,
6. The information processing device according to claim 5, wherein in the simulation mode, a virtual side mirror image generated by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on a side of the vehicle in the virtual space is displayed on the side mirror display.
The vehicle is equipped with a rearview mirror display that displays a rearview mirror image captured toward the rear of the vehicle by a rearview mirror camera installed on a rear surface of the vehicle,
6. The information processing device according to claim 5, wherein in the simulation mode, a virtual rearview mirror image generated by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on the rear of the vehicle in the virtual space is displayed on the rearview mirror display.
The vehicle includes a rear seat display provided in a rear seat of the vehicle,
6. The information processing device according to claim 5, wherein in the simulation mode, a rear seat viewpoint image generated by capturing an image of the virtual space as seen through the rear seat display with a virtual camera arranged in the virtual space so as to correspond to the viewpoint of the rear seat occupant in accordance with the occupant position and attitude data is displayed on the rear seat display.
An information processing device,
Communicating with other information processing devices;
Acquiring driving operation data indicating amounts of driving operations for a plurality of types of operation systems for operating a vehicle;
supplying control signals to a plurality of types of drive systems that drive the vehicle to control operation of the vehicle;
When the vehicle is in a driving mode, the vehicle is controlled by control data for controlling the driving of the vehicle in accordance with the driving operation data.
In a simulation mode in which a driving simulation is performed using the vehicle, control of the operation of the vehicle is performed using control data that controls the behavior of the vehicle so as to reproduce the behavior of the vehicle in the virtual space determined according to the driving operation data, based on vehicle setting parameters acquired by communication from the other information processing device.
and when an operation is performed to instruct switching from the driving mode to the simulation mode, determining whether to transition to the simulation mode according to a stopped state of the vehicle.
The computer of the information processing device
Communicating with other information processing devices;
Acquiring driving operation data indicating amounts of driving operations for a plurality of types of operation systems for operating a vehicle;
supplying control signals to a plurality of types of drive systems that drive the vehicle to control operation of the vehicle;
When the vehicle is in a driving mode, the vehicle is controlled by control data for controlling the driving of the vehicle in accordance with the driving operation data.
In a simulation mode in which a driving simulation is performed using the vehicle, control of the operation of the vehicle is performed using control data that controls the behavior of the vehicle so as to reproduce the behavior of the vehicle in the virtual space determined according to the driving operation data, based on vehicle setting parameters acquired by communication from the other information processing device.
and when an operation is performed to instruct switching from the driving mode to the simulation mode, determining whether to transition to the simulation mode according to a stopped state of the vehicle.
A communication unit that communicates with other information processing devices mounted in the vehicle;
a virtual space generation unit that performs a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle via the communication unit, and that generates a simulation image by capturing images of the virtual space in all directions centered on the vehicle using a virtual camera that is arranged in the virtual space so as to correspond to a predetermined position of the vehicle;
an image conversion processing unit that performs image conversion processing on the simulation image based on vehicle position and attitude data indicating a position and attitude of the vehicle and occupant position and attitude data including at least a viewpoint position of an occupant of the vehicle.
The information processing device according to claim 11, wherein the virtual space generation unit determines a behavior of the vehicle in the virtual space, generates vehicle setting parameters for controlling the behavior of the vehicle so as to reproduce the behavior, and causes the vehicle to transmit the vehicle setting parameters via the communication unit.
The vehicle can perform a driving simulation while being stored in a garage device having a garage display that displays an image surrounding the vehicle,
The information processing device according to claim 11 , wherein the simulation image output from the image conversion processing unit is displayed on the garage display.
The video conversion processing unit includes:
a viewpoint detection unit that identifies a viewpoint position of the occupant relative to the garage display by setting a rectangular parallelepiped space in which the vehicle is inscribed at a position of the vehicle relative to the garage display based on the vehicle position and orientation data, and setting a viewpoint position of the occupant relative to the rectangular parallelepiped space based on the occupant position and orientation data;
14. The information processing device according to claim 13, further comprising: a geometric transformation unit that performs a geometric transformation such that the simulation image generated by the virtual space generation unit is projected onto the garage display as a projection surface, based on the viewpoint position of the occupant identified by the viewpoint detection unit.
15. The information processing device according to claim 14, wherein the virtual space generation unit generates a virtual side mirror image by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on a side of the vehicle within the virtual space, transmits the virtual side mirror image to the vehicle via the communication unit, and displays the virtual side mirror image on a side mirror display of the vehicle.
15. The information processing device according to claim 14, wherein the virtual space generation unit generates a virtual rearview mirror image by photographing the virtual space toward the rear of the vehicle with a virtual camera arranged on the rear of the vehicle in the virtual space, transmits the virtual rearview mirror image to the vehicle via the communication unit, and displays the virtual rearview mirror image on a rearview mirror display of the vehicle.
The information processing device according to claim 14, wherein the virtual space generation unit generates a rear seat viewpoint image by capturing an image of the virtual space as seen through a rear seat display provided in the rear seat of the vehicle using a virtual camera arranged in the virtual space to correspond to the viewpoint of an occupant in the rear seat of the vehicle in accordance with the occupant position and attitude data, transmits the image to the vehicle via the communication unit, and displays the image on the rear seat display.
An information processing device,
Communicating with other information processing devices mounted in the vehicle;
performing a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle through the communication; and capturing images of the virtual space in all directions from the vehicle center with a virtual camera disposed in the virtual space so as to correspond to a predetermined position of the vehicle, thereby generating a simulation image;
and performing image conversion processing on the simulation image based on vehicle position and attitude data indicating the position and attitude of the vehicle and occupant position and attitude data including at least the viewpoint positions of occupants of the vehicle.
The computer of the information processing device
Communicating with other information processing devices mounted in the vehicle;
performing a driving simulation in which the vehicle virtually travels in a virtual space in accordance with driving operation data acquired from the vehicle through the communication, and capturing images of the virtual space in all directions from the vehicle center with a virtual camera disposed in the virtual space so as to correspond to a predetermined position of the vehicle, thereby generating a simulation image;
and performing image conversion processing on the simulation image based on vehicle position and attitude data indicating the position and attitude of the vehicle and occupant position and attitude data including at least the viewpoint positions of occupants of the vehicle.