JP2023104401A - Information processing device, on-vehicle virtual display device, virtual display system, and virtual experience providing method - Google Patents

Abstract

To reduce user's dissatisfaction caused by a vehicle form used.SOLUTION: An information processing device according to an embodiment has a control unit which causes a display part provided in a passenger compartment of a vehicle to display images. The control unit causes the display part to display a virtual interior image of a passenger compartment space which is larger than the passenger compartment.SELECTED DRAWING: Figure 5

Description

The present invention relates to an information processing device, an in-vehicle virtual display device, a virtual display system, and a virtual experience providing method.

Various devices have been developed to entertain a user while in a vehicle. For example, conventionally, by displaying a background image according to the situation of the vehicle, for example, a tourist spot image of the tourist spot in the vicinity of the tourist spot, etc. There is known a technique for allowing passengers to enjoy their time in a car (for example, see Patent Document 1).

On the other hand, in recent years, there are signs that ultra-compact mobility will spread due to its advantages such as being able to turn in a small radius and being easy to drive. Ultra-compact mobility is a vehicle that seats about one or two people, and it is conceivable that it will be used, for example, for activities during travel, like conventional rental bicycles.

By the way, when traveling in a group with multiple people, when using ultra-compact mobility for activities at the destination, each member of the group (or by a small number of people) must use the ultra-compact mobility individually (or by a small number of people) because the compartment of the ultra-compact mobility is small. I have to ride.

Then, despite the group trip, each member cannot communicate with other members while riding in the ultra-compact mobility, so there is a risk that the enjoyment of group activities will be halved.

JP 2019-121959 A

As described above, due to constraints such as the shape of the vehicle, such as the size and the number of passengers, the user may not be able to fully enjoy the group trip, or the user may feel unsatisfied because the vehicle is narrow and oppressive. there were.

One aspect of the embodiments has been made in view of the above, and is an information processing device, an in-vehicle virtual display device, a virtual display system, and a virtual display system capable of reducing user dissatisfaction caused by the type of vehicle used. The purpose is to provide an experience provision method.

An information processing apparatus according to an aspect of an embodiment is an information processing apparatus having a control unit that displays an image on a display unit provided inside a vehicle. The control unit causes the display unit to display a virtual interior image of a vehicle interior space that is larger than the vehicle interior.

According to one aspect of the embodiment, user dissatisfaction caused by the type of vehicle used can be reduced.

FIG. 1 is a schematic explanatory diagram of a vehicle according to an embodiment. FIG. 2 is a schematic explanatory diagram of the virtual experience providing method according to the embodiment. FIG. 3 is a diagram showing a configuration example of the virtual experience providing system according to the embodiment. FIG. 4 is a schematic plan view showing a configuration example of the vehicle according to the embodiment. FIG. 5 is a block diagram showing a configuration example of an in-vehicle device according to the embodiment. FIG. 6 is a diagram showing an example of a layout information table. FIG. 7 is a diagram showing an example of an interior DB. FIG. 8 is a diagram showing an example of an avatar DB. FIG. 9 is an explanatory diagram (part 1) of audio output and temperature control. FIG. 10 is an explanatory diagram (part 2) of audio output and temperature control. FIG. 11 is an explanatory diagram of a case where a virtual co-riding image is displayed on a display that functions as a digital rearview mirror. FIG. 12 is a flowchart illustrating a processing procedure executed by the in-vehicle device according to the embodiment; FIG. 13 is a diagram showing a configuration example of a ride-sharing experience providing system according to a modification.

Hereinafter, embodiments of an information processing device, an in-vehicle virtual display device, a virtual display system, and a virtual experience providing method disclosed in the present application will be described in detail with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

Further, hereinafter, the information processing device or the in-vehicle virtual display device according to the embodiment is assumed to be the control device 10 mounted in the vehicle V. FIG. In addition, hereinafter, the virtual display system according to the embodiment is a passenger experience providing system 1 that allows the user to simulate a situation in which a virtual fellow passenger (hereinafter referred to as "virtual fellow passenger" as appropriate) exists in the vehicle compartment. shall be

Further, hereinafter, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numbers with hyphens after the same reference numerals. For example, a plurality of configurations having substantially the same functional configurations are distinguished as control device 10-1, control device 10-2, . . . as necessary. However, when there is no particular need to distinguish between a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are used. For example, the control device 10-1, the control device 10-2, .

Also, in the following description, the vehicle V according to the embodiment is assumed to be a one-seater ultra-compact mobility vehicle.

First, an overview of the virtual experience providing method according to the embodiment will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 is a schematic explanatory diagram of a vehicle V according to an embodiment. Moreover, FIG. 2 is a schematic explanatory diagram of the virtual experience providing method according to the embodiment.

As shown in FIG. 1, the vehicle V according to the embodiment is a single-passenger vehicle known as a micromobility. Since it is a one-seat vehicle, there is only a driver's seat, and there is no passenger seat or rear seat.

Here, it is assumed that one person is on board, but the number of occupants of the vehicle V when applying this embodiment is not limited. Further, the vehicle V may be an electric vehicle using an electric motor as power, or may be a vehicle using other power such as an internal combustion engine.

Moreover, as shown in FIG. 1 , the vehicle V includes a display device 5 and a control device 10 . The display device 5 is, for example, a window display incorporated in the window of the vehicle V, as shown in FIG. The display device 5 is a non-transmissive display using a liquid crystal display device or the like. The display device 5 can also be realized by a transmissive display, a projection device that projects an image onto a window provided with a screen function by white processing, semi-reflection processing, or the like.

In this embodiment, the display device 5 is a non-transmissive display, and is provided so as to be able to display the scenery outside the vehicle captured by an in-vehicle camera (not shown) in the vehicle interior as a vehicle window scenery image while the vehicle V is running. shall be

The control device 10 performs display control of the display device 5 and causes the display device 5 to display various images.

In addition, the control device 10 performs manual driving control of the vehicle V based on the user's driving operation, and automatic driving control of the vehicle V based on instructions from an external device, external conditions obtained from an on-vehicle sensor (not shown), and the like. conduct.

When such a vehicle V is used, for example, for a group trip with a plurality of people, each member of the group must ride in an individual vehicle V because the vehicle V is a single-seater. .

Therefore, although each member is on a group trip, while each member is on board the vehicle V, it is not possible to sufficiently communicate with each other just by using simple communication such as voice communication using radio or the like. Can not. In addition, each member feels a strong sense of isolation because they are alone in the vehicle V. For this reason, there is a risk that the enjoyment of group activities will be halved for each member.

Even if the vehicle V has a small number of passengers (capacity smaller than the number of people in a group), such as a two-seater, the same problem exists, although the degree is different. In addition, even for a user who travels alone, driving a small vehicle such as the vehicle V gives a strong sense of oppression due to the narrow interior space, and there is also a problem that the user is likely to get mentally tired.

Therefore, in the virtual experience providing method according to the embodiment, an image that virtually provides a vehicle interior space suitable for the user is displayed on the display device 5 provided in the vehicle interior.

A specific description will be given with reference to FIG. In the following description, an example will be described from the viewpoint of the member M-1 who is in the driver's seat of the vehicle V-1.

The virtual experience providing method according to the embodiment superimposes an actual train window scenery image, an interior image of the interior of the vehicle that does not actually exist, and an image of the member M at another point on the display device 5 in the vehicle interior. Display a virtual co-located image. As a result, the virtual experience providing method according to the embodiment allows the user to simulate a situation in which a virtual fellow passenger exists in the vehicle interior, that is, provides a virtual co-riding experience.

Specifically, as shown in FIG. 2, the method for providing a virtual experience according to the embodiment displays an image of the actual vehicle window scenery seen from the driver's seat by the member M-1 on the display device 5 of the vehicle V-1. , for example, an interior image of a vehicle with a front passenger seat and a real-time image of member M-2 at another location are superimposed and displayed. That is, the interior image is an interior image of another vehicle having a larger cabin space than the vehicle V. FIG.

As a result, although the member M-1 is in the vehicle V-1 with one passenger, there is a passenger seat in the vehicle interior, and the member M-2 is sitting in the passenger seat, and the vehicle V-1 You can experience a simulated experience as if you were on board.

That is, it is possible to provide the member M-1 with a ride experience that is not restricted by the shape of the actual vehicle V-1. In addition, since it is possible to make the member M-1 feel that the passenger compartment is wider than it actually is, it is possible to improve the comfort during riding.

Further, the virtual experience providing method according to the embodiment outputs not only the image but also the voice uttered by the member M-2 in real time into the vehicle V-1. At this time, since the image and voice of the member M-2 are synchronized, it is possible to give the impression to the member M-1 that the virtual fellow passenger is speaking.

In addition, the method for providing a virtual experience according to the embodiment cooperates with an in-vehicle air conditioner to control the temperature in the vehicle interior and the air flow in the interior, thereby providing the member M-1 with the virtual passenger's body temperature. It can make you feel the change and provide a more advanced virtual experience. Specific examples of the output of these sounds and temperature control will be described later with reference to FIGS. 9 and 10. FIG.

Further, in the virtual experience providing method according to the embodiment, it is possible to set an arbitrary interior form for the interior image. For example, an interior image equivalent to the interior of the vehicle to be displayed in the virtual image (enlarges the current vehicle interior without discomfort), an interior image equivalent to the interior of the vehicle V-2 in which the member M-2 to be displayed is riding (member It is possible to set the interior image of a luxury car etc. . Also, as for the image of the member M at the other location, it is possible to set a real image or an arbitrary avatar (selected from the viewpoint of personal preference and personal information protection). A specific example of this point will be described later with reference to FIGS. 6 to 8. FIG.

Moreover, in the virtual experience providing method according to the embodiment, not only the member M who is the accompanying person but also the non-accompanying member M can virtually ride together. A specific example of this point will be described later with reference to FIG.

In FIG. 2, the view from the viewpoint of member M-1 riding in vehicle V-1 is taken as an example, but the same applies to the view from the viewpoint of member M-2 riding in vehicle V-2 be. However, if both member M-1 and member M-2 use the virtual experience provision method, for member M-1 and member M-2, display units provided in windows in opposite directions It is preferable to display a virtual companion image. That is, for example, for member M-1, the right window displays a virtual co-located image containing the image of member M-2, and for member M-2, the left window contains the image of member M-1. Display a virtual co-located image. As a result, the positions and directions of the members M-1 and M-2 in the real world and the members M-1 and M-2 in the virtual image match each other in the respective vehicle interiors. It is possible to communicate without any sense of incongruity, such as being able to face the virtual passenger and talk with each other with natural movements of -2.

In the embodiment, the image of the member M-2 riding in another vehicle is displayed to provide the member M-1 with the feeling of riding with the member M-2. By displaying the omitted image (displaying the virtual interior of the vehicle V), it is possible to provide the member M-1 with a sense of enlargement of the vehicle interior.

Hereinafter, a configuration example of the ride-on experience providing system 1 to which the virtual experience providing method according to the embodiment is applied will be described more specifically.

FIG. 3 is a diagram showing a configuration example of the ride-sharing experience providing system 1 according to the embodiment. Here, a configuration example in a situation where two vehicles communicate with each other will be given. As shown in FIG. 3, the ride-on experience providing system 1 includes a control device 10-1 and a control device 10-2.

The control device 10-1 is mounted on the vehicle V-1. Control device 10-2 is mounted on vehicle V-2. Control device 10-1 and control device 10-2 are provided so as to be able to communicate with each other via network N. FIG. The network N is a communication network that enables inter-vehicle communication, such as a cellular V2X (C-V2X: Cellular Vehicle to Everything) communication network, a short-range wireless communication network, the Internet, or the like.

Next, FIG. 4 is a schematic plan view showing a configuration example of the vehicle V according to the embodiment. As shown in FIG. 4, the vehicle V is equipped with cameras 3a, 3b, 3c, 3d, 3e and 3f, displays 5a, 5b and 5c, and speakers 6a, 6b, 6c and 6d.

The cameras 3a, 3b, 3c, 3d, 3e, and 3f correspond to an example of the camera section 3 described later. The displays 5a, 5b, and 5c correspond to examples of the display device 5 described above and the display section 7 described later. The speakers 6a, 6b, 6c, and 6d correspond to an example of the audio output section 6, which will be described later.

The camera 3a is provided so as to be able to photograph the member M viewed from the right direction in FIG. The camera 3b is provided so as to be able to photograph the member M viewed from the left direction in FIG. The camera 3c is provided so as to photograph the member M viewed from the front. The camera 3d is provided so as to be able to capture a landscape image seen to the left of the vehicle V in FIG. The camera 3e is provided so as to be able to capture a landscape image seen to the right of the vehicle V in FIG. The camera 3f is provided so as to be able to photograph a landscape image behind the vehicle V (lower side in FIG. 4).

The position, shape, and position data of each window of the vehicle V are stored. By using these data and the viewpoint position data, it is possible to extract and generate a car window landscape image seen from an arbitrary viewpoint position in the vehicle cabin from the landscape images photographed by the cameras 3d, 3e, and 3f. can.

For the image of the member M, the images taken by the cameras 3a, 3b, and 3c are used for image recognition using, for example, a model for human detection, and for moving object recognition (the interior of the vehicle is stationary with respect to the camera, and the scenery around the vehicle is moving). A human portion can be cut out and generated by a process such as movement (movement associated with movement of a vehicle, movement of a person regardless of vehicle movement). Images captured by the cameras 3a, 3b, and 3c, that is, images of the member M from three different directions can be obtained, and 3D (three-dimensional) image data of the member M can be generated based on these images.

The display 5a is arranged in the right window. A display 5b is arranged in the left window. The display 5c is arranged in the vicinity of the camera 3c, for example, displays an image captured by the camera 3c in a mirror-inverted manner, and is provided so as to function as a digital rearview mirror.

The speakers 6a, 6b, 6c, and 6d are configured by flat speakers, for example, and are provided by being embedded in the interior (wall surface) of the vehicle compartment. A speaker 6a is arranged near the right window. The speaker 6b is arranged near the left window. Speakers 6c and 6d are arranged behind member M, for example, near the rear window.

Next, FIG. 5 is a block diagram showing a configuration example of the control device 10 according to the embodiment. It should be noted that FIG. 5 shows only the constituent elements necessary for explaining the features of this embodiment, and omits the description of general constituent elements.

Moreover, each component illustrated in FIG. 5 is functionally conceptual, and does not necessarily need to be physically configured as illustrated. For example, the specific form of distribution/integration of each block is not limited to the one shown in the figure. It is possible to integrate and configure.

In addition, in the description using FIG. 5, the description of components that have already been described may be simplified or omitted.

Also, although FIG. 5 shows the control device 10-1, the configuration of the control device 10-2 is the same.

As shown in FIG. 5, an operation unit 2, a camera unit 3, an audio input unit 4, an audio output unit 6, and a display unit 7 are connected to the control device 10. FIG. The operation unit 2 is composed of operation input components such as push buttons and a touch panel for receiving operations by users (that is, each member M).

The camera unit 3 is an in-vehicle camera mounted on the vehicle V, and corresponds to the aforementioned cameras 3a, 3b, 3c, 3d, 3e, and 3f. The voice input unit 4 is, for example, a microphone, which is set in front of the user such as a steering wheel or a rearview mirror, and collects voice uttered by the user.

The audio output unit 6 corresponds to the above-described speakers 6a, 6b, 6c, and 6d, and outputs, for example, voices uttered by virtual fellow passengers into the vehicle interior.

The display unit 7 is, for example, a window display incorporated in the window of the vehicle V and corresponds to the displays 5a, 5b, 5c described above. The display unit 7 displays visual information including a virtual co-riding image in the vehicle interior based on output control by the control unit 13, which will be described later.

The control device 10 also includes a communication unit 11 , a storage unit 12 and a control unit 13 . The communication unit 11 is implemented by, for example, a NIC (Network Interface Card) or the like. The communication unit 11 is wirelessly connected to an external device (for example, the control device 10-2 in the case of the control device 10-1) via the network N, and transmits and receives various types of information to and from the external device.

For example, the communication unit 11 establishes a communication connection with an external device when the virtual ride experience provision function is turned on, and establishes a communication connection with the external device when the virtual ride experience provision function is turned off. disconnect.

The storage unit 12 is implemented by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. In the example of FIG. 5, the storage unit 12 is provided with a layout information table 12A that stores layout information 12a, an interior DB (database) 12B that stores interior decoration information 12b, and an avatar DB 12C that stores avatar information 12c. .

The layout information 12a is information regarding the layout position of the virtual fellow passenger in the virtual vehicle interior space, etc., when executing the function of providing the virtual fellow passenger experience. The layout information 12a is set by a setting unit 13a, which will be described later, based on the user's operation of the operation unit 2, for example.

FIG. 6 is a diagram showing an example of the layout information table 12A. As shown in FIG. 6, the layout information 12a is obtained, for example, from the data of the "virtual passenger" item, the "network" item, the "layout position" item, the "interior ID" item, and the "avatar ID" item. These data form one record of the layout information table 12A.

In order to make the explanation easier to understand, only the item name and type name are described for each of the above items and types (indicating the type of data, etc.), and the item name is used for the data related to the item and type. It is expressed by adding "data". For example, the item "virtual fellow passenger" is expressed as "virtual fellow passenger", and the data itself is expressed as "virtual fellow passenger data".

In the layout information table 12A, the identifier data of the member M of the virtual fellow passenger candidate is set for the virtual fellow passenger. It should be noted that data of non-accompanied persons other than companions can also be set for such virtual fellow passengers. The virtual fellow passenger is the primary key of the layout information 12a, and each data of the same record is stored in association with the virtual fellow passenger data, which is the primary key data. In other words, the person indicated by the virtual fellow passenger data is associated with data such as the network and layout position (features indicated by the data).

Information related to the network necessary for establishing a communication connection with the virtual passenger, such as a communication type and an IP (Internet Protocol) address, is set in the network.

The layout position is set with layout position data of the virtual fellow passenger in the virtual vehicle interior space. In the example of FIG. 6, member M-2 as a virtual fellow passenger is laid out in the front passenger seat, and member M-3 is laid out in the right rear seat.

Interior ID data, which is identification information of an interior image of a virtual vehicle interior space, is set in the interior ID. An example of the interior DB 12B will now be described. FIG. 7 is a diagram showing an example of the interior DB 12B.

As shown in FIG. 7, in the interior DB 12B, data related to the interior image associated with each interior ID data is set. In other words, the interior image data and the description data of the image are set in association with the interior ID data as primary key data. In the example of FIG. 7, for example, the interior image data associated with the interior ID "I011" indicates "passenger seat of luxury car", and the description data indicates "actual image of passenger seat of luxury car". Similarly, the interior image associated with the interior ID "I012" indicates "rear seat of a luxury car", and the description data indicates "a photographed image in the direction of the rear seat of a luxury car".

Note that the interior image data does not have to be the photographed image data. For example, in the example of FIG. 7, the interior image data linked to the interior ID "I021" is the image data of "spaceship passenger seat", which is computer graphics image data generated by a computer graphics method or the like. indicates that there is Similarly, the interior image data associated with the interior ID "I022" is the image data of the "backseat of the spacecraft", indicating that it is computer graphics image data.

Returning to the description of FIG. The avatar ID is set with avatar ID data that is identification information of the avatar to be used when the avatar is to be used to display the virtual fellow passenger.

Here, an example of avatar DB12C is demonstrated. FIG. 8 is a diagram showing an example of the avatar DB 12C. As shown in FIG. 8, in the avatar DB 12C, data related to avatar images linked to each avatar ID data described above is set. In other words, the avatar image data and the description data of the avatar image data are set in association with the avatar ID data as primary key data.

Such avatar image data may be, for example, a photographed image of an actor or the like that actually exists, such as the avatar image data with the avatar ID "A001", or the avatar image data with the avatar ID "A002", Computer graphics images such as anime characters that do not exist in reality may also be used. As a result, the user's favorite avatar can ride as a virtual fellow passenger regardless of whether it actually exists or not.

Further, each interior image data and each avatar image data shown in FIGS. 7 and 8 are image data capable of 3D display and moving image reproduction display.

Returning to the description of FIG. If avatar ID data is set in the avatar ID of the layout information table 12A, the virtual fellow passenger is replaced with an avatar using avatar image data corresponding to the set avatar ID data. If no avatar ID data is set in the avatar ID, the virtual fellow passenger is not replaced with an avatar, and the actual photographed image of the person is used.

Returning to the description of FIG. The control unit 13 is a controller, and various programs (not shown) stored in the storage unit 12 are executed by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, using the RAM as a work area. It is realized by being Also, the control unit 13 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 13 includes a setting unit 13a, an acquisition unit 13b, a generation unit 13c, an output control unit 13d, and a transmission unit 13e, and realizes or executes information processing functions and actions described below.

Based on the user's operation of the operation unit 2, the setting unit 13a sets each piece of information related to the function of providing the virtual ride-on experience, that is, the layout information 12a, the interior information 12b, and the avatar information 12c used for realizing the function. Select and set as data for control.

The acquisition unit 13b acquires the car window scenery image data photographed by the camera unit 3 . In addition, the acquisition unit 13b acquires image data and audio data of each member M who becomes a virtual fellow passenger.

For example, when the member M-2 of the control device 10-2 is assumed to be a virtual fellow passenger in the vehicle V-1 in which the control device 10-1 is mounted, the acquisition unit 13b of the control device 10-1, via the communication unit 11, , the image data of the member M-2 photographed by the control device 10-2 and the sound data of the member M-2 collected by the control device 10-2 are obtained.

In addition, the acquisition unit 13b acquires image data of the member M captured by the camera unit 3, and causes the transmission unit 13e to transmit the image data to another control device 10 for which the member M is desired to be a virtual fellow passenger. In addition, the acquisition unit 13b acquires voice data of the member M collected by the voice input unit 4, and causes the transmission unit 13e to transmit the voice data to another control device 10 that wants to use the member M as a virtual fellow passenger.

The generation unit 13c converts the image data of the car window scenery acquired by the acquisition unit 13b, the interior image data according to the layout information 12a, and the image data of the member M who is a virtual fellow passenger into the layout position of each member M and the driving position. Processing is performed according to the position (viewpoint position) of the person (passenger of the vehicle displaying the virtual image) (viewpoint position conversion, clipping processing, etc. are performed on the image seen from the viewpoint position), and these processed images are superimposed. to generate a virtual co-located image. At this time, the generator 13c superimposes the interior image data on the car window scenery image data, and further superimposes the image data of the member M on the interior image data to generate a virtual co-riding image.

The output control unit 13d causes the display unit 7 to display the virtual shared image generated by the generation unit 13c. In addition, the output control unit 13d outputs the voice data of the member M who is the virtual fellow passenger acquired by the acquisition unit 13b as if the virtual fellow passenger is speaking. In other words, sound image localization control such as selecting the speakers 6a, 6b, 6c, and 6d for outputting sound and adjusting the level and phase of the sound output from each speaker is performed, and the localization feeling according to the layout of each virtual passenger is performed. to output the spoken voice. In addition, the output control unit 13d also performs air conditioning control such as temperature control of the air conditioner according to the layout of each virtual fellow passenger so that the body temperature of the virtual fellow passenger can be felt.

A specific example of such audio output and temperature control will be described with reference to FIGS. 9 and 10. FIG. FIG. 9 is an explanatory diagram (part 1) of audio output and temperature control. FIG. 10 is an explanatory diagram (2) of audio output and temperature control.

As shown in FIG. 9, the output control unit 13d causes the speaker 6b incorporated in the display device 5, for example, to output sound as if the member M-2, who is a virtual fellow passenger, is speaking. In other words, the sound is output from the speaker 6b in the direction of the virtual fellow passenger to simulate a state in which the virtual fellow passenger is speaking. At this time, the speaker 6b is preferably arranged near the position (direction) corresponding to the mouth of the virtual passenger.

Further, as shown in the figure, the output control unit 13d performs temperature control so that the body temperature of the virtual passenger can be felt by controlling the heat source 8 incorporated near the display device 5, for example. At this time, the heat source 8 is preferably arranged near the position corresponding to the body of the virtual passenger. As the heat source 8, an infrared heater, an in-vehicle air conditioner (a hot air outlet is installed near the display device 5), or the like may be used.

Regarding the output of audio, the output control unit 13d performs sound image localization control for the audio output unit 6, for example, as shown in FIG. sound image localization control by adjusting the phase and volume of the voice of a remote person corresponding to the person) according to the positions of the speakers 6a, 6b, 6c, and 6d and the target sound image localization position). , the sound image may be localized at a position corresponding to the mouth of the virtual passenger. In such a case, the speakers 6a, 6b, 6c, and 6d of the audio output unit 6 do not necessarily have to be incorporated in the display device 5, and normal vehicle-mounted speakers for sound reproduction devices can be used.

Returning to the description of FIG. The transmission unit 13e transmits the image data of the member M captured by the camera unit 3 and the voice data of the member M collected by the voice input unit 4 to the other control device 10 via the communication unit 11. Send.

By the way, until now, the case where the display device 5 arranged on the left or right side of the member M who is riding in the vehicle V is displayed as an example of a virtual shared image has been taken as an example. A virtual co-passenger image may be displayed on the display 5c.

An example of such a case will be described with reference to FIG. FIG. 11 is an explanatory diagram of a case where a virtual co-riding image is displayed on the display 5c functioning as a digital rearview mirror. Note that, in the explanation using FIG. 11 as well, the case seen from the viewpoint of the member M-1 who gets on the vehicle V-1 will be taken as an example.

In such a case, as shown in FIG. 11, the generation unit 13c generates an actual rearview mirror image (image taken by the camera 3c) viewed by the member M-1 from the driver's seat and an interior image of the vehicle selected by the member M-1. For example, the interior image of a vehicle with rear seats and the image of the member M at another location are superimposed and displayed on the display device 5 . The images of the member M at other locations are the image of the member M-2 and the image of the other member M-3 who are riding in the vehicle V-2.

At this time, the generation unit 13c superimposes the interior image on the rearview mirror image and further superimposes the image of each member M on the interior image to generate a virtual shared image.

As a result, although member M-1 is a single-seater vehicle V-1, there are rear seats in the vehicle interior, and members M-2 and M-3 are sitting in the rear seats. It is possible to simulate a situation as if one were riding in the vehicle V-1. That is, it is possible to provide the member M-1 with a ride experience that is not restricted by the actual vehicle V-1. In addition, since it is possible to make the member M-1 feel that the passenger compartment space is wider toward the rear seat than it actually is, it is possible to improve comfort during riding.

In the case of the example of FIG. 11, the output control unit 13d outputs the voice of each member M from the speaker 6d shown in FIG. are output from the speaker 6c shown in FIG. Alternatively, sound image localization processing may be performed using a plurality of speakers 6a, 6b, 6c, and 6d.

It is also possible to display the virtual passenger image of the front passenger seat shown in FIG. 2 and the virtual passenger image of the rear seat shown in FIG. . In this case, the control device 10-1 of the vehicle V-1 relays voice signals between the control devices 10 of the related vehicles, etc., so that the members M can talk with each other.

Next, a processing procedure executed by the control device 10 will be described with reference to FIG. 12 . FIG. 12 is a flow chart showing a processing procedure executed by the control device 10 according to the embodiment. The processing of this flowchart is repeatedly executed while the vehicle is running.

As shown in FIG. 12, first, the control unit 13 determines whether or not the provision function of the virtual ride experience is turned on (step S101).

If the same function is not turned on (step S101, No), the control unit 13 terminates the process. When the function is turned on (step S101, Yes), the control unit 13 reads each piece of information related to the provision function of the virtual ride experience set in advance based on the operation of the operation unit 2 by the user (step S102). Then, based on the layout information 12a, the communication unit 11 establishes a communication connection with the member M who is a virtual fellow passenger (step S103).

After establishing the communication connection, the control unit 13 acquires the necessary images and voices of each member M based on the layout information 12a (step S104). Then, the control unit 13 superimposes the actual car window scenery image, the interior image, and the image of each member M to generate a virtual co-riding image (step S105).

Then, the control unit 13 displays each generated virtual fellow passenger image on the display unit 7 (in the direction) determined as the display destination for each virtual fellow passenger (step S106). In addition, the control unit 13 determines a speaker for each virtual fellow passenger to output as if the virtual fellow passenger is speaking (or sets each control parameter in the sound image localization process), and outputs the voice of each member M. (step S107).

Then, the control unit 13 determines whether or not the provision function of the virtual ride experience is turned off (step S108).

If the same function is not turned off (step S108, No), the control section 13 repeats the processing from step S104. When the function is turned off (step S108, Yes), the communication unit 11 disconnects the communication connection with the member M who is the virtual fellow passenger (step S109). Then, the process ends.

By the way, until now, based on the inter-vehicle communication between the control devices 10, the case where each of the control devices 10 executes the function of providing a virtual ride experience has been taken as an example. 100 may collectively perform the same function.

FIG. 13 is a diagram showing a configuration example of a ride-sharing experience providing system 1A according to a modification. As shown in FIG. 13 , the ride-on experience providing system 1A further includes a server device 100 .

The server device 100 is provided so as to be able to communicate with each of the control devices 10 via the network N, and when receiving a request from the control device 10 for the function of providing a virtual ride-on-ride experience, the server device 100 selects a target vehicle for the ride-on-ride experience. A car window scenery image, an image of each member M, and a sound are acquired from each of the control devices 10 of the vehicle.

Server device 100 also holds comprehensive layout information table 12A, interior DB 12B, and avatar DB 12C, and generates virtual co-riding images and sounds for each control device 10 based on this information.

Then, the server device 100 transmits the generated virtual co-riding image and voice to each control device 10 and causes each control device 10 to output them.

Moreover, as already mentioned, not only the member M who is the accompanying person but also the non-accompanying member M can virtually ride with the passenger. In such a case, as shown in FIG. 13, the co-riding experience providing system 1A further includes a non-companion device 30. FIG.

The non-accompanying person device 30 is, for example, a device set in a home or the like used by a member M-3 who does not accompany the traveler but who wants to enjoy group activities together with the members M-1 and M-2 who use the vehicle V. is.

Like the control device 10, the non-accompanying person device 30 has a camera unit 3, an audio input unit 4, a display device 5, and an audio output unit 6. , acquisition processing, generation processing, output control processing, and transmission processing. The installation positions of the camera unit 3 and the audio output unit 6 in the non-accompanied person device 30 are specified by the user in a manual or the like, or the installation conditions (position, orientation, etc.) are specified by the user. 30 so that the non-accompanied person device 30 performs image processing (view point conversion processing, etc.) and audio processing (localization adjustment processing) based on the installation conditions of the camera unit 3 and the audio output unit 6 defined by these. If so, a more realistic ride-on experience can be realized.

Then, the non-accompanying person device 30 transmits the image and voice of the member M-3 to the server device 100, and the virtual companion device for the control device 10 including the member M-3 on the server device 100 as a virtual fellow passenger. Generate images and sounds. In addition, the server device 100 generates a virtual companion image and sound including the member M-1 and the member M-2 as virtual fellow passengers for the non-accompanying person device 30 .

Then, the non-accompanying person device 30 acquires the virtual fellow passenger image and the audio for the non-accompanying person device 30 from the server device 100 and causes the display device 5 and the audio output unit 6 to output them.

As a result, it is possible to share a virtual ride experience among the members M of the group, for example, even between remote locations.

Among members M of the same group, especially when a non-accompanied person such as member M-3 is included, it is preferable that the same image be superimposed on the virtual co-passenger image for the car window scenery image.

In addition, the scenery image of the front car window (windshield) was not displayed in the above embodiment because it interferes with driving. For passengers, a landscape image of the front window (taken by installing a camera) may be displayed on a display in the house or on a display installed on the windshield of the automatic driving vehicle.

This makes it easier for each member M to communicate on a common topic based on, for example, the same car window scenery image, and enables the group activities to proceed smoothly.

Further, although the display device 5 has been assumed to be a non-transmissive display so far, the display device 5 may be a transmissive display. In such a case, the control unit 13 may superimpose the interior image and the image of the member M who is a virtual fellow passenger on the actual train window scenery through the display without superimposing the train window scenery image. .

Also, until now, the display device 5 was a display provided on a window or a rearview mirror, but the display device 5 is not limited to the window or the rearview mirror, and the display device 5 uses the wall surface of the vehicle interior including the window as the display screen. It may be a display.

As described above, the control device 10 (corresponding to an example of an “information processing device”) according to the embodiment includes the control section 13 for displaying an image on the display section 7 provided in the vehicle compartment of the vehicle V. It is a device. The control unit 13 causes the display unit 7 to display a virtual interior image of a vehicle interior space larger than the vehicle interior.

Therefore, according to the control device 10 according to the embodiment, it is possible to reduce the user's dissatisfaction caused by the type of vehicle used.

In addition, the control unit 13 superimposes a landscape image around the vehicle on the car window portion of the virtual interior image.

Therefore, according to the control device 10 according to the embodiment, it is possible to reflect the landscape image around the vehicle in the virtual interior image.

In addition, the control unit 13 displays a virtual passenger image in which an image of a virtual passenger (corresponding to an example of a “virtual passenger”) based on an image of a real person existing at another location is superimposed on the virtual interior image. Display on part 7.

Therefore, according to the control device 10 according to the embodiment, it is possible to provide a virtual experience in which a real person existing at another location virtually rides with the vehicle.

The control unit 13 also localizes the voice of the virtual fellow passenger according to the display position of the virtual fellow passenger.

Therefore, according to the control device 10 according to the embodiment, it is possible to output the voice of the member M, who is a virtual fellow passenger, as if the virtual fellow passenger were speaking.

In addition, the control unit 13 causes the display unit 7 provided in the rearview mirror inside the vehicle to display the virtual co-riding image.

Therefore, according to the control device 10 according to the embodiment, although each member M is a one-seat vehicle V, there is a rear seat in the vehicle interior, and another member M sits in the rear seat. You can simulate the situation as if you were in the car. That is, it is possible to provide each member M with a ride experience that is not restricted by the actual vehicle V. In addition, since it is possible to make each member M feel that the vehicle interior space is wider toward the rear seat than it actually is, it is possible to improve comfort during riding.

Further, when the control unit 13 displays the virtual co-riding image in cooperation with the control device 10 of the vehicle V at the other location, the virtual co-riding image is displayed so that the relative positional relationship between the real person and the virtual fellow passenger in each vehicle V matches. show who

Therefore, according to the control device 10 according to the embodiment, the positions and directions of the actual members M and the virtual images of the members M match each other in the respective vehicle interiors, so that the members M can be arranged in a natural manner. It is possible to communicate without any sense of incongruity, for example, it is possible to face the virtual passenger and have a conversation with each other through movement.

Further, when superimposing a landscape image around the vehicle on the car window portion of the virtual interior image, the control unit 13 uses the same landscape image as the landscape displayed on the vehicle V at the other point for display.

Therefore, according to the control device 10 according to the embodiment, it is possible to provide a virtual experience of sharing the scenery from the train window with the member M who is present at another location.

In addition, the control device 10 according to the embodiment includes cameras 3a, 3b, and 3c (corresponding to an example of a "vehicle interior camera") that capture images inside the vehicle, and cameras 3d, 3e, and 3f that capture the scenery around the vehicle ( (equivalent to an example of a "landscape camera"), an audio input unit 4 (equivalent to an example of a "microphone") that acquires the sound in the vehicle interior, speakers 6a, 6b, 6c, and 6d that output sound to the interior of the vehicle, It is an in-vehicle virtual display device having a display unit 7 installed near a side car window in the vehicle interior, a communication unit 11 that communicates with an external device at another location, and a control unit 13 that performs various controls.

The control unit 13 receives a virtual interior image of a vehicle interior space larger than the vehicle interior, a landscape image of the vehicle surroundings captured by the cameras 3d, 3e, and 3f, and images received by the communication unit 11 from an external device at another location. A virtual fellow passenger image is generated by superimposing a virtual fellow passenger image based on image data of a real person present at the spot. Further, the control unit 13 causes the display unit 7 to display the generated virtual co-riding image. Further, the control unit 13 causes the speakers 6a, 6b, 6c, and 6d to output sounds corresponding to the voice data of the real person existing at the other location from the external device at the other location received by the communication unit 11 . Further, the control unit 13 causes the communication unit 11 to transmit voice data corresponding to the voice acquired by the voice input unit 4 to an external device at another location. Further, the control unit 13 causes the communication unit 11 to transmit vehicle interior image data corresponding to images captured by the cameras 3a, 3b, and 3c to an external device at another location. Further, the control unit 13 causes the communication unit 11 to transmit scenery image data corresponding to the images captured by the cameras 5d, 5e, and 5f to an external device at another location.

Therefore, according to the control device 10 according to the embodiment, it is possible to reduce the user's dissatisfaction caused by the type of vehicle used while cooperating with an external device at another location.

Also, the ride-sharing experience providing system 1 according to the embodiment is a virtual display system configured by a plurality of control devices 10 capable of mutual communication. Each of the control devices 10 includes cameras 3a, 3b, and 3c for capturing images inside the vehicle, cameras 3d, 3e, and 3f for capturing scenery around the vehicle, an audio input unit 4 for acquiring audio in the vehicle, and Speakers 6a, 6b, 6c, and 6d for outputting sound into the cabin, a display unit 7 installed near the side windows in the cabin, a communication unit 11 for communicating with other control devices 10, and various controls. and a control unit 13 for performing.

The control unit 13 receives a virtual interior image of a vehicle interior space larger than the vehicle interior, landscape images of the vehicle surroundings captured by the cameras 3d, 3e, and 3f, and other control devices 10 at other locations received by the communication unit 11. A virtual fellow passenger image is generated by superimposing a virtual fellow passenger image based on image data of a real person existing at the other location. Further, the control unit 13 causes the display unit 7 to display the generated virtual co-riding image. Further, the control unit 13 outputs voice corresponding to the voice data of the real person existing at the other location from the other control device 10 at the other location received by the communication unit 11 from the speakers 6a, 6b, 6c, and 6d. output. Further, the control unit 13 causes the communication unit 11 to transmit voice data corresponding to the voice acquired by the voice input unit 4 to the other control device 10 . Further, the control unit 13 causes the communication unit 11 to transmit vehicle interior image data corresponding to the images captured by the cameras 3 a , 3 b , and 3 c to the other control device 10 . Further, the control unit 13 causes the communication unit 11 to transmit scenery image data corresponding to the images captured by the cameras 5d, 5e, and 5f to the other control device 10 .

Therefore, according to the control device 10 according to the embodiment, while cooperating with other control devices 10 at other locations, it is possible to reduce user dissatisfaction caused by the type of vehicle used.

When the virtual fellow passenger image is displayed in cooperation with the other control device 10, the control unit 13 displays the virtual fellow passenger so that the relative positional relationship between the real person and the virtual fellow passenger in each vehicle V matches. Let

Therefore, according to the co-riding experience providing system 1 according to the embodiment, the positions and directions of the actual members M and the virtual images of the members M match each other in each vehicle interior. It is possible to communicate without any sense of incongruity, for example, it is possible to have a conversation facing the virtual passenger with natural movements.

Further, the virtual experience providing method according to the embodiment is a virtual experience providing method for displaying a virtual fellow passenger in the vehicle interior of the vehicle V, wherein the virtual interior image of the vehicle interior space larger than the vehicle interior is obtained by communication. A virtual fellow-passenger image is displayed by superimposing a virtual fellow-passenger image based on an image of a real person existing at another point.

Therefore, according to the virtual experience providing method according to the embodiment, it is possible to reduce the user's dissatisfaction caused by the type of vehicle used.

In addition, in the embodiment described above, the vehicle V is assumed to be a micro-mobility, but the vehicle V is not limited to a micro-mobility. For example, the vehicle V may be able to accommodate two or more people. Even if the vehicle V can accommodate two or more people in this way, by using the interior image of the wider vehicle interior space, the virtual fellow passenger can ride in the wider interior space with respect to the occupant of the vehicle V. You can simulate the situation you are in.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

1, 1A Riding Experience Providing System 2 Operation Unit 3 Camera Unit 4 Audio Input Unit 5 Display Device 6 Audio Output Unit 7 Display Unit 8 Heat Source 10 In-vehicle Device 11 Communication Unit 12 Storage Unit 12A Layout Information Table 12B Interior DB
12C Avatar DB
13 control unit 13a setting unit 13b acquisition unit 13c generation unit 13d output control unit 13e transmission unit 30 non-accompanying person device 100 server device V vehicle

Claims (11)

An information processing device having a control unit for displaying an image on a display unit provided in a vehicle interior,
The control unit
causing the display unit to display a virtual interior image of a vehicle interior space larger than the vehicle interior;
Information processing equipment. The control unit
superimposing a landscape image around the vehicle on the car window portion of the virtual interior image;
The information processing device according to claim 1 . The control unit
causing the display unit to display a virtual passenger image in which an image of a virtual fellow passenger based on an image of a real person existing at another location is superimposed on the virtual interior image;
The information processing apparatus according to claim 1 or 2. The control unit
localizing the voice of the virtual fellow passenger according to the display position of the virtual fellow passenger;
The information processing apparatus according to claim 3. The control unit
displaying the virtual co-riding image on the display unit provided in the rearview mirror in the vehicle compartment;
The information processing apparatus according to claim 3 or 4. The control unit
When displaying the virtual fellow passenger image in cooperation with an information processing device of a vehicle at another location, the virtual fellow passenger is displayed such that the relative positional relationship between the real person in each vehicle and the virtual fellow passenger matches. let
The information processing apparatus according to claim 3, 4 or 5. The control unit
When an image of the scenery around the vehicle is superimposed on the car window portion of the virtual interior image, the same scenery image as that displayed on the vehicle at another point is used for display.
The information processing device according to claim 6 . an in-vehicle camera for capturing an image inside the vehicle;
A landscape camera that captures the scenery around the vehicle,
a microphone that acquires the sound in the vehicle interior;
a speaker for outputting sound into the vehicle interior;
A display unit installed near the side car window in the passenger compartment,
a communication unit that communicates with an external device at another location;
An in-vehicle virtual display device having a control unit that performs various controls,
The control unit
A virtual interior image of a vehicle interior space larger than the vehicle interior, a scenery image around the vehicle captured by the scenery camera, and a real person existing at the other location received from the external device at the other location received by the communication unit. Generate a virtual fellow passenger image by superimposing a virtual fellow passenger image based on the image data of
displaying the generated virtual co-riding image on the display unit;
causing the speaker to output a voice corresponding to the voice data of a real person existing at the other location received from the external device at the other location, received by the communication unit;
causing the communication unit to transmit audio data corresponding to the audio acquired by the microphone to the external device at the other location;
causing the communication unit to transmit vehicle interior image data corresponding to an image captured by the vehicle interior camera to the external device at the other location;
causing the communication unit to transmit landscape image data corresponding to the image captured by the landscape camera to the external device at the other location;
In-vehicle virtual display. A virtual display system comprising a plurality of in-vehicle virtual display devices capable of communicating with each other,
Each of the in-vehicle virtual display devices includes:
an in-vehicle camera for capturing an image inside the vehicle;
A landscape camera that captures the scenery around the vehicle,
a microphone that acquires the sound in the vehicle interior;
a speaker for outputting sound into the vehicle interior;
A display unit installed near the side car window in the passenger compartment,
a communication unit that communicates with other in-vehicle virtual display devices;
and a control unit that performs various controls,
The control unit
A virtual interior image of a vehicle interior space larger than the vehicle interior, a landscape image of the surroundings of the vehicle captured by the landscape camera, and the other location from the other in-vehicle virtual display device received by the communication unit. generating a virtual passenger image by superimposing an image of a virtual fellow passenger based on image data of an existing real person;
Displaying the generated virtual co-riding image on the display unit,
causing the speaker to output a sound corresponding to the voice data of the real person existing at the other location from the other in-vehicle virtual display device received by the communication unit;
causing the communication unit to transmit audio data corresponding to the audio acquired by the microphone to the other in-vehicle virtual display device;
causing the communication unit to transmit vehicle interior image data corresponding to an image captured by the vehicle interior camera to the other in-vehicle virtual display device;
causing the communication unit to transmit scenery image data corresponding to an image captured by the scenery camera to the other in-vehicle virtual display device;
Virtual display system. The control unit
When displaying the virtual fellow passenger image in cooperation with the other in-vehicle virtual display device, displaying the virtual fellow passenger so that the relative positional relationship between the real person and the virtual fellow passenger in each vehicle matches,
10. A virtual display system according to claim 9. A virtual experience providing method for displaying a virtual passenger in a vehicle interior, comprising:
Displaying a virtual passenger image in which a virtual interior image of a passenger compartment space larger than the passenger compartment and an image of a virtual fellow passenger based on an image of a real person existing at another location obtained by communication are superimposed;
How we provide virtual experiences.

Images