WO2021095198A1

WO2021095198A1 - Meetup assistance device and meetup assistance method

Info

Publication number: WO2021095198A1
Application number: PCT/JP2019/044677
Authority: WO
Inventors: 下谷　光生; 直志宮原; 義典上野; 圭作福田; 家田　邦代; 祐介荒井
Original assignee: 三菱電機株式会社
Priority date: 2019-11-14
Filing date: 2019-11-14
Publication date: 2021-05-20
Also published as: JPWO2021095198A1; JP7282199B2

Abstract

A meetup assistance device (10) is provided with a meetup point storage unit (12) for storing a meetup point of a user and another person, a virtual object storage unit (13) for storing a virtual meetup point object, which is an image indicating the position of the meetup point, and a position information acquisition unit (14) for acquiring the position of the user. In addition, on the basis of the position of the user and the position of the meetup point a display control unit (16) of the meetup assistance device (10) displays the virtual meetup point object at a position corresponding to a meetup point in an actual scene that can be seen by the user through a transparent screen or in an image of the user's surroundings as displayed on the screen.

Description

Merge support device and merge support method

The present invention relates to a merging support device that supports merging between a user and another person (merging partner).

In the automobile industry, with the keyword "sharing," services such as ride hailing, ride sharing, and dynamic shuttles are being developed and put into practical use so that service users can use vehicles owned by others. .. In these services, it is important to facilitate the merging of service users and vehicles.

For example, Patent Documents 1 and 2 below propose a merging support device that supports merging by displaying a merging point (meeting place) on a map. Further, in Patent Document 3 below, the service user can correctly identify the reserved vehicle by providing the terminal device of the service user with an authenticator for authenticating the reserved vehicle to be the merging partner in advance. The system has been proposed.

Japanese Unexamined Patent Publication No. 2019-053547 Japanese Unexamined Patent Publication No. 2016-085334 Japanese Unexamined Patent Publication No. 2019-28629

According to the merging support device of Patent Documents 1 and 2, the user can confirm the position of the merging point on the map. However, if the user is not good at reading the map, or if there are no landmarks (so-called "landmarks") near the confluence, it is difficult to match the map with the real world. The user may not be able to easily reach the confluence.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a merging support device capable of presenting the location of a merging point in the real world to a user.

The merging support device according to the present invention includes a merging point storage unit that stores a merging point between a user and another person, a virtual object storage unit that stores a virtual merging point object that is an image showing the position of the merging point, and a user. Based on the position information acquisition unit that acquires the position and the position of the user and the position of the confluence point, the virtual confluence point object can be viewed from the user through a transparent screen, or the actual view around the user displayed on the screen. It is provided with a display control unit that displays at a position corresponding to a confluence point in an image.

According to the present invention, it is possible to present to the user a virtual world in which a virtual confluence point object exists at a position corresponding to a confluence point in the real world. Therefore, the user can intuitively recognize the position of the confluence in the real world.

The object, features, aspects, and advantages of the present invention will be made clearer by the following detailed description and accompanying drawings.

It is a figure which shows the structure of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the example of the real world. It is a figure which shows the example of the virtual world provided by the merging support device which concerns on Embodiment 1. FIG. It is a figure which shows the example of the real scene and the virtual confluence point object seen from the user. It is a flowchart which shows the operation of the merging support device which concerns on Embodiment 1. FIG. It is a figure which shows the hardware configuration example of the merging support device. It is a figure which shows the hardware configuration example of the merging support device. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a flowchart for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the structure of the merging support apparatus which concerns on Embodiment 2. It is a figure which shows the image of the real scene and the example of the virtual confluence point object displayed on the display device. It is a flowchart which shows the operation of the merging support apparatus which concerns on Embodiment 2. It is a figure which shows the structure of the merging support device which concerns on Embodiment 3. It is a figure which shows the example of the real world. It is a figure which shows the example of the virtual world provided by the merging support device which concerns on Embodiment 3. It is a figure which shows the image of the real scene and the example of the virtual confluence point object displayed on the screen of a mobile terminal. It is a flowchart which shows the operation of the mobile terminal which concerns on Embodiment 3. It is a flowchart which shows the modification of the operation of the mobile terminal which concerns on Embodiment 3. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 3. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 3. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 3. It is a figure for demonstrating the modification of the merging support apparatus which concerns on Embodiment 3. It is a figure for demonstrating the modification of the mobile terminal which concerns on Embodiment 3. It is a figure for demonstrating the modification of the mobile terminal which concerns on Embodiment 3. It is a figure for demonstrating the modification of the mobile terminal which concerns on Embodiment 3. It is a figure which shows the structure of the merging support system which concerns on Embodiment 4. It is a flowchart which shows the operation of the merging support server in the merging support system which concerns on Embodiment 4. FIG.

<Embodiment 1>
FIG. 1 is a diagram showing a configuration of a merging support device 10 according to the first embodiment of the present invention. As shown in FIG. 1, the merging support device 10 of the first embodiment is mounted on the vehicle 100. However, the merging support device 10 does not have to be permanently installed in the vehicle 100, and may be configured as a portable device that can be brought into the vehicle 100.

In the present embodiment, the user of the merging support device 10 is the driver of the vehicle 100. Unless otherwise specified, the "user" in the following description is intended to be the driver of the vehicle 100.

The merging support device 10 is connected to a head-up display (HUD) 21, a display device 22, an operation input device 23, a positioning device 24, a communication device 25, and a map information storage device 26 included in the vehicle 100.

The head-up display 21 and the display device 22 are means for the merging support device 10 to present various information as images to the user (driver of the vehicle 100). In particular, the head-up display 21 can display an image on the actual scene seen by the user by using the windshield of the vehicle 100 as a transparent screen. The display device 22 is a general display device installed on the center panel or the instrument panel of the vehicle 100, and displays, for example, the operation screen of the merging support device 10. The display device 22 is composed of, for example, a liquid crystal display or an organic EL display (Organic ElectroLuminescence display).

Since the image displayed by the head-up display 21 is displayed in the actual scene, it looks as if it exists in the real world. Hereinafter, the image displayed by the head-up display 21 is referred to as a "virtual object", and the world in which the virtual object is regarded as existing is referred to as a "virtual world".

The operation input device 23 is a means for the user to input an operation for the merging support device 10. The operation input device 23 may be a hardware key such as a push button, an operation lever, or a keyboard, or a software key displayed on the screen. For example, the software key as the operation input device 23 may be displayed on the display device 22, and in that case, the display device 22 and the operation input device 23 may be configured as one touch panel.

The positioning device 24 calculates the position of the vehicle 100 based on a positioning signal received from a GNSS (Global Navigation Satellite System) satellite such as a GPS (Global Positioning System) satellite. Here, the position of the vehicle 100 is assumed to be equal to the position of the user of the merging support device 10. The communication device 25 is a means for the merging support device 10 to communicate with the outside of the vehicle 100.

The map information storage device 26 is a storage medium that stores map information including location information such as roads and POI (Point of Interest). The map information storage device 26 may be provided in the navigation system of the vehicle 100. Further, the map information storage device 26 may be configured as a server installed outside the vehicle 100, in which case the merging support device 10 acquires map information from the map information storage device 26 through the communication device 25.

The merging support device 10 includes a communication unit 11, a merging point storage unit 12, a virtual object storage unit 13, a position information acquisition unit 14, a map information acquisition unit 15, and a display control unit 16.

The communication unit 11 communicates with the outside of the vehicle 100 via the communication device 25. The merging point storage unit 12 memorizes the merging point between the user and another person who is the merging partner. The merging partner may be a pedestrian or a vehicle. The user can operate the operation input device 23 to perform direct or indirect communication with the merging partner via the communication device 25, and can pre-arrange a merging point with the merging partner, and the determined merging point. The points are stored in the confluence point storage unit 12. Here, indirect communication means that the user and the merging partner communicate with each other via another system composed of a server or the like (for example, an arbitration system at the merging point), and direct communication is , Means that the user and the merging partner communicate without going through another system.

The virtual object storage unit 13 stores an image of a virtual merging point object, which is a virtual object indicating the position of the merging point.

The position information acquisition unit 14 acquires the position (user's position) of the vehicle 100 calculated by the positioning device 24. Further, the position information acquisition unit 14 can calculate the direction of the vehicle 100 from the history of the position of the vehicle 100. Even if the position information acquisition unit 14 performs processing for improving the accuracy of the position of the vehicle 100, such as map matching using map information and positioning by autonomous navigation using the speed sensor and the direction sensor of the vehicle 100, the position information acquisition unit 14 may perform processing such as map matching. Good. Further, the orientation of the vehicle 100 may be measured by using the orientation sensor of the vehicle 100.

The map information acquisition unit 15 acquires map information around the position of the vehicle 100 acquired by the position information acquisition unit 14 from the map information storage device 26.

The display control unit 16 is based on the position and orientation of the vehicle 100 acquired by the position information acquisition unit 14, the position of the merging point stored in the virtual object storage unit 13, and the map information acquired by the map information acquisition unit 15. The position (relative position) of the merging point as seen by the user (driver of the vehicle 100) is calculated, and the virtual merging point object is placed at the position corresponding to the merging point in the actual view seen by the user using the head-up display 21. indicate. More specifically, the display control unit 16 virtually merges at the intersection of the windshield and the straight line connecting the user's eyes and the merge point so that the virtual merge point object appears to be superimposed on the merge point when viewed from the user. Display the point object. As a result, to the user, it appears as if a virtual confluence object exists at the confluence. That is, the display control unit 16 provides the user with a virtual world in which the virtual confluence point object appears to exist at the confluence point in the real world.

For example, it is assumed that the point MP in the real world as shown in FIG. 2 is defined as the merging point between the user and the merging partner, and the vehicle 100 driven by the user reaches the vicinity of the merging point MP. Information on the position of the merging point MP is stored in the merging point storage unit 12 of the merging support device 10. In this case, the display control unit 16 uses the head-up display 21 to provide the user with a virtual world as shown in FIG. 3 in which the virtual merging point object 200 exists at the merging point MP. As shown in FIG. 4, the user who is the driver of the vehicle 100 can see the virtual merging point object 200 at a position corresponding to the merging point MP in the actual view seen through the windshield 30 of the vehicle 100.

In addition, in FIG. 3 and the figures shown thereafter, hatching is attached to the virtual object so that the real world and the virtual world can be easily distinguished, but the virtual object actually displayed by the head-up display 21 is used. Does not need to be hatched. Here, as an example of the virtual confluence point object, an image simulating a sign pillar installed at a public transportation stop (bus stop) is used, but the virtual confluence point object may be any image.

As described above, the merging support device 10 according to the first embodiment presents the location of the merging point to the user by presenting the virtual world in which the virtual merging point object appears to exist at the merging point in the real world. .. Therefore, the user can intuitively recognize the location of the confluence in the real world. As a result, it becomes easier for the user to reach the confluence.

Here, the operation of the merging support device 10 will be described with reference to the flowchart of FIG. When the merging support device 10 is activated and the merging point is determined between the user and the merging partner, the merging point storage unit 12 stores the merging point (step S101).

Subsequently, the position information acquisition unit 14 acquires the position of the vehicle 100 calculated by the positioning device 24, that is, the position of the user (step S102). At this time, the position information acquisition unit 14 calculates the direction of the vehicle 100 based on the history of the position of the vehicle 100. The map information acquisition unit 15 acquires map information around the user's position from the map information storage device 26 (step S103).

The display control unit 16 is based on the user's position acquired by the position information acquisition unit 14, the position of the confluence stored in the confluence point storage unit 12, and the map information acquired by the map information acquisition unit 15. It is determined whether or not the position of is near the confluence (step S104).

The determination in step S104 may be determined simply by whether or not the straight line distance from the user to the confluence point is equal to or less than a predetermined threshold value (for example, 100 m). For example, the road on which the user is located and the confluence point are located. Even if the straight line distance from the user to the confluence is short, it may take a long time for the user to reach the confluence, such as when the road crosses over. Therefore, in the present embodiment, the route length (distance) from the user to the confluence is calculated using the map information, and the determination in step S104 is performed depending on whether or not the route length is equal to or less than the threshold value.

If the user's position is near the merging point (YES in step S104), the display control unit 16 indicates the position and orientation of the vehicle 100 acquired by the position information acquisition unit 14 and the merging point stored in the virtual object storage unit 13. Based on the position and the map information acquired by the map information acquisition unit 15, the position of the confluence point as seen by the user is calculated (step S105). Then, it is confirmed whether or not the position of the confluence point as seen by the user is within the display area of the head-up display 21 (step S106).

If the position of the merging point seen by the user is within the display area of the head-up display 21 (YES in step S106), the display control unit 16 uses the head-up display 21 to see the virtual merging point object from the user. The display is superimposed on the position of the confluence (step S107), and the process returns to step S102.

On the other hand, when the position of the user is far from the merging point (NO in step S104), and when the position of the merging point seen by the user is outside the display area of the head-up display 21 (NO in step S106), the display is displayed. The control unit 16 confirms whether or not the virtual confluence point object is displayed by the head-up display 21, deletes it if it is displayed (step S108), and returns to step S102.

Regarding the determination in step S104, if the road where the user is located and the road where the confluence is located are grade-separated, but there are stairs or slopes that allow walking between the two roads, the user It is also possible to get off the vehicle 100 and walk to pick up the merging partner. Therefore, the determination in step S104 may be made based on the path length through the stairs or slopes. If YES is determined in step S104 based on such a determination, it is preferable to notify the user to that effect.

Also, even if there are stairs or slopes, if the arrival time at the confluence is later than driving the vehicle 100 along the road, step without considering the stairs or slopes. It is preferable that the determination of S104 is performed. Alternatively, the user may be notified that using the stairs or slope will result in a later arrival time at the confluence than driving the vehicle 100 along the road.

[Hardware configuration example]
6 and 7 are diagrams showing an example of the hardware configuration of the merging support device 10, respectively. Each function of the component of the merging support device 10 shown in FIG. 1 is realized by, for example, the processing circuit 50 shown in FIG. That is, the merging support device 10 stores the merging point between the user and another person, stores a virtual merging point object which is an image showing the position of the merging point, acquires the user's position, and acquires the user's position and the merging point. A processing circuit 50 is provided for displaying a virtual confluence point object at a position corresponding to a confluence point of a real scene seen from a user through a transparent screen based on the position of. The processing circuit 50 may be dedicated hardware, or may be a processor (Central Processing Unit (CPU), processing unit, arithmetic unit, microprocessor, microprocessor, etc.) that executes a program stored in the memory. It may be configured by using a DSP (also called a Digital Signal Processor).

When the processing circuit 50 is dedicated hardware, the processing circuit 50 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable). GateArray), or a combination of these, etc. The functions of the components of the merging support device 10 may be realized by individual processing circuits, or these functions may be collectively realized by one processing circuit.

FIG. 7 shows an example of the hardware configuration of the merging support device 10 when the processing circuit 50 is configured by using the processor 51 that executes the program. In this case, the functions of the components of the merging support device 10 are realized by software (software, firmware, or a combination of software and firmware). The software or the like is described as a program and stored in the memory 52. The processor 51 realizes the functions of each part by reading and executing the program stored in the memory 52. That is, the merging support device 10 has a process of storing the merging point between the user and another person when executed by the processor 51, a process of storing a virtual merging point object which is an image showing the position of the merging point, The result is the process of acquiring the user's position and the process of displaying the virtual confluence object at the position corresponding to the confluence of the actual view seen from the user through the transparent screen based on the user's position and the position of the confluence. A memory 52 for storing a program to be executed is provided. In other words, it can be said that this program causes the computer to execute the procedure and method of operation of the components of the merging support device 10.

Here, the memory 52 is a non-volatile or non-volatile memory such as a RAM (RandomAccessMemory), a ROM (ReadOnlyMemory), a flash memory, an EPROM (ErasableProgrammableReadOnlyMemory), and an EEPROM (ElectricallyErasableProgrammableReadOnlyMemory). Volatile semiconductor memory, HDD (Hard Disk Drive), magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc) and its drive device, etc., or any storage medium used in the future. You may.

The configuration in which the functions of the components of the merging support device 10 are realized by either hardware or software has been described above. However, the present invention is not limited to this, and a configuration in which a part of the components of the merging support device 10 is realized by dedicated hardware and another part of the components is realized by software or the like may be used. For example, for some components, the function is realized by the processing circuit 50 as dedicated hardware, and for some other components, the processing circuit 50 as the processor 51 is stored in the memory 52. It is possible to realize the function by reading and executing it.

As described above, the merging support device 10 can realize each of the above-mentioned functions by hardware, software, or a combination thereof.

[Modification example]
The merging point may be determined by any method. For example, an arbitration system for arranging the location of the merging point between the user of the merging support device 10 and the merging partner may be constructed on the Internet. In that case, the merging point storage unit 12 of the merging support device 10 acquires and stores the determined merging point information from the arbitration system through the communication unit 11.

The image of the virtual confluence point object may be selected from a plurality of images by the user operating the operation input device 23. Further, the image of the virtual merging point object may be an image designated by the merging partner, or the virtual merging point object image may be consulted and determined between the user of the merging support device 10 and the merging partner. For example, an arbitration system for arranging an image of a virtual merging point object between a user of the merging support device 10 and a merging partner may be constructed on the Internet. In that case, the virtual object storage unit 13 of the merging support device 10 acquires and stores an image of the determined virtual merging point object from the arbitration system through the communication unit 11.

When determining the image of the virtual merging point object, when the intention of the merging partner is taken into consideration, both the user of the merging support device 10 (driver of the vehicle 100) and the merging partner can see what kind of image the image of the virtual merging point object is. I can recognize what it is. For example, when the vehicle 100 is a taxi, there may be a situation where the driver of the vehicle 100 and the merging partner have no acquaintance, but both the driver of the vehicle 100 and the merging partner recognize the image of the virtual merging point object. Then, by confirming the images of the virtual merging point objects with each other, it is possible to authenticate each other, which can contribute to the improvement of security.

Further, the display control unit 16 may provide the user with a virtual world including not only a virtual merging point object indicating the position of the merging point but also a virtual object indicating additional information for merging support. For example, as shown in FIG. 8, together with the virtual merging point object 200, a virtual object 201 simulating a user arriving at the merging point (here, a vehicle 100 driven by the user) and a virtual merging partner arriving at the merging point are simulated. A virtual world in which the object 202 resides may be provided to the user. In this case, the display control unit 16 uses the head-up display 21 to display the virtual object 201 simulating the vehicle 100 on the road in front of the virtual merging point object 200 as shown in FIG. 9, and the virtual merging point object A virtual object 202 simulating a merging partner is displayed beside 200.

Further, for example, as shown in FIG. 10, a virtual world in which a virtual object 203 indicating a stop position (parking space) when the vehicle 100 arrives at the merging point exists together with the virtual merging point object 200 may be provided to the user. .. In this case, the display control unit 16 uses the head-up display 21 to display the virtual object 203 indicating the stop position of the vehicle 100 on the road in front of the virtual confluence point object 200 as shown in FIG.

Further, when the vehicle 100 advances to a position where the vehicle 100 passes the confluence point MP as in the virtual world shown in FIG. 12, the position of the confluence point MP as seen by the user is outside the display area of the head-up display 21. At this time, the display control unit 16 may display the virtual object 204 indicating that the merging point MP is close to the vehicle 100 but is not at a position where the virtual merging point object 200 can be displayed, as shown in FIG. 13, for example. In FIG. 13, the virtual object 204 shows an image of the virtual confluence object 200 in an inconspicuous display mode (size, color, brightness, etc.), and an image of an arrow indicating the direction in which the confluence MP is located. Although an example consisting of is shown, the virtual object 204 may be only one of these two images. Further, an animation effect such as blinking the virtual object 204 may be used.

As shown in FIG. 13, the operation of the merging support device 10 when displaying the virtual object 204 indicating that the position of the merging point MP as seen by the user is outside the display area of the head-up display 21 is shown in FIG. Shown in the flowchart. The flowchart of FIG. 14 is obtained by adding step S109 to be executed when NO is determined in step S106 to the flowchart of FIG.

That is, in the flowchart of FIG. 14, when the position of the merging point as seen by the user is outside the display area of the head-up display 21 (NO in step S106), the display control unit 16 displays the virtual object 204 indicating that fact. , Displayed using the head-up display 21 (step S109). Since the other steps are the same as those in FIG. 5, the description here will be omitted.

Further, the communication unit 11 acquires information on the position of the merging partner and the planned passage route from the mobile terminal or navigation device of the merging partner, and the display control unit 16 converts a virtual object indicating the information into a virtual merging point object. It may be displayed together with 200. Specifically, the display control unit 16 displays a virtual object indicating the position of the merging partner at a position corresponding to the position of the merging partner as seen by the user, and also sets the planned passage route of the merging partner as seen by the user. A virtual object indicating the planned passage route of the merging partner may be displayed at the corresponding position.

For example, it is assumed that the point MP in the real world as shown in FIG. 15 is defined as the merging point between the user and the merging partner 101, and the vehicle 100 driven by the user and the merging partner 101 reach the vicinity of the merging point MP. Information on the merging point MP is stored in the merging point storage unit 12 of the merging support device 10, and the communication unit 11 sequentially acquires information on the position of the merging partner 101 and the planned passage route from the mobile terminal of the merging partner 101. To do. In this case, the display control unit 16 uses the head-up display 21 to indicate the virtual merging point object 200 indicating the merging point MP, the virtual object 205 indicating the position of the merging partner 101, and the planned passage route of the merging partner 101. A virtual world as shown in FIG. 16 in which the virtual object 206 exists is provided to the user. As shown in FIG. 17, the user who is the driver of the vehicle 100 sees the virtual merging point object 200, the virtual object 205 indicating the position of the merging partner 101, and the merging partner in the actual view seen through the windshield 30 of the vehicle 100. You can see the virtual object 206 that shows the planned route of 101.

FIG. 17 shows a state in which the merging partner 101 is hidden behind the building and cannot be seen. However, when the merging partner 101 comes out to a position where the user can see it, the virtual object 205 can be seen above the merging partner 101 as shown in FIG. .. Therefore, the user can quickly find the merging partner 101 even in a busy place.

The information of the merging partner acquired by the communication unit 11 and displayed by the display control unit 16 is not limited to the information of the current position and the planned passage route. For example, the communication unit 11 may acquire information on the estimated time of arrival at the merging point of the merging partner, and display a character string such as "5 minutes before arrival" as a virtual object.

<Embodiment 2>
FIG. 19 is a diagram showing a configuration of a merging support device 10 according to a second embodiment of the present invention. The merging support device 10 of the second embodiment is also mounted on the vehicle 100, and the user of the merging support device 10 is the driver of the vehicle 100. However, the merging support device 10 of the second embodiment is connected to the photographing device 27 instead of the head-up display 21 shown in FIG.

The photographing device 27 is a camera that photographs an actual scene around the vehicle 100 (that is, around the user). Here, it is assumed that the photographing device 27 photographs an actual scene in front of the vehicle 100.

The display control unit 16 of the merging support device 10 according to the second embodiment displays an image of the actual scene taken by the photographing device 27 (hereinafter referred to as “actual scene image”) on the screen of the display device 22. Further, the display control unit 16 is based on the position and orientation of the vehicle 100 acquired by the position information acquisition unit 14, the position of the merging point stored in the virtual object storage unit 13, and the map information acquired by the map information acquisition unit 15. Then, the relative position of the merging point as seen from the user (driver of the vehicle 100) is calculated, and the virtual merging point object is displayed at the position corresponding to the merging point in the actual scene image displayed on the screen of the display device 22. .. As a result, the screen of the display device 22 displays an image as if a virtual confluence point object exists at the confluence point. That is, the display control unit 16 provides the user with a virtual world in which the virtual confluence point object exists at the confluence point through the screen of the display device 22.

For example, when the display control unit 16 provides the user with a virtual world similar to that shown in FIG. 3, the display control unit 16 displays an image as shown in FIG. 20 on the screen of the display device 22. The image of FIG. 4 is a superposition of the virtual confluence point object 200 at a position corresponding to the confluence point MP in the actual scene image taken by the photographing device 27.

As described above, the merging support device 10 according to the second embodiment merges by presenting the user who is the driver of the vehicle 100 with a virtual world in which the virtual merging point object appears to exist at the merging point in the real world. Present the location of the point. Therefore, the user can intuitively recognize the location of the confluence in the real world, and it becomes easy to reach the confluence.

Here, the operation of the merging support device 10 will be described with reference to the flowchart of FIG. When the merging support device 10 is activated and the merging point is determined between the user and the merging partner, the merging point storage unit 12 stores the merging point (step S201).

Subsequently, the display control unit 16 displays the actual scene image around the vehicle 100 photographed by the photographing device 27 on the screen of the display device 22 (step S202). Then, the position information acquisition unit 14 acquires the position of the vehicle 100 calculated by the positioning device 24, that is, the position of the user (step S203). At this time, the position information acquisition unit 14 calculates the direction of the vehicle 100 based on the history of the position of the vehicle 100. The map information acquisition unit 15 acquires map information around the user's position from the map information storage device 26 (step S204).

The display control unit 16 is based on the user's position acquired by the position information acquisition unit 14, the position of the confluence stored in the confluence point storage unit 12, and the map information acquired by the map information acquisition unit 15. It is determined whether or not the position of is near the confluence (step S205). The determination in step S205 may be the same as in step S104 in FIG.

If the user's position is near the confluence (YES in step S205), the display control unit 16 sets the display control unit 16 on the actual view as seen by the user based on the position and orientation of the vehicle 100, the position of the confluence, and the map information. The position of the confluence of the above is calculated (step S206).

Then, it is confirmed whether or not the position of the confluence point as seen by the user is within the range of the actual scene image taken by the photographing device 27 (step S207). If the position of the merging point as seen by the user is within the range of the actual scene image (YES in step S207), the display control unit 16 determines the position of the merging point as seen by the user in the actual scene image displayed on the screen of the display device 22. The virtual confluence point object is superimposed and displayed (step S208), and the process returns to step S202.

On the other hand, when the position of the user is far from the merging point (NO in step S205) and when the position of the merging point seen by the user is out of the range of the actual scene image (NO in step S207), the display control unit 16 However, it is confirmed whether or not the virtual confluence point object is displayed on the screen of the display device 22, and if it is displayed, it is deleted (step S209), and the process returns to step S202.

[Modification example]
The various modifications shown in the first embodiment are also applicable to the merging support device 10 of the second embodiment.

In the flowchart of FIG. 21, only the actual scene image is displayed on the screen of the display device 22 first, and then the virtual merging point object is superimposed and displayed on the screen. However, the actual scene image and the virtual merging point object are superimposed and displayed (composite processing). ) After going first, the image may be displayed on the display device 22 as a real scene image in which the virtual confluence point object is synthesized.

When the display device 22 has a superimpose function, the display device 22 displays the actual scene image directly (without passing through the display control unit 16) acquired from the photographing device 27 on the screen, and the display control unit 16 displays the display device 16. The virtual confluence point object may be superimposed on the actual scene image displayed on the screen of 22.

In the second embodiment, an example is shown in which the photographing device 27 captures an actual view in front of the vehicle 100, but the photographing device 27 may capture an actual view on the side, rear, or rear side of the vehicle 100. Good. For example, when the photographing device 27 captures a real scene behind or behind the vehicle 100, the photographing device 27 and the display device 22 function as an electronic mirror that electronically realizes the rear view mirror of the vehicle 100. In other words, the merging support device 10 of the second embodiment is applicable to the electronic mirror system of the vehicle 100.

Further, a plurality of photographing devices 27 having different shooting directions are connected to the merging support device 10, and the display control unit 16 creates a surround image obtained by synthesizing a plurality of actual scene images photographed by the plurality of photographing devices 27. It may be displayed on the screen of the display device 22.

2 of the technique of using the head-up display 21 shown in the first embodiment, the technique of using the actual view image in front of the vehicle 100 shown in the second embodiment, the technique of using the above-mentioned electronic mirror, and the technique of using the surround image. You may combine one or more. As a result, the area for displaying the virtual confluence point object is expanded, so that the probability that the virtual confluence point object is displayed is high. Further, when a plurality of head-up displays 21 or display devices 22 are used, the virtual confluence point object may be displayed only on a part of the plurality of screens. At that time, the display control unit 16 may display a mark notifying the user on which screen the virtual confluence point object is displayed on the screen on which the virtual confluence point object is not displayed.

In the second embodiment, the vehicle 100 may be remotely controlled by a driver (user of the merging support device 10) who is away from the vehicle 100. In that case, the display device 22 is installed near the user at a remote location. When the vehicle 100 is remotely controlled, the position of the vehicle 100 acquired by the position information acquisition unit 14 is not actually the position of the user, but the position information acquisition unit 14 calculates the position of the vehicle 100 by the positioning device 24. Is regarded as the user's position and is acquired. Then, the display control unit 16 displays the virtual merging point object at a position corresponding to the merging point of the image of the actual scene around the vehicle 100.

[Hardware configuration]
The merging support device 10 of the second embodiment can also be realized by the same hardware configuration as in FIGS. 6 and 7. For example, when the merging support device 10 is realized by the processing circuit 50 shown in FIG. 6, the merging support device 10 stores the merging point between the user and another person, and is an image showing the position of the merging point. Store the point object, get the user's position, and based on the user's position and the position of the confluence, place the virtual confluence object at the position corresponding to the confluence of the real-world images around the user displayed on the screen. A processing circuit 50 for displaying the image is provided.

Further, when the merging support device 10 is composed of the processor 51 and the memory 52 shown in FIG. 7, the memory 52 is a process of storing the merging point between the user and another person when executed by the processor 51. The virtual merging point object, which is an image showing the position of the merging point, is stored on the screen, the user's position is acquired, and the virtual merging point object is displayed on the screen based on the user's position and the merging point position. A program that will be executed as a result of the process of displaying at the position corresponding to the confluence of the displayed images of the actual scene around the user is stored.

<Embodiment 3>
FIG. 22 is a diagram showing the configuration of the merging support device 10 according to the third embodiment of the present invention. The merging support device 10 of the third embodiment is mounted on a mobile terminal 300 such as a smartphone. That is, the merging support device 10 is realized by the processor of the mobile terminal 300 executing a merging support application program (hereinafter referred to as "merging support application") stored in the memory. Except for this, the configuration of the merging support device 10 in FIG. 22 is the same as that in FIG.

The user of the merging support device 10 is a user of the mobile terminal 300. Unless otherwise specified, the "user" in the following description is intended as a user of the mobile terminal 300. Here, it is assumed that the user is a pedestrian, but the user can also bring the mobile terminal 300 into the vehicle. Further, the merging partner may be a pedestrian or a vehicle, but here, it is assumed to be a vehicle. Further, the vehicle of the merging partner may be a vehicle without a driver (for example, an unmanned taxi) that travels by remote control or automatic driving.

The photographing device 27 of the mobile terminal 300 photographs the actual scenery around the user, but the photographing direction depends on the direction in which the user holds the mobile terminal 300, and therefore does not necessarily match the traveling direction of the user. Therefore, the display control unit 16 of the merging support device 10 according to the third embodiment needs to recognize the orientation of the mobile terminal 300 in order to recognize the range of the actual scene image captured by the photographing device 27. The orientation recognition method of the mobile terminal 300 may be any method. For example, the display control unit 16 may calculate the orientation of the mobile terminal 300 using the orientation sensor (not shown) of the mobile terminal 300, or the actual scene image and the map information acquisition unit 15 taken by the photographing device 27 may calculate the orientation of the mobile terminal 300. The orientation of the mobile terminal 300 may be determined by collating with the acquired map information.

The display control unit 16 displays an actual scene image taken by the photographing device 27 on the screen of the display device 22 (sometimes referred to as a “screen of the mobile terminal 300”). Further, the display control unit 16 indicates the direction of the recognized mobile terminal 300, the position of the mobile terminal 300 acquired by the position information acquisition unit 14, the position of the confluence stored in the virtual object storage unit 13, and the map information acquisition unit 15. Calculates the relative position of the merging point as seen by the user based on the map information acquired by the user, and displays the virtual merging point object at the position corresponding to the merging point in the actual scene image displayed on the screen of the display device 22. .. As a result, the screen of the display device 22 displays an image as if a virtual confluence point object exists at the confluence point. That is, the display control unit 16 provides the user with a virtual world in which the virtual confluence point object exists at the confluence point through the screen of the display device 22.

For example, it is assumed that the point MP in the real world as shown in FIG. 23 is defined as the merging point between the user 301 and the merging partner, and the user 301 reaches the vicinity of the merging point MP. Information on the position of the merging point MP is stored in the merging point storage unit 12 of the merging support device 10. In this case, the display control unit 16 uses the display device 22 to provide the user with a virtual world as shown in FIG. 24 in which the virtual confluence point object 400 exists at the confluence point MP. On the screen of the mobile terminal 300 of the user 301, as shown in FIG. 25, an image in which the virtual merging point object 400 is superimposed is displayed at a position corresponding to the merging point MP in the actual scene image taken by the photographing device 27.

As described above, the merging support device 10 according to the third embodiment presents the user of the mobile terminal 300 with a virtual world in which the virtual merging point object appears to exist at the merging point in the real world, thereby presenting the location of the merging point. To present. Therefore, the user can intuitively recognize the location of the confluence in the real world, and it becomes easy to reach the confluence.

Here, the operation of the mobile terminal 300 will be described with reference to the flowchart of FIG. 26. First, when the user of the mobile terminal 300 operates the operation input device 23 to activate an application (merging point determination application) for determining the merging point with the merging partner, the merging point determination application activates the communication device 25. Direct or indirect communication with the merging partner is performed, and a merging point is negotiated with the merging partner (step S301). The merging point determined by the merging point determination application is stored in the merging point storage unit 12 (step S302).

After that, the user activates the merging support device 10 by activating the merging support application at an arbitrary timing (for example, when the user approaches the merging point to some extent) (step S303).

When the merging support device 10 is activated, the display control unit 16 displays an actual view image of the user's surroundings taken by the photographing device 27 on the screen of the display device 22 (step S304). Then, the position information acquisition unit 14 acquires the position of the mobile terminal 300 calculated by the positioning device 24, that is, the position of the user (step S305). The map information acquisition unit 15 acquires map information around the user's position from the map information storage device 26 (step S306).

The display control unit 16 is based on the user's position acquired by the position information acquisition unit 14, the position of the confluence stored in the confluence point storage unit 12, and the map information acquired by the map information acquisition unit 15. It is determined whether or not the position of is near the confluence (step S307). The determination in step S307 may be the same as in step S104 in FIG.

If the user's position is near the confluence (YES in step S307), the display control unit 16 determines the actual view as seen by the user based on the position and orientation of the mobile terminal 300, the position of the confluence, and the map information. The position of the upper confluence is calculated (step S308).

Then, the display control unit 16 confirms whether or not the position of the confluence point as seen by the user is within the range of the actual scene image taken by the photographing device 27 (step S309). If the position of the merging point as seen by the user is within the range of the actual scene image (YES in step S309), the display control unit 16 determines the position of the merging point as seen by the user in the actual scene image displayed on the screen of the display device 22. The virtual confluence point object is superimposed and displayed (step S310), and the process returns to step S304.

On the other hand, when the position of the user is far from the merging point (NO in step S307) and when the position of the merging point seen by the user is out of the range of the actual scene image (NO in step S309), the display control unit 16 However, it is confirmed whether or not the virtual confluence point object is displayed on the screen of the display device 22, and if it is displayed, it is deleted (step S311), and the process returns to step S304.

[Modification example]
When the position of the merging point is outside the range of the actual scene image captured by the photographing device 27 (that is, when NO in step S309 of FIG. 26), the modification described with reference to FIGS. 12 to 14 in the first embodiment. Similar to the example, a virtual object indicating that the merging point is outside the display area may be displayed on the screen of the mobile terminal 300.

In the flowchart of FIG. 26, since steps S304 to S311 are repeatedly executed, the screen displayed on the display device 22 is sequentially updated. Therefore, even if the virtual merging point object 400 is displayed on the screen of the display device 22, the merging point is out of the real scene image area as soon as the user changes the direction in which the mobile terminal 300 is held, and the virtual merging point object 400 is displayed. Disappear. Therefore, the user may be able to pause the image displayed on the screen of the display device 22. As a result, the user can easily compare the image of the virtual world displayed on the display device 22 with the landscape of the real world.

Further, when the user rotates while shooting the real scene image with the mobile terminal 300 and acquires a plurality of real scene images having different shooting directions, the display control unit 16 synthesizes the real scene images to generate a panoramic image. , The virtual confluence object may be displayed in the panoramic image.

In the flowchart of FIG. 26, it is assumed that the merging point determination application and the merging support application are separate applications, but the function of the merging point determination application may be incorporated in the merging support application. An example of the operation of the mobile terminal 300 when the merging point determination application has the function of the merging support application will be described with reference to FIG. 27 based on a flowchart.

When the user of the mobile terminal 300 operates the operation input device 23 to activate the merging support application (step S401), the merging support application performs direct or indirect communication with the merging partner via the communication device 25. Arrange a merging point with the merging partner (step S402). The determined merging point is stored in the merging point storage unit 12 (step S403).

After that, the position information acquisition unit 14 acquires the position of the mobile terminal 300 calculated by the positioning device 24, that is, the position of the user (step S404). The map information acquisition unit 15 acquires map information around the user's position from the map information storage device 26 (step S405). The display control unit 16 is based on the user's position acquired by the position information acquisition unit 14, the position of the confluence stored in the confluence point storage unit 12, and the map information acquired by the map information acquisition unit 15. It is determined whether or not the position of is near the confluence (step S406). If the user's position is near the confluence (YES in step S406), the user is notified to that effect by using, for example, the vibration function of the mobile terminal 300 (step S407).

The user who received the notification can operate the operation input device 23 to instruct the merging support device 10 to display the virtual merging point object. If there is no display instruction of the virtual confluence point object for a certain period of time (NO in step S408), the process returns to step S404.

When the merging support device 10 receives a display instruction of the virtual merging point object from the user (YES in step S408), the display control unit 16 displays an actual view image of the user's surroundings taken by the photographing device 27 on the display device 22. Is displayed on the screen of (step S409). Then, the position information acquisition unit 14 acquires the position (user's position) of the mobile terminal 300 calculated by the positioning device 24 (step S410). The map information acquisition unit 15 acquires map information around the user's position from the map information storage device 26 (step S411).

The display control unit 16 determines whether or not the user's position is near the merging point in the same manner as in step S406, and if the user's position is near the merging point (YES in step S412), the mobile terminal 300 The position of the merging point in the actual view as seen by the user is calculated based on the position and direction of the merging point, the position of the merging point, and the map information (step S413).

Then, it is confirmed whether or not the position of the confluence point as seen by the user is within the range of the actual scene image taken by the photographing device 27 (step S414). If the position of the merging point as seen by the user is within the range of the actual scene image (YES in step S414), the display control unit 16 determines the position of the merging point as seen by the user in the actual scene image displayed on the screen of the display device 22. The virtual confluence point object is superimposed and displayed (step S415), and the process returns to step S409.

When the user's position moves away from the merging point (NO in step S412), the display control unit 16 confirms and displays whether or not the virtual merging point object is displayed on the screen of the display device 22. If so, it is deleted (step S416), and the process returns to step S404. When the position of the confluence point seen by the user is outside the range of the actual scene image (NO in step S414), the display control unit 16 performs the same processing as in step S416 (step S417), and steps S409. Return to.

Also in the third embodiment, as in the first embodiment, the user or the merging partner selects the image to be the virtual merging point object, or the arbitration system for arranging the image of the virtual merging point object is used. It may be possible to change it.

Further, the display control unit 16 may provide the user with a virtual world including not only a virtual merging point object indicating the position of the merging point but also a virtual object indicating additional information for merging support. For example, as shown in FIG. 28, along with the virtual merging point object 400, there are a virtual object 401 simulating a user 301 arriving at the merging point and a virtual object 402 simulating a merging partner (here, a vehicle) arriving at the merging point. A virtual world to be provided may be provided to the user. In this case, as shown in FIG. 29, the display control unit 16 displays the virtual object 401 simulating the user 301 beside the virtual merging point object 400 on the screen of the mobile terminal 300, and in front of the virtual merging point object 400. A virtual object 402 simulating a merging partner is displayed on the road.

Further, the communication unit 11 acquires information on the position of the merging partner and the planned passage route from the mobile terminal or navigation device of the merging partner, and the display control unit 16 converts a virtual object indicating the information into a virtual merging point object. It may be displayed together with 400. Specifically, the display control unit 16 displays a virtual object indicating the position of the merging partner at a position corresponding to the position of the merging partner as seen by the user, and also sets the planned passage route of the merging partner as seen by the user. A virtual object indicating the planned passage route of the merging partner may be displayed at the corresponding position.

For example, it is assumed that the point MP in the real world as shown in FIG. 30 is defined as the merging point between the user 301 and the merging partner vehicle 302, and the user 301 and the merging partner vehicle 302 reach the vicinity of the merging point MP. .. Information on the merging point MP is stored in the merging point storage unit 12 of the merging support device 10, and the communication unit 11 acquires information on the position of the vehicle 302 and the planned passage route from the navigation device of the merging partner vehicle 302. .. In this case, the display control unit 16 uses the screen of the mobile terminal 300 to display the virtual merging point object 400 indicating the merging point MP, the virtual object 403 indicating the position of the merging partner vehicle 302, and the merging partner vehicle 302. A virtual world as shown in FIG. 30 in which a virtual object 404 indicating a planned passage route exists is provided to the user 301. On the screen of the mobile terminal 300 of the user 301, as shown in FIG. 31, a virtual merging point object 400, a virtual object 403 indicating the position of the merging partner vehicle 302, and a planned passage route of the vehicle 302 are displayed in the actual scene image. An image in which the indicated virtual object 404 is superimposed on the actual scene image is displayed.

The mobile terminal 300 is configured to be able to switch between the display of a real scene image including the virtual confluence point object as shown in FIG. 25 (display of the virtual world) and the display of a map showing the positions of the user and the confluence point as shown in FIG. 32. It may have been done. In that case, the map display area and the actual scene image display area on the screen of the mobile terminal 300 may be changed according to the posture of the mobile terminal 300.

For example, when the mobile terminal 300 is laid horizontally (this state is defined as an elevation angle of 0 degrees), a map as shown in FIG. 32 is displayed, and when the user holds the mobile terminal 300 vertically to capture a real scene (this state is defined as an elevation angle of 0 degrees) This state is defined as an elevation angle of 90 degrees), and a real scene image as shown in FIG. 25 may be displayed. Alternatively, when the elevation angle of the mobile terminal 300 is 90 degrees, both the actual scene image and the map may be displayed as shown in FIG. 33. Further, when the user poses for displaying the actual scene image, both the actual scene image and the map may be displayed as shown in FIG. 33 regardless of the elevation angle of the mobile terminal 300.

Further, when the mobile terminal 300 is oriented horizontally, the actual scene image as shown in FIG. 25 is displayed, and when the mobile terminal 300 is oriented vertically, both the actual scene image and the map are displayed as shown in FIG. 34. May be good. Further, in a state where both the actual scene image and the map are displayed on the screen of the mobile terminal 300, the user can swipe the screen of the mobile terminal 300 to expand the display area of the actual scene image and the display area of the map (area). The ratio) may be changed.

<Embodiment 4>
In the fourth embodiment, the vehicle 100 having the merging support device 10 according to the first embodiment (or the second embodiment) and the user of the mobile terminal 300 having the merging support device 10 according to the third embodiment are merged. An example of a merging system to support is shown.

FIG. 35 is a diagram showing the configuration of the merging support system according to the fourth embodiment. The merging support system includes a vehicle 100 and a mobile terminal 300 each equipped with a merging support device 10, and a merging support server 500. The vehicle 100 and the mobile terminal 300 can communicate with the merging support server 500 through a communication network such as the Internet. Although FIG. 35 shows one vehicle 100 and one mobile terminal 300, there are a plurality of vehicles 100 and mobile terminals 300.

The merging support server 500 provides a pick-up service using the vehicle 100 to the user of the mobile terminal 300, and a plurality of vehicles 100 that can be dispatched are registered in the merging support server 500 in advance. Hereinafter, the user of the mobile terminal 300 who intends to receive this service is referred to as a "service user".

Hereinafter, the operation of the merging support server 500 will be described with reference to the flowchart of FIG. 36. The merging support server 500 receives a vehicle allocation request from the service user's mobile terminal 300 (step S501). The vehicle dispatch request from the mobile terminal 300 includes at least information on the place and time when the service user wishes to dispatch the vehicle, but also includes information such as the number of passengers and the vehicle type desired to be dispatched. Good. The merging support server 500 that has received the vehicle allocation request consults the vehicle allocation to the one that is close to the service user's request from among the plurality of registered vehicles 100 (step S502).

After that, the merging support server 500 receives a response from any of the vehicles 100 that consulted the vehicle allocation to the effect that the vehicle can be dispatched (step S503). Then, the information of the vehicle 100 that can be dispatched is transmitted to the mobile terminal 300 of the service user (step S504). When there are a plurality of vehicles 100 that can be dispatched, information on the plurality of vehicles 100 is transmitted to the service user's mobile terminal 300, and the service user selects one of them.

The merging support server 500 receives the information of the vehicle 100 selected by the service user and the profile of the service user from the mobile terminal 300 of the service user (step S505). As a result, a transfer contract is established between the service user and the selected vehicle 100.

When the contract is established, the merging support server 500 notifies the vehicle 100 selected by the service user of the conclusion of the contract and transmits an image of the virtual merging point object (step S506). Further, the merging support server 500 also notifies the mobile terminal 300 of the service user of the establishment of the contract and transmits an image of the virtual merging point object (step S507).

The operation of the merge support server 500 is now complete. After that, the merging support device 10 of the vehicle 100 performs the operation described in the first embodiment, and the merging support device 10 of the mobile terminal 300 performs the operation described in the third embodiment, whereby the vehicle 100 and the service user perform the operation described in the third embodiment. Support the merger with.

It is preferable that the same image as the image of the virtual confluence point object is transmitted to the vehicle 100 and the mobile terminal 300 for which the contract has been concluded. Alternatively, the merging support server 500 may operate as an arbitration system for arranging an image as a virtual merging point object between the service user and the driver of the vehicle 100 to be dispatched. When the same virtual confluence point object is used in the vehicle 100 and the mobile terminal 300, for example, when the service user merges with the vehicle 100, the mobile terminal 300 on which the virtual confluence point object is displayed is used by the driver of the vehicle 100. You can check the images of the virtual confluence points objects with each other by presenting them to each other. As a result, mutual authentication can be performed, which can contribute to the improvement of security.

[Modification example]
In the first and third embodiments, it is determined whether or not the vehicle 100 has approached the merging point (for example, the determination of step S104 in FIG. 5), or whether or not the user of the mobile terminal 300 has approached the merging point (for example,). The determination in step S307 of FIG. 26) was performed by the merging support device 10 included in the vehicle 100 or the mobile terminal 300. However, the merging support server 500 sequentially acquires the positions of the vehicle 100 and the mobile terminal 300, makes a determination thereof, and notifies the vehicle 100 or the mobile terminal 300 when the vehicle 100 or the mobile terminal 300 approaches the merging point. You may.

<Embodiment 5>
In the fifth embodiment, an example of a transfer system for picking up and dropping an unspecified number of passengers having a mobile terminal 300 equipped with the merge support device 10 by using the vehicle 100 provided with the merge support device 10 will be shown.

In the transfer system of the fifth embodiment, one or a plurality of vehicles 100 equipped with the merging support device 10 travel on a specific patrol route like a patrol bus, and are carried at a plurality of boarding / alighting points in the middle of the patrol route. It provides a service for getting on and off an unspecified passenger having a terminal 300. Hereinafter, the vehicle 100 that provides this service is referred to as a "patrol vehicle", and the user of the mobile terminal 300 who intends to receive the service is referred to as a "service user".

The configuration of the transfer system of the fifth embodiment may be the same as that of FIG. 35. Information on the patrol route and each boarding / alighting point is distributed from the merging support server 500 to the patrol vehicle 100 and the mobile terminal 300 of the service user. A plurality of boarding / alighting points are stored as merging points in the merging point storage unit 12 of the merging support device 10 of the patrol vehicle 100. Similarly, a plurality of boarding / alighting points are stored as merging points in the merging point storage unit 12 of the merging support device 10 of the service user's mobile terminal 300. Therefore, the merging support device 10 of the patrolling vehicle 100 and the mobile terminal 300 provides the driver and the service user of the patrolling vehicle 100 with a virtual world in which the virtual merging point object exists at a position corresponding to the boarding / alighting point. Hereinafter, the virtual confluence point object indicating the position of the boarding / alighting point is referred to as a “virtual boarding / alighting point object”.

The driver and service user of the patrol vehicle 100 can recognize the position of the boarding / alighting point by confirming the position of the virtual boarding / alighting point object in the virtual world provided by the merging support device 10. Therefore, in the transfer system, it is not necessary to actually install a signpost on the patrol route. Therefore, in the transfer system, it is possible to flexibly change the location of the boarding / alighting point and the patrol route.

For example, the merging support server 500 collects traffic information, weather information, distribution information of service users, etc., and for example, the patrol route is not changed so as to avoid a congested section, and the boarding / alighting point is not exposed to wind and rain in bad weather. By changing to a location or adding an area with many service users to the patrol route, it is possible to provide a highly convenient transfer service. Further, the merging support server 500 may change the image of the virtual boarding / alighting point object as needed.

Further, the merging support server 500 may receive a request from the service user for the place to be the boarding / alighting point, and determine the boarding / alighting point based on the request. For example, it is assumed that a place where a plurality of service users can reach within 10 minutes is set as a boarding / alighting point. Further, the merging support server 500 may temporarily increase the number of boarding / alighting points in response to a request from a service user who has difficulty walking, such as an elderly person or an injured person.

The change information of the image of the patrol route, the boarding / alighting point, and the virtual boarding / alighting point object is distributed from the merging support server 500 to the patrol vehicle 100 and the mobile terminal 300.

In the present invention, each embodiment can be freely combined, and each embodiment can be appropriately modified or omitted within the scope of the invention.

Although the present invention has been described in detail, the above description is exemplary in all embodiments and the present invention is not limited thereto. It is understood that innumerable variations not illustrated can be assumed without departing from the scope of the present invention.

10 merging support device, 11 communication unit, 12 merging point storage unit, 13 virtual object storage unit, 14 position information acquisition unit, 15 map information acquisition unit, 16 display control unit, 21 head-up display, 22 display device, 23 operation input Devices, 24 positioning devices, 25 communication devices, 26 map information storage devices, 27 imaging devices, 30 windshields, MP merging points, 100 user vehicles, 101 merging partners, 200,400 virtual merging point objects, 201-206,401 ~ 404 virtual object, 300 mobile terminal, 301 mobile terminal user, 302 merging partner's vehicle, 500 merging support server.

Claims

A confluence storage unit that stores the confluence between the user and others,
A virtual object storage unit that stores a virtual confluence point object that is an image showing the position of the confluence point, and
A location information acquisition unit that acquires the user's position, and
Based on the position of the user and the position of the confluence point, the virtual confluence point object is viewed from the user through a transparent screen, or the confluence point in an image of the actual scene around the user displayed on the screen. Display control unit that displays at the position corresponding to
Confluence support device equipped with.
The user is the driver of the vehicle
The display control unit sees the virtual merging point object through a transparent screen from the driver, or the merging in an image of a real scene around the vehicle displayed on the screen of a display device mounted on the vehicle. Display at the position corresponding to the point,
The merging support device according to claim 1.
The merging support device is mounted on a mobile terminal and is mounted on a mobile terminal.
The display control unit displays the virtual merging point object at a position corresponding to the merging point in an image of a real scene around the user displayed on the screen of the mobile terminal.
The merging support device according to claim 1.
The other person is a driver-less vehicle that travels by remote control or automatic driving.
The merging support device according to claim 3.
The merging point storage unit acquires information on the merging point from an arbitration system that decides the location of the merging point between the user and the other person.
The merging support device according to claim 1.
The virtual object storage unit acquires an image to be the virtual confluence point object from an arbitration system for arranging an image to be the virtual confluence point object between the user and the other person.
The merging support device according to claim 1.
When the merging support device is used by both the user and the other person, the virtual merging point object common to the merging support device of the user and the merging support device of the other person is used.
The merging support device according to claim 1.
The virtual confluence object is an image simulating a signpost installed at a public transportation stop.
The merging support device according to claim 1.
The display control unit displays an image simulating the user or the other person who has arrived at the merging point together with the virtual merging point object.
The merging support device according to claim 1.
The display control unit displays an image showing a stop position when the vehicle arrives at the merging point together with the virtual merging point object.
The merging support device according to claim 2.
Further equipped with a communication unit for acquiring information on the position of the other person by communication,
The display control unit further displays an image indicating the position of the other person at a position corresponding to the position of the other person in the actual scene or the image of the actual scene.
The merging support device according to claim 1.
Further equipped with a communication unit that acquires information on the route to be passed until the other person arrives at the confluence by communication.
The display control unit further displays an image showing the planned passage route of the other person at a position corresponding to the position of the planned passage route of the other person in the actual scene or the image of the actual scene.
The merging support device according to claim 1.
When the position of the merging point is outside the range of the actual view seen from the user through the transparent screen or the range of the image of the actual view around the user, an image indicating the direction in which the merging point exists is displayed. To do,
The merging support device according to claim 1.
The vehicle is a vehicle that travels on a patrol route around a plurality of predetermined confluence points and allows an unspecified other person to get on and off at the confluence point.
The merging support device according to claim 2.
The patrol route and the plurality of confluence points are changed based on any one or more of distribution information, traffic information, and weather information of the other person who uses the vehicle.
The merging support device according to claim 14.
The mobile terminal can display a map showing the position of the user and the position of the confluence, and can display an image of the actual scene on which the virtual confluence object is superimposed, according to the posture of the mobile terminal. , The area for displaying the map and the area for displaying the image of the actual scene on the screen are changed.
The merging support device according to claim 3.
The vehicle is a vehicle that is remotely controlled by the driver at a location distant from the vehicle.
The position information acquisition unit acquires the position of the vehicle as the position of the user, and obtains the position of the user.
The display control unit displays the virtual merging point object at a position corresponding to the merging point in the image of the actual scene around the vehicle displayed on the screen.
The merging support device according to claim 2.
It is possible to pose the image of the real scene including the virtual confluence object.
The merging support device according to claim 1.
The image of the actual scene is a surround image obtained by synthesizing a plurality of images having different shooting directions.
The merging support device according to claim 1.
The merging point storage unit of the merging support device memorizes the merging point between the user and others,
The virtual object storage unit of the merging support device stores a virtual merging point object which is an image showing the position of the merging point.
The position information acquisition unit of the merging support device acquires the position of the user,
Based on the position of the user and the position of the merging point, the display control unit of the merging support device can see the virtual merging point object from the user through a transparent screen, or the user displayed on the screen. Displayed at the position corresponding to the confluence in the image of the actual scene around the
Joining support method.