WO2019207944A1

WO2019207944A1 - Information processing device, program and information processing method

Info

Publication number: WO2019207944A1
Application number: PCT/JP2019/007190
Authority: WO
Inventors: 忍及川
Original assignee: 株式会社デンソー
Priority date: 2018-04-25
Filing date: 2019-02-26
Publication date: 2019-10-31
Also published as: JP2019191914A

Abstract

This information processing device is provided with: a location acquisition unit (100) which acquires location information (302) on a vehicle that is called and coming to a meeting place and location information (300) on a user; a vehicle spot calculation unit (110) which calculates a spot of the vehicle, which is to be displayed in user image data (310) that is captured by the user camera 16, by means of the vehicle location information and the user location information acquired by the location acquisition unit; and an image presentation unit (120) which presents, on the user image data, the spot of the vehicle, which is calculated by the vehicle spot calculation unit.

Description

Information processing apparatus, program, and information processing method

Cross-reference of related applications

This application is based on Japanese Patent Application No. 2018-83762 filed on April 25, 2018, the contents of which are incorporated herein by reference.

The present disclosure relates to an information processing device, a program, and an information processing method, and particularly relates to an information processing device, a program, and an information processing method that are used in a service for attracting vehicles.

Conventionally, a service in which a user calls a vehicle such as a taxi through a mobile terminal has been widespread.

In such a service, Patent Document 1 discloses a navigation device in which a user can easily check a vehicle coming to a meeting place. The navigation device generates voice guidance information indicating from which direction the vehicle enters the meeting place based on the position information and route information of the set meeting place. This device transmits the generated guidance information to the user's terminal. The user can easily confirm the vehicle by listening to the voice guidance information.

However, in the apparatus of Patent Document 1, when the meeting place is crowded with many taxi vehicles, it may be difficult for the user to specify which vehicle the called vehicle is.

JP 2006-25874 A

This disclosure is intended to provide an information processing apparatus that makes it easy for a user to recognize a called vehicle and solves the above-described problems.

In an aspect of the present disclosure, the information processing apparatus includes: a position acquisition unit that acquires position information of a vehicle that is calling to a meeting place and position information of a user; the position information of the vehicle that is acquired by the position acquisition unit; A vehicle location calculation unit that calculates a location of the vehicle displayed in user image data captured by the user's camera according to the user's location information, and the location of the vehicle calculated by the vehicle location calculation unit, An image presentation unit for presenting on the user image data.

In another aspect of the present disclosure, the information processing apparatus includes: a position acquisition unit that acquires position information of a vehicle that is calling to a meeting place and a user's position information; the vehicle acquired by the position acquisition unit; According to the position information, the user location calculation unit that calculates the location of the user displayed in the vehicle external image data acquired from the external camera of the vehicle, and the location of the user calculated by the user location calculation unit, An image presenting unit for presenting on the vehicle external image data.

In another aspect of the present disclosure, an information processing program executed by the information processing apparatus causes the position acquisition unit to acquire the position information of the vehicle and the position information of the user that are calling to the meeting place, and causes the vehicle location calculation unit to Based on the position information of the vehicle acquired by the position acquisition unit and the position information of the user, the location of the vehicle displayed in the user image data captured by the user's camera is calculated, and the image presentation unit The location of the vehicle calculated by the vehicle location calculation unit is presented on the user image data.

In another aspect of the present disclosure, an information processing method executed by an information processing apparatus acquires position information of a vehicle and a user who are calling to a meeting place, and acquires the acquired position information of the vehicle and the user The position of the vehicle displayed in the user image data captured by the user's camera is calculated based on the position information, and the calculated position of the vehicle is presented on the user image data.

According to the information processing apparatus, the information processing program, and the information processing method, first, the position information of the vehicle and the position information of the user calling to the meeting place are acquired. Then, the location of the vehicle displayed in the image data of the user's camera is calculated based on the position information. Then, the calculated location of the vehicle is presented on the image data. Thereby, it is possible to provide an information processing apparatus that allows the user to easily specify the called vehicle.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. The drawing
It is a system configuration diagram of an information processing system according to the first embodiment of the present disclosure, It is a block diagram which shows the control structure of the user terminal shown in FIG. 1, a motor vehicle terminal, and a server, It is a block diagram showing a functional configuration of the information processing system according to the first embodiment of the present disclosure, It is a flowchart of a vehicle presentation process according to the first embodiment of the present disclosure, 5 is a screen example of the vehicle presentation process shown in FIG. It is a block diagram which shows the function structure of the information processing system which concerns on 2nd embodiment of this indication, It is a flowchart of the vehicle user presentation processing according to the second embodiment of the present disclosure, FIG. 8 is a screen example of the vehicle user presentation process shown in FIG. It is a block diagram which shows the function structure of the information processing system which concerns on 3rd embodiment of this indication, It is a block diagram showing a functional configuration of an information processing system according to a fourth embodiment of the present disclosure, It is a block diagram showing a functional configuration of an information processing system according to a fifth embodiment of the present disclosure, It is an example of a screen of an information processing system concerning a 5th embodiment of this indication.

<First embodiment>
[System configuration of information processing system X]
First, the system configuration of the information processing system X according to the first embodiment of the present disclosure will be described with reference to FIG.

The information processing system X of the present embodiment is a system that enables confirmation when a user calls a car such as a taxi or a hire.

1A, the information processing system X includes a user terminal 1, an automobile terminal 2, and a server 3, which are connected by a network 4.

The user terminal 1 can use services on the web provided by the server 3, and includes a smartphone with a camera (Smart Phone), a mobile phone, a PDA (Personal Data Assistant), a PC (Personal Computer), a personal navigation device, It is a device such as a home appliance. In the present embodiment, an example in which the user terminal 1 is a smartphone will be described. This smartphone functions as the information processing apparatus of this embodiment by installing and executing application software (hereinafter referred to simply as “application”), which will be described later.

The car terminal 2 is an in-car terminal of a car that uses services on the web provided by the server 3. The in-vehicle terminal is a smartphone, a mobile phone, a PDA, a car navigation device, or the like.

In the present embodiment, an example in which the automobile terminal 2 is also a smartphone will be described as shown in FIG. In the case of this example, the smartphone is placed on the holder H on the dashboard D of the automobile (vehicle) and used for car navigation by the driver (operator) of the automobile. With this arrangement, the outside and inside of the vehicle can be imaged with the out camera and the in camera of the smartphone, respectively. In addition, the holder H is driven like an electric head by control from the automobile terminal 2 and can change the direction of imaging by the camera of the smartphone.

The server 3 is a server that performs management in a vehicle call service. Specifically, the server 3 receives a request for calling a car from the user terminal 1, searches for the car terminal 2 of the car that can be called, and mediates both terminals. For example, the server 3 estimates user characteristics from information such as the purchase history, viewing history, and travel history of the user, and performs matching of charges, drivers, vehicles, and the like based on the characteristics.

At this time, the server 3 introduces the driver of the vehicle registered in the service, the vehicle type of the vehicle, image data obtained by imaging the appearance of the vehicle, streaming data, description language data, audio data, etc. (hereinafter simply referred to as “image data etc.”). Can be distributed to the user. The image data or the like may be a still image or a moving image. Furthermore, the server 3 can also display word-of-mouth, rating, etc. for each vehicle.

In addition, the server 3 may set an application to be installed in the user terminal 1, obtain various information, manage the operation of the application, and distribute the application itself.

The network 4 is a mobile phone network, a WAN (Wide Area Network) such as the Internet (registered trademark), or an IP network such as a LAN (Local Area Network) such as WiFi (registered trademark) or a wireless LAN. These networks and the like may be, for example, always connectable (connected) from the user terminal 1 and the automobile terminal 2, and in the case of a mobile phone network, 4G (4th generation), 5G (5th generation), etc. It may be a standard low-latency and high-speed communication network.

In the second to fifth embodiments described later, the system configuration may be the same.

Next, the device configuration of the user terminal 1, the car terminal 2, and the server 3 will be described with reference to FIG. 2A shows the configuration of the user terminal 1, FIG. 2B shows the configuration of the automobile terminal 2, and FIG. 2C shows the configuration of the server 3.

The user terminal 1 includes a control unit 10, a storage unit 11, an input unit 12, a display unit 13, a connection unit 14, a sensor group 15, a camera 16, a voice input / output unit 18, and the like. Each unit is connected to the control unit 10 and controlled in operation by the control unit 10.

The automobile terminal 2 includes a control unit 20, a storage unit 21, an input unit 22, a display unit 23, a connection unit 24, an external camera 26, an in-vehicle camera 27, a voice input / output unit 28, and the like. Each unit is connected to the control unit 20 and controlled in operation by the control unit 20.

The server 3 includes a control unit 30, a storage unit 31, a connection unit 34, and the like. Each unit is connected to the control unit 30 and controlled in operation by the control unit 30.

The

control units

10, 20, and 30 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU, a TPU (Tensor Processing Unit), a DFP (Data Flow Processor), a DSP (Digital Signal Processor), It is an information processing unit including other ASIC (Application Specific Processor).

Each

control unit

10, 20, 30 reads a program stored in the auxiliary storage unit of each

corresponding storage unit

11, 21, 31 and develops this program in the main storage unit and executes the program, which will be described later. It is made to operate as each functional block (functional unit). The

control units

10, 20, and 30 control the entire devices of the user terminal 1, the automobile terminal 2, and the server 3.

Storage units

11, 21, and 31 are a main storage unit such as a RAM (Random Access Memory), an auxiliary storage unit such as a ROM (Read Only Memory), an SSD (Solid State Disk), an HDD (Hard Disk Drive), and a flash memory card. Or non-transitory tangible storage medium that may include optical recording media and the like.

Programs for controlling the operation of the user terminal 1, the automobile terminal 2, and the server 3 are stored in the auxiliary storage units of the

storage units

11, 21, and 31, respectively. This program is an OS (Operating System), an application, and the like. In addition, the

storage units

11, 21, and 31 also store various data.

The

input units

12 and 22 are a touch panel, a pointing device such as a mouse, a button, a keyboard, an acceleration sensor, a line-of-sight sensor, a biometric authentication sensor, and the like. The input unit 12 acquires input to the web page form, various instructions, and the like by the user.

Display units

13 and 23 are an LCD (Liquid Crystal Display), an organic EL display, an LED (Light Emitting Diode), or the like.

The

connection units

14, 24, and 34 are connection means including a wireless transceiver and a LAN interface for connecting to the network 4. The

connection units

14, 24, and 34 may include USB (Universal Serial Bus), RS-232C, various flash memory cards, SIM cards, interfaces to Bluetooth (registered trademark), and the like.

Sensor groups

15 and 25 are GNSS (Global Navigation Satellite System) position information sensors, gyro sensors, magnetic sensors, altitude sensors, and the like. In the present embodiment, the sensor group 15 detects the position (coordinates), orientation, speed, acceleration, etc. (hereinafter referred to as “position etc.”) of the map of the device itself.

The camera 16 is provided in the user terminal 1 to take an image of the user and the user's surroundings. The camera 16 is a camera in which an imaging element such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor and an optical element such as a lens are combined. The camera 16 captures a still image or a moving image mainly directed by the user and stores the image data in the storage unit 11.

The voice input /

output units

18 and 28 are a microphone and an A / D (Analog-to-Digital) converter for voice input, a D / A (Digital-to-Analog) converter for voice output, an amplifier, a speaker, and a vibration motor. Etc. With these configurations, the voice input /

output units

18 and 28 perform notification such as a voice call of a user or driver, input of a voice command, information notification at the time of calling, navigation, and the like.

The external camera 26 is a camera for external imaging of the vehicle. In this embodiment, the smartphone's out camera is used for external imaging. Image data captured by this out-camera is stored in the storage unit 21.

The in-vehicle camera 27 is a camera for in-vehicle imaging of the vehicle. In this embodiment, the in-camera of a smart phone is used for in-vehicle imaging. Image data captured by the in-camera is also stored in the storage unit 21.

In the camera 16, the external camera 26, and the in-vehicle camera 27, whether a still image or a moving image is captured, the image resolution, the zoom magnification, and the like can be changed via an application installed in the user terminal 1. These changes may also be possible in the automobile terminal 2.
[Functional configuration of information processing system Xa]
Here, the functional configuration of the information processing system Xa according to the first embodiment of the present disclosure will be described with reference to FIG. Also in the following embodiments, the information processing system X, the user terminal 1, the server 3, and the like having different functional unit configurations are indicated by adding lowercase letters a, b, c,.

The control unit 10a of the user terminal 1a includes a position acquisition unit 100, a vehicle location calculation unit 110, and an image presentation unit 120.

The storage unit 11a of the user terminal 1a stores position information 300 and user image data 310.

The automobile terminal 2 a includes location information 302.

The server 3 stores map data 340.

The position acquisition unit 100 acquires the position information 302 and the user position information 300 of the vehicle calling to the meeting place. In the present embodiment, the meeting place is a position designated by the user through the application and contacted with the driver of the automobile via the server 3. This meeting place can be indicated on the map data 340. The position information 302 is acquired from the automobile terminal 2 via the server 3. The user position information 300 is acquired by the sensor group 15 of the own device.

The vehicle location calculation unit 110 is displayed in the user image data 310 in which the user's periphery is imaged by the user's camera 16 based on the vehicle location information 302 and the user location information 300 acquired by the location acquisition unit 100. Calculate the location of the vehicle. In the present embodiment, for example, the vehicle location calculation unit 110 calculates the position of the called vehicle on the map data 340 from the position information 302, and the position of the user position information 300 on the user's map data 340; The imaging direction of the camera 16 is calculated. And the vehicle location calculation part 110 collates with the map data 340 by DNN (Deep | neural | network) etc. by performing perspective projection conversion, affine conversion, etc. for the user image data 310. FIG. Thereby, the vehicle location calculation part 110 can grasp | ascertain the coordinate of the building in the user image data 310, or a road. Further, the vehicle location calculation unit 110 recognizes the image of the automobile on the road by image recognition using DNN or the like, and searches for a match with the coordinates such as the position of the called vehicle. If there is a match, the vehicle location calculation unit 110 calculates the coordinates as the location of the vehicle.

The image presentation unit 120 presents the vehicle location calculated by the vehicle location calculation unit 110 on the user image data 310. In the present embodiment, for example, AR (Augmented） Reality) can be used to add an annotation to a captured image with a circle or the like so that the vehicle image is emphasized.

At this time, the image presentation unit 120 can also show the building or road name recognized by the image recognition in the user image data 310. Further, the image presentation unit 120 can acquire the map data 340 from the server 3, arrange the vehicle position information 302 and the user position information 300 on the map, and show them on the display unit 13.

In addition, the image presentation unit 120 can also show the user the direction of arrival of the called vehicle.

The position information 300 is information such as the position of the own apparatus (user position) acquired by the sensor group 15 of the user terminal 1a that is the own apparatus. The position information 300 may be away from the meeting place.

The position information 302 is information such as the position of the automobile terminal 2a (the position of the vehicle) acquired by the sensor group 25 of the automobile terminal 2a. In the present embodiment, for example, the position information 302 is acquired from the vehicle terminal 2 a of the vehicle being called by the position acquisition unit 100 of the user terminal 1 via the server 3.

User image data 310 is image data of a still image or a moving image captured by the camera 16. In the present embodiment, an AR display scenery including the surrounding roads and buildings around the user, that is, a car that is called to come, is imaged by the user.

Here, the control unit 10a of the user terminal 1 executes a program such as an application on an OS (Operating System) stored in the storage unit 11a, so that the position acquisition unit 100, the vehicle location calculation unit 110, and the image presentation are performed. It functions as the unit 120 or the like. At this time, some or all of the functions executed by the

control units

10, 20, and 30 may be configured in hardware by one or a plurality of ICs, DSPs, programmable logic circuits, and the like.

Moreover, each part of the above-mentioned user terminal 1 becomes a hardware resource which performs the method of this indication.
[Vehicle presentation processing by information processing system Xa]
Next, vehicle presentation by the information processing system Xa according to the first embodiment of the present disclosure will be described with reference to FIGS.

In the vehicle presentation process according to the present embodiment, the position information 302 of the vehicle calling to the meeting place and the position information 300 of the user are acquired. And the location of the vehicle displayed in the user image data 310 acquired from the user's camera is calculated from the acquired position information 302 and position information 300. Thereafter, the calculated location of the vehicle is presented on the user image data 310 so that the user can easily recognize the called vehicle.

In the vehicle presentation process of the present embodiment, the control unit 10a of the user terminal 1 mainly executes a program such as an application stored in the storage unit using hardware resources in cooperation with each unit.

Below, with reference to the flowchart of FIG. 4, the detail of the vehicle presentation process of this embodiment is demonstrated for every step.
(Step S101)
First, the position acquisition unit 100 performs a position acquisition process.

Here, the user activates the application on the user terminal 1, connects to the server 3, and requests the vehicle to be called. The server 3 sets a vehicle to be called, notifies the vehicle terminal 2 of the vehicle, and enables communication with the user terminal 1.

In this state, the position acquisition unit 100 acquires the position information 302 from the automobile terminal 2a of the vehicle calling to the meeting place and stores it in the storage unit 11a.

The position acquisition unit 100 also acquires the position information 300 of the user terminal 1a from the sensor group 15 and stores it in the storage unit 11a.
(Step S102)
Next, the vehicle location calculation unit 110 performs a vehicle location calculation process.

It is considered that the user is interested in where the vehicle is running, particularly when calling and waiting for the vehicle, so that the user image data 310 at various locations is captured and displayed as an AR.

Therefore, when the vehicle arrives at a distance close to the user terminal 1a and can be captured as the user image data 310, the image presentation unit 120 may notify the user to that effect. This notification is performed by displaying a pop-up or the like on the screen of the application.

The vehicle location calculation unit 110 calculates, based on the vehicle location information 302 and the user location information 300, the direction in which the vehicle comes when the called vehicle comes close, and directs the camera 16 of the user terminal to the user. To instruct. This instruction may be performed on the screen of the application.

The user points the camera 16 in the instructed direction so as to collide with a landmark known to the user.

When the camera 16 is directed in the direction in which the vehicle comes, the vehicle location calculation unit 110 may indicate to the user by voice or vibration by the voice input / output unit 18.

When the user captures the user image data 310 with the camera 16, the vehicle location calculation unit 110 recognizes the vehicle displayed therein and calculates the location where the called vehicle is captured.
(Step S103)
Next, the image presentation unit 120 performs image presentation processing.

The image presentation unit 120 presents the calculated vehicle location on the user image data 310 by AR.

Referring to the screen example 500 of the application shown in FIG. 5, the image presentation unit 120 first displays a message such as “User, driver has arrived” or the like in the display column 610 when the called vehicle is captured. Then, the image presentation unit 120 displays the user image data 310 on the screen of the application, and displays the vehicle location 700 that is the calculated location of the vehicle as an AR.

The image presentation unit 120 also displays the map data 340 acquired from the server 3, displays the position of the user with the pointer U and displays the position of the vehicle with the pointer P thereon. On this map data 340, the route of the vehicle to the meeting place, the arrow of the traveling direction of the vehicle, etc. may be shown. This makes it easier to recognize the vehicle that has been called to the user.

This completes the vehicle presentation process according to the first embodiment of the present disclosure.

By configuring as above, the following effects can be obtained.

Conventionally, a service for attracting vehicles has been provided. However, when the meeting place of the vehicle is congested or the arrival point is shifted, the user may not be able to recognize the called vehicle. In addition, since the user does not know the appearance of the called vehicle, for example, even if information such as the vehicle type is given from the server, there is a possibility that the user does not know.

In such a case, the vehicle cannot be specified only by the above-mentioned service, and the driver of the vehicle who had been bothered to call must be contacted separately.

On the other hand, the user terminal 1a of the information processing system Xa of the present embodiment includes the position acquisition unit 100 that acquires the position information 302 of the vehicle that is calling to the meeting place and the position information 300 of the user, and the position acquisition unit 100. The vehicle location calculation unit 110 that calculates the location of the vehicle displayed in the user image data 310 acquired from the user's camera based on the acquired vehicle location information 302 and the user location information 300, and the vehicle location calculation unit 110 It is an information processing apparatus provided with the image presentation part 120 which presents the location of the calculated vehicle on the user image data 310, It is characterized by the above-mentioned.

This configuration makes it possible to know the location of the called vehicle on the image in real time. For this reason, the user can easily find out which vehicle is the called vehicle. Since it is possible to know where the vehicle has arrived, it is possible to get on without getting lost.

In addition, it is possible to selectively collate and recognize the building positional relationship on the image after checking it against landmarks known by the user. For this reason, it becomes easy for the user to specify the called vehicle.

In the above-described embodiment, the example in which the automobile terminal 2 is a smartphone or the like fixed to the holder H has been described.

However, the automobile terminal 2 may be a car navigation device, an external camera 26 and an in-vehicle camera 27 separately connected thereto. The external camera 26 may also be any combination of a front camera, a rear camera, a side camera, and the like. Further, the external camera 26 is a radar that can create image data used during automatic driving, LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a camera used for operation management and remote operation of the sensor group 25, and the like. It may be. Further, the external camera 26 and / or the in-vehicle camera 27 may be capable of capturing a wide range like a 360 ° camera. Further, as the external camera 26, an image of an external camera such as a monitoring camera around the vehicle may be acquired. As a result, it is possible to easily distinguish between valleys of buildings, places where there are many passengers getting on and off the airport, places where roads are hierarchical, and the like.

In the embodiment described above, the map data 340 is used to detect that the vehicle has come close. However, the vehicle location calculation unit 110 may detect that the user terminal 1 and the automobile terminal 2 have come close by using bidirectional communication or the like. In addition, the direction in which the camera 16 is directed can be presented to the user based on the strength of radio waves of bidirectional communication.

In addition, the vehicle location calculation unit 110 may calculate the location of the vehicle from only the position information 300 and the location information 302 without performing image recognition. Further, the vehicle location calculation unit 110 may recognize characters from the user image data 310 by performing character recognition of a taxi company or the like described in the two-dimensional barcode or the vehicle instead of the vehicle itself.

The server 3 may be able to mediate a call between the user and the driver by SIP (Session Initiation Protocol) or the like. In addition, the server 3 can also have a function of performing chat, messenger transmission / reception, e-mail transmission / reception (hereinafter referred to as “chat etc.”). This chat or the like may be input by voice input from the user terminal 1 so that automatic translation can be performed.

In addition, a plurality of user terminals 1, automobile terminals 2, and servers 3 may be provided.

The user terminal 1, the automobile terminal 2, and the server 3 may include other components in addition to the above-described units, and each unit may include a plurality of control units.

Any one of the respective units and any combination thereof may be integrally configured. For example, the input unit 12 and the display unit 13 may be integrally formed. Moreover, each of the

control units

10, 20, and 30 and the

storage units

11, 21, and 31 may be integrally formed.

Furthermore, an example in which the information processing system X of the present embodiment is used for a service for calling a vehicle such as a taxi or a hire has been described.

It can also be used for autonomous driving community cars, sharing cars, waiting for home delivery, and the like.
<Second embodiment>
Next, the information processing system Xb according to the second embodiment of the present disclosure will be described. The vehicle terminal 2b also presents the user location so that the vehicle driver can also confirm the user. On the user side, AR is presented for the location of the vehicle that has been called as in the user terminal 1a of the first embodiment described above. At this time, the appearance image data 350 is acquired from the server 3 and image recognition is performed. In addition to this, an image of the in-vehicle camera 27 of the automobile terminal 2b is also displayed. Furthermore, the imaging directions of the external camera 26 and the in-vehicle camera 27 can be operated from the user terminal 1b.
[Functional configuration of information processing system Xb]
Here, the functional configuration of the information processing system Xb according to the second embodiment of the present disclosure will be described with reference to FIG.

The control unit 10b of the user terminal 1b includes a position acquisition unit 100, a vehicle location calculation unit 110, an image presentation unit 120, and a camera control unit 130.

The storage unit 11b of the user terminal 1b stores position information 300, user image data 310, vehicle exterior image data 320, and vehicle interior image data 330.

The control unit 10b of the automobile terminal 2b includes a position acquisition unit 200, a user location calculation unit 210, and an image presentation unit 220.

The storage unit 11b of the automobile terminal 2b stores position information 302, user image data 310, vehicle exterior image data 320, and vehicle interior image data 330.

The server 3 stores map data 340 and appearance image data 350.

6, the same reference numerals as those in the second embodiment in FIG. 3 indicate the same components.

In the present embodiment, the vehicle location calculation unit 110 acquires the appearance image data 350 of the vehicle from the server 3 in addition to the processing of the first embodiment. On this basis, the vehicle location calculation unit 110 collates with the map data 340, and then uses the appearance image data 350 to recognize the image of the vehicle called by DNN or the like. Thereby, the location of the vehicle is accurately calculated.

In addition to the process of the first embodiment, the image presentation unit 120 also obtains and presents in-vehicle image data 330 captured by the in-vehicle camera 27 of the vehicle from the automobile terminal 2b.

At this time, the image presentation unit 120 displays the position of the vehicle on the map data 340 with a pointer, and also displays the appearance image data 350 in the vicinity of the pointer. As the appearance image data 350 to be displayed, a small image such as thumbnail data may be used.

In addition, for example, the image presentation unit 120 can also obtain and present the outside image data 320 captured by the external camera 26 of the automobile terminal 2b from the automobile terminal 2b. At this time, the outside image data 320 may indicate a place where the user's own figure, which is calculated by the user place calculation unit 210 of the automobile terminal 2b, is captured.

The camera control unit 130 controls the external camera 26 and / or the in-vehicle camera 27 of the vehicle. In the present embodiment, for example, the camera control unit 130 moves the orientation of the smartphone holder H as illustrated in FIG. 1B according to an instruction on the user's application, or zoom magnification of the zoom lens. It is possible to perform control to change the value. This control can be performed via an application installed in the automobile terminal 2b.

The position acquisition unit 200 acquires the position information 302 of the host vehicle and the position information 300 of the user, similar to the position acquisition unit 100 of the user terminal 1b.

The user location calculation unit 210 uses the vehicle location information acquired by the location acquisition unit 200 and the user location information 300 to determine the location of the user displayed in the vehicle outside image data 320 captured by the external camera 26 of the vehicle. calculate. For example, similar to the vehicle location calculation unit 110, the user recognition is performed by collating with the map data 340 by perspective projection transformation, affine transformation, or the like and using DNN or the like. Here, when there are a large number of persons in the outside-vehicle image data 320, the user location calculation unit 210 may recognize the person whose position is the closest as the user. At this time, the user location calculation unit 210 can acquire user information from the user terminal 1b and select a person who is most likely to be a user based on gender or age.

The image presentation unit 220 is a functional unit similar to the image presentation unit 120.

The image presentation unit 220 presents, for example, the user location calculated by the user location calculation unit 210 on the outside image data 320 in AR. In addition, the image presentation unit 220 acquires user image data 310 from the user terminal 1b, acquires the vehicle location calculated by the vehicle location calculation unit 110 of the user terminal 1b from the user terminal 1a, and presents it. It is also possible to do.

The vehicle outside image data 320 is still image data or moving image data captured by the external camera 26 of the automobile terminal 2b.

The in-vehicle image data 330 is still image or moving image data captured by the in-vehicle camera 27 of the automobile terminal 2b.

The appearance image data 350 is image data obtained by capturing the appearance of the vehicle. The image data may include data having different formats and sizes, such as recognition data used when the vehicle location calculation unit 110 recognizes an image, thumbnail data to be displayed on the user's map, and the like.
[Vehicle user presentation processing by information processing system Xb]
Next, a vehicle user presentation process by the information processing system Xb according to the second embodiment of the present disclosure will be described with reference to FIGS.

In the vehicle user presentation process of the present embodiment, in addition to the same process as the vehicle presentation process of the first embodiment, the location of the user is presented in the vehicle outside image data 320 captured from the automobile terminal 2b. In addition, in-vehicle image data 330 is also presented. At this time, the orientation of the camera of the automobile terminal 2b and the like are controlled according to a user instruction. This makes it easier for both the driver and the user of the vehicle to call and recognize each other.

In the vehicle presentation process of the present embodiment, the control unit 10b of the user terminal 1b and the control unit 20b of the automobile terminal 2b mainly store programs such as applications stored in the storage unit 11b and the storage unit 21b, respectively. To work with hardware resources.

Below, with reference to the flowchart of FIG. 7, the detail of the vehicle user presentation process of this embodiment is demonstrated for every step.
(Step S111)
First, the position acquisition unit 100 of the user terminal 1a performs a position acquisition process.

This process is performed in the same manner as step S101 in FIG.

In addition to this, in the present embodiment, the position acquisition unit 100 transmits the position information 300 acquired from the sensor group 15 to the automobile terminal 2b.
(Step S211)
Here, the position acquisition unit 200 of the automobile terminal 2b performs a position acquisition process.

The position acquisition unit 200 acquires the position information 302 from the sensor group 25 during traveling and stores it in the storage unit 21b.

The position acquisition unit 200 transmits the position information 302 to the user terminal 1b in a state where the application is activated and communication is possible between the automobile terminal 2 and the user terminal 1.

In addition, in the present embodiment, the position acquisition unit 200 also acquires the position information 300 of the user terminal 1a and stores it in the storage unit 21b.
(Step S112)
Next, the camera control unit 130 of the user terminal 1a controls the external camera 26 and / or the in-vehicle camera 27 of the vehicle.

A screen example 510 in FIG. 8A shows an example of image data 320 outside the vehicle of the external camera 26 displayed on the display unit 13 of the user terminal 1b and / or the display unit 23 of the automobile terminal 2b. According to this example, the user can press and instruct each arrow of the button 720 displayed superimposed on the outside-vehicle image data 320 and move the electric pan head of the holder in the direction of the arrow. Further, the zoom magnification of the external camera 26 or the in-vehicle camera 27 can be changed by the zoom mark of the button 730.
(Step S113)
Next, the vehicle location calculation unit 110 performs a vehicle location calculation process.

This process is also performed in the same manner as step S102 in FIG.

In addition to this, in this embodiment, the vehicle location calculation unit 110 acquires the vehicle appearance image data 350 from the server 3. Then, the vehicle location calculation unit 110 uses the appearance image data 350 to recognize the vehicle by image recognition using DNN or the like, and calculates the location of the vehicle.
(Step S213)
Here, the user location calculation unit 210 of the automobile terminal 2b performs a user location calculation process.

The user location calculation unit 210 recognizes the user displayed in the outside-vehicle image data 320 captured by the external camera 26 and calculates the location displayed by the user.
(Steps S114 and S214)
The image presentation unit 120 of the user terminal 1a and the image presentation unit 220 of the automobile terminal 2b perform image transmission / reception processing.

Here, the image presentation unit 120 transmits the user image data 310 in which the location of the vehicle has been presented to the automobile terminal 2b. Further, the image presentation unit 220 transmits the vehicle outside image data 320 and the vehicle interior image data 330 on which the user's location has been presented to the user terminal 1b. Transmission / reception of these image data may be performed by PtP (Peer to Peer) or via the server 3. Each unit receives these image data and stores them in the storage units 11b and 21b, respectively.
(Steps S115 and S215)
The image presentation unit 120 of the user terminal 1a and the image presentation unit 220 of the automobile terminal 2b each perform an image presentation process.

In addition to the processing similar to the image presentation processing in step S103 of FIG. 4, the image presentation unit 120 switches or overlaps the images of the camera 16, the external camera 26, and the in-vehicle camera 27 on the display unit 13. It can be displayed.

Also, the image presentation unit 220 of the automobile terminal 2b can switch these images and display them on the display unit 23.

8A shows an example in which the user location 710, which is the calculated user location, is AR-displayed in the image data 320 outside the vehicle.

8B shows an example in which in-vehicle image data 330 is displayed in addition to the images in the screen example 500 in FIG. 5 of the first embodiment. Further, on the map data 340, a thumbnail image of the appearance image data 350 is displayed in a balloon format in the vicinity of the pointer P indicating the position of the vehicle.

These screen examples are displayed on the display unit 13 and / or the display unit 23 in real time (real time).

Thus, the vehicle user presentation process according to the second embodiment of the present disclosure is completed.

By configuring as above, the following effects can be obtained.

In the conventional calling service, for example, there are cases where a vehicle driver does not know who is a customer (user) where there are many people. In this case, even if the vehicle arrives at the meeting place, the user cannot be recognized, which may cause a problem.

On the other hand, the automobile terminal 2b of the information processing system Xb according to the present embodiment uses the vehicle position information 302 acquired by the position acquisition unit 200 and the user position information 300 to acquire the outside of the vehicle acquired from the external camera 26 of the vehicle. A user location calculation unit 210 that calculates the location of the user displayed in the image data 320 is further provided, and the image presentation unit 120 and / or the image presentation unit 220 determine the location of the user calculated by the user location calculation unit 210. It is an information processing apparatus that is presented on the image data 320 outside the vehicle.

With this configuration, the driver can recognize the user's location from the surrounding images while driving. For this reason, it becomes easy for the driver to grasp the user, and the driver can easily reach the meeting place.

In addition, the user terminal 1b of the present embodiment may also be able to display the vehicle exterior image data 320 on the image presentation unit 120. For this reason, the user can also know which side the vehicle is traveling in real time. That is, by knowing the current state of the vehicle, it is possible to easily grasp where the called vehicle has arrived.

In this way, both the user and the driver can recognize each other, thereby reducing the possibility of waiting problems.

In the user terminal 1b of the present embodiment, the vehicle location calculation unit 110 acquires the vehicle appearance image data 350, and recognizes the vehicle from the appearance image data 350, thereby calculating the location of the vehicle. It is characterized by being.

With this configuration, it is possible to reliably recognize the called vehicle from the user image data 310 and calculate the location of the vehicle. That is, the recognition rate can be remarkably increased by image recognition using actual vehicle data, rather than DNN that categorizes a general automobile. For this reason, the user can surely recognize the called vehicle, and trouble can be prevented.

In the user terminal 1b of the present embodiment, the image presentation unit 120 displays the position of the vehicle with a pointer on the acquired map data 340 and also displays the appearance image data 350 in the vicinity of the pointer. It is characterized by being.

With this configuration, the user can easily discriminate the called vehicle on the map, and more easily find a vehicle that is nearby. In particular, when the called vehicle is characteristic in appearance such as a vehicle type or paint, it can be easily discriminated.

Conventionally, in the vehicle calling service, the user cannot know the status of the calling vehicle. Users sometimes worried that the car they actually called was in a different atmosphere from the vehicle image.

On the other hand, the user terminal 1b of the present embodiment is characterized in that the image presentation unit 120 is an information processing apparatus that also presents in-vehicle image data 330 acquired from the in-vehicle camera 27 of the vehicle.

This configuration makes it possible to check the status of the called vehicle on the app in real time. That is, not only the situation outside the vehicle but also the situation inside the vehicle can be confirmed. This makes it possible to know the situation inside the vehicle. As a result, it is possible to give the user a sense of security more than pre-registered drivers and vehicle images, word of mouth, ratings, and the like.

Furthermore, it was important to know the situation inside the car for community cars, sharing cars, and so on. This is because when there are a large number of people in the vehicle, it is necessary to request a call service separately.

Also, some users have a need to share a taxi or hire with several people (carpooling) and use it cheaply. However, when the number of carpooling increases, it sometimes feels annoying, and the user sometimes feels annoyed whether to use a separate calling service.

On the other hand, since it becomes possible to know the situation inside the vehicle, the user can determine how many people are riding and whether there is a possibility of riding. For this reason, it becomes possible for the user to easily select whether or not to call separately, and the usability of the user can be improved.

The user terminal 1b of the present embodiment is an information processing apparatus that further includes a camera control unit 130 that controls the external camera 26 and / or the in-vehicle camera 27 of the vehicle.

With this configuration, the user can operate from the app of the user terminal 1b where the car terminal 2b captures an image. That is, the user can remotely operate the camera of the automobile terminal 2b, and can freely view the image of the vehicle while remotely operating it.

This makes it possible to browse wherever you want to show the driver and where you want to browse. Therefore, the driver can be surely shown where the user is waiting in a parking lot position, a place where there are many pick-ups, and the like. In addition, it is possible for the user himself / herself to find a person wearing easily understandable clothes such as his wife or a child, for example, and to acquire this image so that the vehicle can be easily identified.

In addition, in the information processing system Xb of this embodiment, the user image data 310 captured by the user terminal 1b can be transmitted to the automobile terminal 2b.

For this reason, it is possible to transmit the image of the position where the user is, the current situation, etc. to the driver, and the driver can save time and effort to find the user. For this reason, service can be improved.

In the above-described embodiment, the external camera 26 and the in-vehicle camera 27 are controlled based on a user instruction.

On the other hand, when the vehicle is stopped, the driver of the vehicle may drive the holder H via the application of the vehicle terminal 2b to change the orientation and zoom magnification of the external camera 26 and the in-vehicle camera 27.

Further, the outside image data 320, the in-vehicle image data 330, and the user image data 310 are not only transmitted to each terminal in real time, but may be stored at each terminal for a specific period. As a result, it is possible to browse an image several seconds to several minutes ago.

In addition, the in-vehicle image data 330 may include not only an image of the in-vehicle camera 27 but also an image of an external camera that allows an appearance image of the car to be seen.
<Third embodiment>
Next, the information processing system Xc according to the third embodiment of the present disclosure will be described with reference to FIG. Also in FIG. 9, the same code | symbol is attached | subjected to the structure similar to the above-mentioned 1st embodiment and 2nd embodiment.

In the first embodiment described above, the location of the vehicle is presented in the user image data 310. In the second embodiment, in addition to the location of the vehicle, the location of the user is presented in the image data 320 outside the vehicle.

On the other hand, as in the information processing system Xc, the user location may be presented only by the automobile terminal 2c. In the example of FIG. 9, the control unit 20 c includes a position acquisition unit 200, a user location calculation unit 210, and an image presentation unit 220. The storage unit 21c stores position information 300 and image data 320 outside the vehicle. The user terminal 1 c includes position information 302.

That is, the automobile terminal 2c of the present embodiment includes the position acquisition unit 200 that acquires the position information 302 of the vehicle that is calling to the meeting place and the position information 300 of the user, and the position of the vehicle and the user that are acquired by the position acquisition unit 100. Based on the information 300, the user location calculation unit 210 that calculates the location of the user displayed in the vehicle external image data acquired from the external camera 26 of the vehicle, and the location of the user calculated by the user location calculation unit 210 It is an information processing apparatus provided with the image presentation part 120 presented on external image data.

With this configuration, it is possible to make it easier for the driver of the automobile to recognize the user, as in the second embodiment described above. In this case, since it is not necessary to confirm the car that the user has called, the burden on the user can be reduced.
<Fourth embodiment>
Next, an information processing system Xd according to the fourth embodiment of the present disclosure will be described with reference to FIG. Also in FIG. 10, the same code | symbol is attached | subjected to the structure similar to the above-mentioned 1st thru | or 3rd embodiment.

In the first to fourth embodiments described above, an example in which the server 3 includes the map data 340 has been described. However, in the present embodiment, for example, the same processing as in the second embodiment described above is performed only by the user terminal 1d and the automobile terminal 2d as in PtP.

In the example of FIG. 10, the control unit 10d and the storage unit 11d of the user terminal 1d and the control unit 20d of the automobile terminal 2d have the same configuration as the control unit 10b, the storage unit 11b, and the control unit 20b of the second embodiment. is there. On the other hand, the storage unit 21d of the automobile terminal 2d stores map data 340 in addition to the data of the storage unit 21b of the second embodiment.

With this configuration, the user and the driver can confirm each other without going through the server 3. In addition, the car terminal 2d stores the map data 340 for the dedicated navigation, and collates based on the map data 340, thereby preventing a shift of the map data 340 between the user terminal 1d and the car terminal 2d, and the location of the user and the vehicle. It becomes easy to calculate.

In this example, the map data 340 is described as being stored in the automobile terminal 2d, but may be the user terminal 1d. In addition, a navigation device that can be connected to the automobile terminal 2b is provided separately, and the map data 340 may be acquired from the navigation device. Furthermore, you may make it recognize a vehicle or a user's location directly, without using the map data 340. FIG.
<Fifth embodiment>
Next, an information processing system Xe according to the fifth embodiment of the present disclosure will be described with reference to FIG. Also in FIG. 11, the same reference numerals are given to the same configurations as those in the first to fourth embodiments described above.

In the first to fourth embodiments described above, the vehicle or user location is calculated from the image data at each terminal. However, like the information processing system Xe, the server 3e may calculate these points. In this case, for example, the server 3e includes a user location calculation unit 210 and a vehicle location calculation unit 110 in the control unit 30e. In addition to the map data 340, the storage unit 31e acquires the position information 300 and the user image data 310 from the user terminal 1e and stores them. Further, the position information 302, the outside image data 320, and the in-house image data are acquired from the automobile terminal 2e and stored in the storage unit 31d. Then, each image data is distributed from the server 3e to each terminal by streaming or the like.

In this example, the control unit 10e of the user terminal 1e includes a position acquisition unit 100, an image presentation unit 120, and a camera control unit 130. The control unit 20e of the automobile terminal 2e includes a position acquisition unit 200 and an image presentation unit 220.

With this configuration, the processing burden on the user terminal 1e and the automobile terminal 2e can be reduced. Further, by accumulating each image data in the server 3e and recognizing each location, it is possible to efficiently learn and to improve the accuracy of collation with the map data 340 and each recognition.

Further, it is possible to configure the storage unit 31e of the server 3e to store the vehicle outside image data 320 and the like for a specific period.

This makes it possible to search the user terminal 1e along the time series to find out where the vehicle was running, thereby making it easier to find the vehicle.

In addition, according to FIG. 12, by accumulating the outside image data 320 of each vehicle using the vehicle calling service, the user can view the outside image data 320 collated with the map data 340. Thus, the user can set the destination.

For example, as shown in a screen example 520 in FIG. 12A, the user sets a destination with a pin on the map. In this case, as shown in the screen example 530 of FIG. 12B, the corresponding outside-vehicle image data 320 can be displayed. Thereby, in the user terminal 1e, a destination can be set by an image of a location where the vehicle has actually moved.

Furthermore, it is possible to set a destination setting of “I want to go to that neighborhood” from the collation result between the map data 340 and the landmark of the user image data 310 on the user terminal 1e.

This configuration makes it possible to easily set the destination even when the user does not know the arrival point. For this reason, it is possible to cope with a case where it is difficult to explain an address at a travel destination or a place visited for the first time. That is, it is possible to easily search for a destination that the user remembers vaguely as “that neighborhood”. In addition, the user can reliably tell the driver where to go by showing the image.

By accumulating the image data 320 that is collated with the map data 340 in the server 3e, it is possible to view the destination image data after the user actually gets on the vehicle. For this reason, even when the user does not know the arrival place only by the address, the user can explain to the driver that he / she wants to go to “that neighborhood” by browsing the destination image.

In addition, 360 ° images can be created from the stored outside-vehicle image data 320 and VR (Virtual Reality) moving images can be distributed in real time.

Furthermore, a configuration in which the in-vehicle image data 330 and the like are stored in the storage unit 31e of the server 3e for a specific period is possible.

Thereby, it is possible for the user to easily determine from the user terminal 1e whether or not to use the calling service based on the past in-vehicle situation and how many people are on the board or whether there is a possibility of getting on.
<Other embodiments>
In the above-described embodiment, only the location of the user calling the vehicle is displayed in the outside image data 320.

However, you may make it display all the parts of the user who is starting the application in the image data 320 outside a vehicle. In addition, on the map data 340, the user data used in the past, the flow and number of people by mobile radio waves, etc. may be shown.

With this configuration, it is possible to determine whether or not the user or driver is the most suitable meeting place, and in some cases, the meeting place can be changed. This can prevent troubles from occurring.

In the embodiment described above, the user image data 310 is described so as to present only the location of the vehicle.

However, even after the user gets on the vehicle, it is also possible to display AR introductions of historical buildings, town history, news, weather, etc. while collating with the map data 340 while moving. At this time, automatic translation or the like may be performed, or information for AR display may be selected by estimating the user's emotion.

In this way, it is possible to create inspiration by matching the knowledge with the experience on the spot. Users may also be idle on the move and do not want to talk to the driver. Even in this case, the user can be provided with convenience by viewing the AR display.

In the first to fifth embodiments of the present disclosure described above, the features of the present disclosure have been described focusing on the main configuration. However, the combination of the structures of the functional units in these embodiments is arbitrary. That is, any one of the configurations of the first to fifth embodiments, a configuration of any combination, or a configuration including all functional units may be used.

Further, the user terminal 1 may further include a functional unit that is not described in the first to fifth embodiments.

The control unit and its method described in the present disclosure are realized by a dedicated computer provided by configuring a processor and a memory programmed to execute one or more functions embodied by a computer program. May be. Alternatively, the control unit and the method thereof described in the present disclosure may be realized by a dedicated computer provided by configuring a processor with one or more dedicated hardware logic circuits. Alternatively, the control unit and the method thereof described in the present disclosure may include a combination of a processor and a memory programmed to execute one or more functions and a processor configured by one or more hardware logic circuits. It may be realized by one or more configured dedicated computers. The computer program may be stored in a computer-readable non-transition tangible recording medium as instructions executed by the computer.

Here, the flowchart described in this application or the process of the flowchart is configured by a plurality of sections (or referred to as steps), and each section is expressed as S101, for example. Further, each section can be divided into a plurality of subsections, while a plurality of sections can be combined into one section. Further, each section configured in this manner can be referred to as a device, module, or means.

Although the present disclosure has been described based on the embodiments, it is understood that the present disclosure is not limited to the embodiments and structures. The present disclosure includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more or less, are within the scope and spirit of the present disclosure.

Claims

A position acquisition unit (100) for acquiring the position information (302) of the vehicle calling to the meeting place and the position information (300) of the user;
The location of the vehicle displayed in the user image data (310) captured by the user's camera (16) is calculated based on the vehicle location information and the user location information acquired by the location acquisition unit. A vehicle location calculator (110);
An information processing apparatus comprising: an image presentation unit (120) that presents the location of the vehicle calculated by the vehicle location calculation unit on the user image data.
The location of the user displayed in the outside image data (320) captured by the external camera (26) of the vehicle based on the position information of the vehicle and the position information of the user acquired by the position acquisition unit. A user location calculation unit (210) for calculating,
The image presentation unit
The information processing apparatus according to claim 1, wherein the user location calculated by the user location calculation unit is presented on the image data outside the vehicle.
The vehicle location calculator
The information processing apparatus according to claim 1 or 2, wherein the vehicle location is calculated by acquiring the vehicle appearance image data (350) and recognizing the vehicle from the appearance image data.
The image presentation unit
The information processing apparatus according to claim 3, wherein the position of the vehicle is displayed with a pointer on the acquired map data (340), and the appearance image data is displayed in the vicinity of the pointer.
The image presentation unit
The information processing apparatus according to any one of claims 1 to 4, further presenting in-vehicle image data (330) captured by the in-vehicle camera (27) of the vehicle.
The information processing apparatus according to any one of claims 2 to 5, further comprising a camera control unit (130) configured to control at least one of the external camera and the in-vehicle camera of the vehicle.
A position acquisition unit (100) for acquiring the position information (302) of the vehicle calling to the meeting place and the position information (300) of the user;
A user who calculates the location of the user displayed in the vehicle external image data (320) acquired from the external camera (26) of the vehicle based on the vehicle and user position information acquired by the position acquisition unit. A location calculation unit (210);
An information processing apparatus comprising: an image presentation unit (120) that presents the user location calculated by the user location calculation unit on the vehicle external image data.
An information processing program executed by the information processing apparatus (1),
The position acquisition unit (100) is allowed to acquire the position information (320) of the vehicle calling to the meeting place and the position information (300) of the user,
Displayed in the vehicle location calculation unit (110) in the user image data (310) captured by the user's camera (16) based on the vehicle location information and the user location information acquired by the location acquisition unit. To calculate the location of the vehicle to be
A program for causing an image presentation unit (120) to present the vehicle location calculated by the vehicle location calculation unit on the user image data.
An information processing method executed by the information processing apparatus (1),
Obtain the location information (302) and the location information (300) of the vehicle calling to the meeting place,
Based on the acquired position information of the vehicle and the position information of the user, the location of the vehicle displayed in the user image data (310) captured by the user's camera (16) is calculated,
An information processing method for presenting the calculated location of the vehicle on the user image data.