JP2019191914A - Information processor, program, and information processing method - Google Patents

Abstract

To provide an information processor enabling a user to readily identify a vehicle the user has hailed.SOLUTION: A position acquisition unit 100 acquires position information 302 on a vehicle being hailed to a meeting place, and position information 300 on a user. A vehicle spot calculation unit 110 uses the position information 302 on the vehicle and the position information 300 on the user, which are acquired by the position acquisition unit 100, to calculate the spot of the vehicle shown in user image data 310 acquired from a user's camera 16. An image presentation unit 120 presents the spot of the vehicle, which is calculated by the vehicle spot calculation unit 110, in the user image data 310.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an information processing device, a program, and an information processing method, and more particularly, to an information processing device, a program, and an information processing method used in a service that calls a vehicle.

Conventionally, a service in which a user calls a vehicle such as a taxi through a portable terminal has been widespread.
In such a service, Patent Document 1 discloses a navigation device in which a user can easily check a vehicle that is heading for a meeting place. The navigation device generates voice guidance information indicating from which direction the vehicle enters the meeting place based on the position information and route information of the set meeting place. This device transmits the generated guidance information to the user's terminal. The user can easily confirm the vehicle by listening to the voice guidance information.

JP 2006-25874 A

  However, in the apparatus of Patent Document 1, when the meeting place is crowded with many taxi vehicles or the like, it may be difficult for the user to specify which vehicle the called vehicle is.

  The present invention has been made in view of such a situation, and an object of the present invention is to provide an information processing apparatus that makes it easy for a user to recognize a called vehicle and solves the above-described problems.

The information processing apparatus according to the present invention includes a position acquisition unit (100) that acquires position information (302) and a user's position information (300) of a vehicle calling to a meeting place, and the vehicle acquired by the position acquisition unit. Vehicle location calculation unit (110) that calculates the location of the vehicle displayed in user image data (310) captured by the user's camera (16) based on the location information of the user and the location information of the user, It is an information processing apparatus (1) provided with the image presentation part (120) which presents the location of the said vehicle calculated by the vehicle location calculation part on the said user image data.
The information processing apparatus according to the present invention includes a position acquisition unit (100) that acquires position information (302) and a user's position information (300) of a vehicle calling to a meeting place, and the vehicle acquired by the position acquisition unit. And a user location calculation unit (210) for calculating the location of the user displayed in the vehicle external image data (320) acquired from the external camera (26) for the vehicle, based on the location information of the user, and the user It is an information processing device (2) provided with the image presentation part (120) which presents the said user's location calculated by the location calculation part on the said vehicle external image data.
The program according to the present invention is an information processing program executed by the information processing apparatus (1), and includes the position information (320) of the vehicle calling to the meeting place and the position information of the user (100). 300) and the vehicle location calculation unit (110) causes the user image data captured by the user's camera (16) based on the vehicle position information and the user position information acquired by the position acquisition unit. (310) A program for calculating the location of the vehicle displayed in (310) and causing the image presentation unit (120) to present the location of the vehicle calculated by the vehicle location calculation unit on the user image data. It is characterized by that.
The information processing method according to the present invention is an information processing method executed by the information processing apparatus (1), and acquires the position information (302) of the vehicle calling to the meeting place and the position information (300) of the user, Based on the acquired position information of the vehicle and the position information of the user, the location of the vehicle displayed in the user image data (310) captured by the user's camera (16) is calculated, and the calculated position is calculated. It is an information processing method for presenting a location of a vehicle on the user image data.

  According to the present invention, first, the position information of the vehicle and the position information of the user calling to the meeting place are acquired. Then, the location of the vehicle displayed in the image data of the user's camera is calculated based on the position information. Then, the calculated location of the vehicle is presented on the image data. Thereby, it is possible to provide an information processing apparatus that allows the user to easily specify the called vehicle.

1 is a system configuration diagram of an information processing system according to a first embodiment of the present invention. It is a block diagram which shows the control structure of the user terminal shown in FIG. 1, a motor vehicle terminal, and a server. It is a block diagram which shows the function structure of the information processing system which concerns on 1st embodiment of this invention. It is a flowchart of the vehicle presentation process which concerns on 1st embodiment of this invention. It is an example of a screen of the vehicle presentation process shown in FIG. It is a block diagram which shows the function structure of the information processing system which concerns on 2nd embodiment of this invention. It is a flowchart of the vehicle user presentation process which concerns on 2nd embodiment of this invention. It is an example of a screen of the vehicle user presentation process shown in FIG. It is a block diagram which shows the function structure of the information processing system which concerns on 3rd embodiment of this invention. It is a block diagram which shows the function structure of the information processing system which concerns on 4th embodiment of this invention. It is a block diagram which shows the function structure of the information processing system which concerns on 5th embodiment of this invention. It is an example of the screen of the information processing system which concerns on 5th embodiment of this invention.

<First embodiment>
[System configuration of information processing system X]
First, the system configuration of the information processing system X according to the first embodiment of the present invention will be described with reference to FIG.
The information processing system X of the present embodiment is a system that enables confirmation when a user calls a car such as a taxi or a hire.
According to FIG. 1A, the information processing system X includes a user terminal 1, an automobile terminal 2, and a server 3, which are connected via a network 4.

  The user terminal 1 can use services on the web provided by the server 3, and includes a smartphone with a camera (Smart Phone), a mobile phone, a PDA (Personal Data Assistant), a PC (Personal Computer), a personal navigation device, It is a device such as a home appliance. In the present embodiment, an example in which the user terminal 1 is a smartphone will be described. This smartphone functions as an information processing apparatus according to the present embodiment by installing and executing application software (hereinafter simply referred to as “application”) described later.

The automobile terminal 2 is an in-car terminal of an automobile that uses a service on the web provided by the server 3. The in-vehicle terminal is a smartphone, a mobile phone, a PDA, a car navigation device, or the like.
In this embodiment, an example in which the automobile terminal 2 is also a smartphone will be described as shown in FIG. In this example, the smartphone is placed on the holder H on the dashboard D of the automobile (vehicle) and used for car navigation by the driver (operator) of the automobile. With this arrangement, the outside and inside of the vehicle can be imaged with the out camera and the in camera of the smartphone, respectively. In addition, the holder H is driven like an electric head by control from the automobile terminal 2 and can change the direction of imaging by the camera of the smartphone.

The server 3 is a server that performs management in a vehicle call service. Specifically, the server 3 receives a request for calling a car from the user terminal 1, searches for the car terminal 2 of the car that can be called, and mediates both terminals. For example, the server 3 estimates user characteristics from information such as the purchase history, viewing history, and travel history of the user, and performs matching of charges, drivers, vehicles, and the like based on the characteristics.
At this time, the server 3 introduces the driver of the vehicle registered in the service, the vehicle type of the vehicle, image data obtained by imaging the appearance of the vehicle, streaming data, description language data, audio data, etc. (hereinafter simply referred to as “image data etc.”). Can be distributed to the user. The image data or the like may be a still image or a moving image. Furthermore, the server 3 can also display word-of-mouth, rating, etc. for each vehicle.
In addition, the server 3 may set an application to be installed in the user terminal 1, obtain various information, manage the operation of the application, or distribute the application itself.

  The network 4 is a cellular phone network, a WAN (Wide Area Network) such as the Internet (registered trademark), or an IP network such as a LAN (Local Area Network) such as WiFi (registered trademark) or a wireless LAN. These networks and the like may be always connected (connected) from the user terminal 1 and the automobile terminal 2, for example, and in the case of a mobile phone network, 4G (4th Generation), 5G (5th Generation), etc. It may be a standard low-latency and high-speed communication network.

  In the second to fifth embodiments described later, the system configuration may be the same.

Next, device configurations of the user terminal 1, the automobile terminal 2, and the server 3 will be described with reference to FIG. 2A shows the configuration of the user terminal 1, FIG. 2B shows the configuration of the automobile terminal 2, and FIG. 2C shows the configuration of the server 3.
The user terminal 1 includes a control unit 10, a storage unit 11, an input unit 12, a display unit 13, a connection unit 14, a sensor group 15, a camera 16, a voice input / output unit 18, and the like. Each unit is connected to the control unit 10 and controlled in operation by the control unit 10.
The automobile terminal 2 includes a control unit 20, a storage unit 21, an input unit 22, a display unit 23, a connection unit 24, an external camera 26, an in-vehicle camera 27, a voice input / output unit 28, and the like. Each unit is connected to the control unit 20 and controlled in operation by the control unit 20.
The server 3 includes a control unit 30, a storage unit 31, a connection unit 34, and the like. Each unit is connected to the control unit 30 and controlled in operation by the control unit 30.

The control units 10, 20, and 30 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU, a TPU (Tensor Processing Unit), a DFP (Data Flow Processor), a DSP (Digital Signal Processor), An information processing unit including other ASIC (Application Specific Processor).
Each control unit 10, 20, 30 reads a program stored in the auxiliary storage unit of each corresponding storage unit 11, 21, 31 and develops this program in the main storage unit and executes the program, which will be described later. It is made to operate as each functional block (functional unit). The control units 10, 20, and 30 control the entire devices of the user terminal 1, the automobile terminal 2, and the server 3.

Storage units 11, 21, and 31 are a main storage unit such as a RAM (Random Access Memory), an auxiliary storage unit such as a ROM (Read Only Memory), an SSD (Solid State Disk), and an HDD (Hard Disk Drive), and a flash memory card. And non-transitory tangible storage media, which may include optical recording media and the like.
The auxiliary storage units of the storage units 11, 21, and 31 store programs for controlling the operation of the user terminal 1, the automobile terminal 2, and the server 3, respectively. This program is an OS (Operating System), an application, and the like. In addition, the storage units 11, 21, and 31 also store various data.

  The input units 12 and 22 are a touch panel, a pointing device such as a mouse, buttons, a keyboard, an acceleration sensor, a line-of-sight sensor, a biometric sensor, and the like. The input unit 12 acquires input to the web page form, various instructions, and the like by the user.

  The display units 13 and 23 are an LCD (Liquid Crystal Display), an organic EL display, an LED (Light Emitting Diode), or the like.

  The connection units 14, 24, and 34 are connection means including a wireless transceiver and a LAN interface for connecting to the network 4. The connection units 14, 24, and 34 may include an interface to USB (Universal Serial Bus), RS-232C, various flash memory cards, SIM cards, Bluetooth (registered trademark), and the like.

  The sensor groups 15 and 25 are GNSS (Global Navigation Satellite System) position information sensors, gyro sensors, magnetic sensors, altitude sensors, and the like. In the present embodiment, the sensor group 15 detects the position (coordinates), orientation, speed, acceleration, etc. (hereinafter referred to as “position etc.”) of the map of the device itself.

  The camera 16 is provided on the user terminal 1 to capture an image of the user himself or the user's surroundings. The camera 16 is a camera in which an imaging element such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor and an optical element such as a lens are combined. The camera 16 captures a still image or a moving image mainly directed by the user and stores the image data in the storage unit 11.

  The voice input / output units 18 and 28 are a microphone and an A / D (Analog to Digital) converter for voice input, a D / A (Digital to Analog) converter for voice output, an amplifier, a speaker, and a vibration motor. Etc. With these configurations, the voice input / output units 18 and 28 perform notification such as a voice call of a user or driver, input of a voice command, information notification at the time of calling, navigation, and the like.

  The external camera 26 is a camera for external imaging of the vehicle. In this embodiment, the smartphone's out camera is used for external imaging. Image data captured by this out-camera is stored in the storage unit 21.

  The in-vehicle camera 27 is a camera for imaging the inside of the vehicle. In this embodiment, the in-camera of a smart phone is used for in-vehicle imaging. Image data captured by the in-camera is also stored in the storage unit 21.

  In the camera 16, the external camera 26, and the in-vehicle camera 27, whether a still image or a moving image is captured, the image resolution, the zoom magnification, and the like can be changed via an application installed in the user terminal 1. These changes may also be possible in the automobile terminal 2.

[Functional configuration of information processing system Xa]
Here, the functional configuration of the information processing system Xa according to the first embodiment of the present invention will be described with reference to FIG. Also in the following embodiments, the information processing system X, the user terminal 1, the server 3, and the like having different functional unit configurations are indicated by adding lowercase letters a, b, c,.
The control unit 10a of the user terminal 1a includes a position acquisition unit 100, a vehicle location calculation unit 110, and an image presentation unit 120.
The storage unit 11a of the user terminal 1a stores position information 300 and user image data 310.
The automobile terminal 2 a includes position information 302.
The server 3 stores map data 340.

  The position acquisition unit 100 acquires the position information 302 and the user position information 300 of the vehicle calling to the meeting place. In the present embodiment, the meeting place is a position designated by the user through the application and contacted with the driver of the automobile via the server 3. This meeting place can be indicated on the map data 340. The position information 302 is acquired from the automobile terminal 2 via the server 3. The user position information 300 is acquired by the sensor group 15 of the own device.

  The vehicle location calculation unit 110 is displayed in the user image data 310 in which the user's surroundings are captured by the user's camera 16 based on the vehicle location information 302 and the user location information 300 acquired by the location acquisition unit 100. Calculate the location of the vehicle. In the present embodiment, for example, the vehicle location calculation unit 110 calculates the position of the called vehicle on the map data 340 from the position information 302, and the position of the user position information 300 on the user's map data 340; The imaging direction of the camera 16 is calculated. And the vehicle location calculation part 110 collates with the map data 340 by DNN (Deep neural network) etc. by performing perspective projection conversion, affine transformation, etc. on the user image data 310. FIG. Thereby, the vehicle location calculation part 110 can grasp | ascertain the coordinate of the building in the user image data 310, or a road. Further, the vehicle location calculation unit 110 recognizes the image of the automobile on the road by image recognition using DNN or the like, and searches for a match with the coordinates such as the position of the called vehicle. If there is a match, the vehicle location calculation unit 110 calculates the coordinates as the location of the vehicle.

The image presentation unit 120 presents the vehicle location calculated by the vehicle location calculation unit 110 on the user image data 310. In the present embodiment, for example, AR (Augmented Reality) can be used to add an annotation to a captured image with a circle or the like so that the vehicle image is emphasized.
At this time, the image presentation unit 120 can also show the user image data 310 also the buildings and road names recognized by the image recognition. Further, the image presentation unit 120 can acquire the map data 340 from the server 3, arrange the vehicle position information 302 and the user position information 300 on the map, and show them on the display unit 13.
In addition, the image presentation unit 120 can also indicate to the user the direction of arrival of the called vehicle.

  The position information 300 is information such as the position of the own apparatus (user position) acquired by the sensor group 15 of the user terminal 1a that is the own apparatus. The position information 300 may be away from the meeting place.

  The position information 302 is information such as the position of the automobile terminal 2a (the position of the vehicle) acquired by the sensor group 25 of the automobile terminal 2a. In the present embodiment, for example, the position information 302 is acquired from the vehicle terminal 2 a of the vehicle being called by the position acquisition unit 100 of the user terminal 1 via the server 3.

  The user image data 310 is still image or moving image image data captured by the camera 16. In the present embodiment, an AR display scenery including the surrounding roads and buildings around the user, that is, a car that is called to come, is imaged by the user.

Here, the control unit 10a of the user terminal 1 executes a program such as an application on an OS (Operating System) stored in the storage unit 11a, so that the position acquisition unit 100, the vehicle location calculation unit 110, and the image presentation are performed. It functions as the unit 120 or the like. At this time, some or all of the functions executed by the control units 10, 20, and 30 may be configured in hardware by one or a plurality of ICs, DSPs, programmable logic circuits, and the like.
Moreover, each part of the above-mentioned user terminal 1 becomes a hardware resource which performs the method of this invention.

[Vehicle presentation processing by information processing system Xa]
Next, with reference to FIGS. 4-5, the vehicle presentation by the information processing system Xa which concerns on 1st embodiment of this invention is demonstrated.
In the vehicle presentation process of the present embodiment, the position information 302 of the vehicle calling to the meeting place and the position information 300 of the user are acquired. And the location of the vehicle displayed in the user image data 310 acquired from the user's camera is calculated from the acquired position information 302 and position information 300. Thereafter, the calculated location of the vehicle is presented on the user image data 310 so that the user can easily recognize the called vehicle.
In the vehicle presentation process of the present embodiment, the control unit 10a of the user terminal 1 mainly executes a program such as an application stored in the storage unit using hardware resources in cooperation with each unit.
Below, with reference to the flowchart of FIG. 4, the detail of the vehicle presentation process of this embodiment is demonstrated for every step.

(Step S101)
First, the position acquisition unit 100 performs a position acquisition process.
Here, the user activates the application on the user terminal 1, connects to the server 3, and requests to call the vehicle. The server 3 sets a vehicle to be called, notifies the vehicle terminal 2 of the vehicle, and enables communication with the user terminal 1.
In this state, the position acquisition unit 100 acquires the position information 302 from the automobile terminal 2a of the vehicle calling to the meeting place and stores it in the storage unit 11a.
The position acquisition unit 100 also acquires the position information 300 of the user terminal 1a from the sensor group 15 and stores it in the storage unit 11a.

(Step S102)
Next, the vehicle location calculation unit 110 performs a vehicle location calculation process.
In particular, when the user calls and waits for the vehicle, the user is wondering where the vehicle is running. Therefore, it is considered that the user captures the user image data 310 at various locations and displays the AR.
For this reason, when the vehicle arrives at a distance close to the user terminal 1a and can be captured as the user image data 310, the image presentation unit 120 may notify the user to that effect. This notification is performed by displaying a pop-up or the like on the screen of the application.

The vehicle location calculation unit 110 calculates, based on the vehicle location information 302 and the user location information 300, the direction in which the vehicle comes when the called vehicle comes close, and directs the camera 16 of the user terminal to the user. To instruct. This instruction may be performed on the screen of the application.
The user points the camera 16 in the instructed direction so as to collide with a landmark known to the user.
When the camera 16 is directed in the direction in which the vehicle comes, the vehicle location calculation unit 110 may indicate to the user by voice or vibration by the voice input / output unit 18.
When the user captures the user image data 310 with the camera 16, the vehicle location calculation unit 110 recognizes the vehicle displayed therein and calculates the location where the called vehicle is captured.

(Step S103)
Next, the image presentation unit 120 performs image presentation processing.
The image presentation unit 120 presents the calculated vehicle location on the user image data 310 by AR.
According to the screen example 500 of the application shown in FIG. 5, the image presentation unit 120 first displays a message such as “User, driver has arrived” in the display column 610 when the vehicle that has been called is captured. Then, the image presentation unit 120 displays the user image data 310 on the screen of the application, and displays the vehicle location 700 that is the calculated location of the vehicle as an AR.
The image presentation unit 120 also displays the map data 340 acquired from the server 3, displays the position of the user with the pointer U and displays the position of the vehicle with the pointer P thereon. On this map data 340, the route of the vehicle to the meeting place, the arrow of the traveling direction of the vehicle, etc. may be shown. This makes it easier to recognize the vehicle that has been called to the user.
Thus, the vehicle presentation process according to the first embodiment of the present invention is completed.

With the configuration described above, the following effects can be obtained.
Conventionally, services for attracting vehicles have been provided. However, when the meeting place of the vehicle is congested or the arrival point is shifted, the user may not be able to recognize the called vehicle. In addition, since the user does not know the appearance of the called vehicle, for example, even if information such as the vehicle type is given from the server, there is a possibility that the user does not know.
In such a case, the vehicle cannot be specified only by the above-described service, and the driver of the vehicle who had bothered to call must be contacted separately.

On the other hand, the user terminal 1a of the information processing system Xa of the present embodiment includes the position acquisition unit 100 that acquires the position information 302 of the vehicle that is calling to the meeting place and the position information 300 of the user, and the position acquisition unit 100 The vehicle location calculation unit 110 that calculates the location of the vehicle displayed in the user image data 310 acquired from the user's camera based on the acquired vehicle location information 302 and the user location information 300, and the vehicle location calculation unit 110 It is an information processing apparatus provided with the image presentation part 120 which presents the location of the calculated vehicle on the user image data 310, It is characterized by the above-mentioned.
By comprising in this way, the location of the called vehicle can be known on the image in real time. For this reason, the user can easily find out which vehicle is the called vehicle. Since it is possible to know where the vehicle has arrived, it is possible to get on without getting lost.
In addition, it is possible to selectively perform collation and recognition on the image with respect to the positional relationship between the buildings in comparison with landmarks known to the user. For this reason, it becomes easy for the user to specify the called vehicle.

In the above-described embodiment, an example in which the automobile terminal 2 is a smartphone or the like fixed to the holder H has been described.
However, the automobile terminal 2 may be a car navigation device and an external camera 26 and an in-vehicle camera 27 separately connected thereto. The external camera 26 may also be any combination of a front camera, a rear camera, a side camera, and the like. Further, the external camera 26 is a radar capable of creating image data used during automatic driving, LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), a camera used for operation management or remote operation of the sensor group 25, and the like. It may be. Further, the external camera 26 and / or the in-vehicle camera 27 may be capable of capturing a wide range like a 360 ° camera. Further, as the external camera 26, an image of an external camera such as a monitoring camera around the vehicle may be acquired. As a result, it is possible to easily distinguish between valleys of buildings, places where there are many passengers getting on and off the airport, places where roads are hierarchical, and the like.

In the above-described embodiment, the map data 340 is used to detect that the vehicle has come close. However, the vehicle location calculation unit 110 may detect that the user terminal 1 and the automobile terminal 2 have come close by using bidirectional communication or the like. In addition, the direction in which the camera 16 is directed can be presented to the user based on the strength of radio waves of bidirectional communication.
In addition, the vehicle location calculation unit 110 may calculate the location of the vehicle from only the position information 300 and the location information 302 without performing image recognition. Further, the vehicle location calculation unit 110 may recognize characters from the user image data 310 by performing character recognition of a taxi company or the like described in the two-dimensional barcode or the vehicle instead of the vehicle itself.

  The server 3 may be able to mediate a call between the user and the driver by SIP (Session Initiation Protocol) or the like. In addition, the server 3 can also have a function of performing chat, messenger transmission / reception, e-mail transmission / reception (hereinafter referred to as “chat etc.”). This chat or the like may be input by voice input from the user terminal 1 so that automatic translation can be performed.

In addition, a plurality of user terminals 1, automobile terminals 2, and servers 3 may be provided.
The user terminal 1, the automobile terminal 2, and the server 3 may include other components in addition to the above-described units, and each unit may include a plurality of control units.
Any one of the units and any combination thereof may be integrally formed. For example, the input unit 12 and the display unit 13 may be formed integrally. Moreover, each of the control units 10, 20, and 30 and the storage units 11, 21, and 31 may be integrally formed.

Furthermore, the example which uses the information processing system X of this embodiment for the service of calling vehicles, such as a taxi and a hire, was described.
It can also be used for autonomous driving community cars, sharing cars, waiting for home delivery, and the like.

<Second embodiment>
Next, the information processing system Xb according to the second embodiment of the present invention will be described. The vehicle terminal 2b also presents the user location so that the vehicle driver can also confirm the user. On the user side, AR is presented for the location of the vehicle that has been called as in the user terminal 1a of the first embodiment described above. At this time, the appearance image data 350 is acquired from the server 3 and image recognition is performed. In addition to this, an image of the in-vehicle camera 27 of the automobile terminal 2b is also displayed. Furthermore, the imaging directions of the external camera 26 and the in-vehicle camera 27 can be operated from the user terminal 1b.

[Functional configuration of information processing system Xb]
Here, the functional configuration of the information processing system Xb according to the second embodiment of the present invention will be described with reference to FIG.

The control unit 10b of the user terminal 1b includes a position acquisition unit 100, a vehicle location calculation unit 110, an image presentation unit 120, and a camera control unit 130.
The storage unit 11b of the user terminal 1b stores position information 300, user image data 310, vehicle exterior image data 320, and vehicle interior image data 330.
The control unit 10b of the automobile terminal 2b includes a position acquisition unit 200, a user location calculation unit 210, and an image presentation unit 220.
The storage unit 11b of the automobile terminal 2b stores position information 302, user image data 310, vehicle exterior image data 320, and vehicle interior image data 330.
The server 3 stores map data 340 and appearance image data 350.
In FIG. 6, the same reference numerals as those in the second embodiment in FIG. 3 indicate similar components.

  In the present embodiment, the vehicle location calculation unit 110 acquires the vehicle appearance image data 350 from the server 3 in addition to the processing of the first embodiment. On this basis, the vehicle location calculation unit 110 collates with the map data 340, and then uses the appearance image data 350 to recognize the image of the vehicle called by DNN or the like. Thereby, the location of the vehicle is accurately calculated.

In addition to the processing of the first embodiment, the image presentation unit 120 also obtains and presents in-vehicle image data 330 captured by the in-vehicle camera 27 of the vehicle from the automobile terminal 2b.
At this time, the image presentation unit 120 displays the position of the vehicle on the map data 340 with a pointer, and also displays the appearance image data 350 in the vicinity of the pointer. As the appearance image data 350 to be displayed, a small image such as thumbnail data may be used.
In addition, the image presentation unit 120 can also obtain and present the outside-vehicle image data 320 captured by the external camera 26 of the automobile terminal 2b from the automobile terminal 2b, for example. At this time, the outside image data 320 may indicate a place where the user's own figure, which is calculated by the user place calculation unit 210 of the automobile terminal 2b, is captured.

  The camera control unit 130 controls the external camera 26 and / or the in-vehicle camera 27 of the vehicle. In the present embodiment, for example, the camera control unit 130 moves the orientation of the smartphone holder H as illustrated in FIG. 1B according to an instruction on the user's application, or zoom magnification of the zoom lens. It is possible to perform control to change the value. This control can be performed via an application installed in the automobile terminal 2b.

  Similar to the position acquisition unit 100 of the user terminal 1b, the position acquisition unit 200 acquires the position information 302 of the host vehicle and the position information 300 of the user.

  The user location calculation unit 210 uses the vehicle location information acquired by the location acquisition unit 200 and the user location information 300 to determine the location of the user displayed in the vehicle outside image data 320 captured by the external camera 26 of the vehicle. calculate. For example, similar to the vehicle location calculation unit 110, the user recognition is performed by collating with the map data 340 by perspective projection transformation, affine transformation, or the like and using DNN or the like. Here, when there are a large number of persons in the outside-vehicle image data 320, the user location calculation unit 210 may recognize the person whose position is the closest as the user. At this time, the user location calculation unit 210 can acquire user information from the user terminal 1b and select a person who is most likely to be a user based on gender or age.

The image presentation unit 220 is a functional unit similar to the image presentation unit 120.
For example, the image presentation unit 220 presents the user location calculated by the user location calculation unit 210 on the image data 320 outside the vehicle in AR. In addition, the image presentation unit 220 acquires user image data 310 from the user terminal 1b, acquires the vehicle location calculated by the vehicle location calculation unit 110 of the user terminal 1b from the user terminal 1a, and presents it. It is also possible to do.

  The vehicle outside image data 320 is still image data or moving image data captured by the external camera 26 of the automobile terminal 2b.

  The in-vehicle image data 330 is still image data or moving image data captured by the in-vehicle camera 27 of the automobile terminal 2b.

  The appearance image data 350 is image data obtained by capturing the appearance of the vehicle. The image data may include data having different formats and sizes, such as recognition data used when the vehicle location calculation unit 110 recognizes an image, thumbnail data to be displayed on the user's map, and the like.

[Vehicle user presentation processing by information processing system Xb]
Next, a vehicle user presentation process by the information processing system Xb according to the second embodiment of the present invention will be described with reference to FIGS.
In the vehicle user presentation process of the present embodiment, in addition to the same process as the vehicle presentation process of the first embodiment, the location of the user is presented in the outside-vehicle image data 320 captured from the automobile terminal 2b. In addition, in-vehicle image data 330 is also presented. At this time, the orientation of the camera of the automobile terminal 2b and the like are controlled according to a user instruction. This makes it easier for both the driver and the user of the vehicle to call and recognize each other.
In the vehicle presentation process of the present embodiment, the control unit 10b of the user terminal 1b and the control unit 20b of the automobile terminal 2b are mainly configured to store programs such as applications stored in the storage unit 11b and the storage unit 21b, respectively. To work with hardware resources.
Below, with reference to the flowchart of FIG. 7, the detail of the vehicle user presentation process of this embodiment is demonstrated for every step.

(Step S111)
First, the position acquisition unit 100 of the user terminal 1a performs a position acquisition process.
This process is performed in the same manner as step S101 in FIG.
In addition to this, in the present embodiment, the position acquisition unit 100 transmits the position information 300 acquired from the sensor group 15 to the automobile terminal 2b.

(Step S211)
Here, the position acquisition unit 200 of the automobile terminal 2b performs a position acquisition process.
The position acquisition unit 200 acquires the position information 302 from the sensor group 25 during traveling and stores it in the storage unit 21b.
The position acquisition unit 200 transmits the position information 302 to the user terminal 1b in a state where the application is activated and communication between the automobile terminal 2 and the user terminal 1 becomes possible.
In addition, in the present embodiment, the position acquisition unit 200 also acquires the position information 300 of the user terminal 1a and stores it in the storage unit 21b.

(Step S112)
Next, the camera control unit 130 of the user terminal 1a controls the external camera 26 and / or the in-vehicle camera 27 of the vehicle.
A screen example 510 in FIG. 8A shows an example of image data 320 outside the vehicle of the external camera 26 displayed on the display unit 13 of the user terminal 1b and / or the display unit 23 of the automobile terminal 2b. According to this example, the user can press and instruct each arrow of the button 720 displayed superimposed on the outside-vehicle image data 320 and move the electric pan head of the holder in the direction of the arrow. Further, the zoom magnification of the external camera 26 or the in-vehicle camera 27 can be changed by the zoom mark of the button 730.

(Step S113)
Next, the vehicle location calculation unit 110 performs a vehicle location calculation process.
This process is also performed in the same manner as step S102 in FIG.
In addition to this, in this embodiment, the vehicle location calculation unit 110 acquires the vehicle appearance image data 350 from the server 3. Then, the vehicle location calculation unit 110 uses the appearance image data 350 to recognize the vehicle by image recognition using DNN or the like, and calculates the location of the vehicle.

(Step S213)
Here, the user location calculation unit 210 of the automobile terminal 2b performs a user location calculation process.
The user location calculation unit 210 recognizes the user displayed in the outside-vehicle image data 320 captured by the external camera 26 and calculates the location displayed by the user.

(Steps S114 and S214)
The image presentation unit 120 of the user terminal 1a and the image presentation unit 220 of the automobile terminal 2b perform image transmission / reception processing.
Here, the image presentation unit 120 transmits the user image data 310 in which the location of the vehicle has been presented to the automobile terminal 2b. Further, the image presentation unit 220 transmits the vehicle outside image data 320 and the vehicle interior image data 330 on which the user's location has been presented to the user terminal 1b. Transmission / reception of these image data may be performed by PtP (Peer to Peer) or via the server 3. Each unit receives these image data and stores them in the storage units 11b and 21b, respectively.

(Steps S115 and S215)
The image presentation unit 120 of the user terminal 1a and the image presentation unit 220 of the automobile terminal 2b each perform an image presentation process.
In addition to the processing similar to the image presentation processing in step S103 of FIG. 4, the image presentation unit 120 switches or overlaps the images of the camera 16, the external camera 26, and the in-vehicle camera 27 on the display unit 13. It can be displayed.
The image presentation unit 220 of the automobile terminal 2b can also switch these images and display them on the display unit 23.

The screen example 510 in FIG. 8A illustrates an example in which the user location 710 that is the calculated user location is displayed in the AR display in the image data 320 outside the vehicle.
In the screen example 512 of FIG. 8B, in-vehicle image data 330 is displayed in addition to the images of the screen example 500 of FIG. 5 of the first embodiment. Further, on the map data 340, a thumbnail image of the appearance image data 350 is displayed in a balloon format in the vicinity of the pointer P indicating the position of the vehicle.
These screen examples are displayed on the display unit 13 and / or the display unit 23 in real time (real time).
Thus, the vehicle user presentation process according to the second embodiment of the present invention is completed.

With the configuration described above, the following effects can be obtained.
In the conventional calling service, for example, a vehicle driver may not know who is a customer (user) where there are many people. In this case, even if the vehicle arrives at the meeting place, the user cannot be recognized, which may cause a problem.
On the other hand, the automobile terminal 2b of the information processing system Xb according to the present embodiment uses the vehicle position information 302 acquired by the position acquisition unit 200 and the user position information 300 to acquire the outside of the vehicle acquired from the external camera 26 of the vehicle. A user location calculation unit 210 that calculates the location of the user displayed in the image data 320 is further provided, and the image presentation unit 120 and / or the image presentation unit 220 determine the location of the user calculated by the user location calculation unit 210. It is an information processing apparatus that is presented on the image data 320 outside the vehicle.

With this configuration, the driver can recognize the location of the user from the surrounding images during traveling. For this reason, it becomes easy for the driver to grasp the user, and the driver can easily reach the meeting place.
In addition, the user terminal 1b of the present embodiment may also be able to display the vehicle exterior image data 320 on the image presentation unit 120. For this reason, the user can also know which side the vehicle is traveling in real time. That is, by knowing the current state of the vehicle, it is possible to easily grasp where the called vehicle has arrived.
In this way, both the user and the driver can recognize each other, thereby reducing the possibility of waiting problems.

In the user terminal 1b of the present embodiment, the vehicle location calculation unit 110 acquires the vehicle appearance image data 350, and recognizes the vehicle by the appearance image data 350, thereby calculating the location of the vehicle. It is characterized by being.
With this configuration, it is possible to reliably recognize the image of the called vehicle from the user image data 310 and calculate the location of the vehicle. That is, the recognition rate can be remarkably increased by image recognition using actual vehicle data, rather than DNN that categorizes a general automobile. For this reason, the user can surely recognize the called vehicle, and trouble can be prevented.

In the user terminal 1b of the present embodiment, the image presentation unit 120 displays the position of the vehicle with a pointer on the acquired map data 340 and also displays the appearance image data 350 in the vicinity of the pointer. It is characterized by being.
By configuring in this way, the user can easily discriminate the called vehicle on the map, and further, it becomes easier to find a vehicle that is nearby. In particular, when the called vehicle is characteristic in appearance such as a vehicle type or paint, it can be easily discriminated.

Conventionally, in the service of calling vehicles, the user cannot know the status of the called vehicles. Users sometimes worried that the car they actually called was in a different atmosphere from the vehicle image.
On the other hand, the user terminal 1b of the present embodiment is characterized in that the image presentation unit 120 is an information processing apparatus that also presents in-vehicle image data 330 acquired from the in-vehicle camera 27 of the vehicle.
By configuring in this way, it is possible to confirm the status of the called vehicle on the app in real time. That is, not only the situation outside the vehicle but also the situation inside the vehicle can be confirmed. This makes it possible to know the situation inside the vehicle. As a result, it is possible to give the user a sense of security more than pre-registered drivers and vehicle images, word of mouth, ratings, and the like.

In addition, it was important to know the situation inside the car in autonomous driving community cars and sharing cars. This is because when there are a large number of people in the vehicle, it is necessary to request a call service separately.
Some users also want to share a taxi, hire, etc. with several people and use it cheaply. However, when the number of carpooling increases, it sometimes feels annoying, and the user sometimes feels annoyed whether to use a separate calling service.
On the other hand, since it is possible to know the situation inside the vehicle, the user can determine how much a person is riding and whether there is a possibility of getting on. For this reason, it becomes possible for the user to easily select whether or not to call separately, and the usability of the user can be improved.

The user terminal 1b of the present embodiment is an information processing apparatus that further includes a camera control unit 130 that controls the external camera 26 and / or the in-vehicle camera 27 of the vehicle.
By configuring in this way, the user can operate from the application of the user terminal 1b as to where the car terminal 2b takes an image. That is, the user can remotely operate the camera of the automobile terminal 2b, and can freely view the image of the vehicle while remotely operating it.
As a result, it is possible to browse a place that the user wants to show to the driver and a place that the user wants to browse. Therefore, the driver can be surely shown where the user is waiting in a parking lot position, a place where there are many pick-ups, and the like. In addition, it is possible for the user himself / herself to find a person wearing easily understandable clothes such as his wife or a child, for example, and to acquire this image so that the vehicle can be easily identified.

In addition, in the information processing system Xb of the present embodiment, user image data 310 captured by the user terminal 1b can be transmitted to the automobile terminal 2b.
For this reason, it becomes possible to transmit the image of the position where the user is present, the current situation, etc. to the driver, and the driver can save time and labor for searching for the user. For this reason, service can be improved.

In the embodiment described above, the external camera 26 and the in-vehicle camera 27 are controlled based on a user instruction.
On the other hand, when the vehicle is stopped, the driver of the car may drive the holder H via the application of the car terminal 2b to change the direction and zoom magnification of the external camera 26 and the in-car camera 27.
Furthermore, the vehicle outside image data 320, the vehicle interior image data 330, and the user image data 310 may not only be transmitted to each terminal in real time but also be storable for a specific period at each terminal. As a result, it is possible to browse images that are several seconds to several minutes old.
In addition, the in-vehicle image data 330 may include not only an image of the in-vehicle camera 27 but also an image of an external camera that allows an appearance image of the car to be seen.

<Third embodiment>
Next, an information processing system Xc according to a third embodiment of the present invention will be described with reference to FIG. Also in FIG. 9, the same code | symbol is attached | subjected to the structure similar to the above-mentioned 1st embodiment and 2nd embodiment.
In the first embodiment described above, the location of the vehicle is presented in the user image data 310. In the second embodiment, in addition to the location of the vehicle, the location of the user is presented in the image data 320 outside the vehicle.
On the other hand, you may comprise so that a user's location may be shown only by the motor vehicle terminal 2c like the information processing system Xc. In the example of FIG. 9, the control unit 20 c includes a position acquisition unit 200, a user location calculation unit 210, and an image presentation unit 220. The storage unit 21c stores position information 300 and image data 320 outside the vehicle. The user terminal 1 c includes position information 302.

That is, the automobile terminal 2c of the present embodiment includes the position acquisition unit 200 that acquires the position information 302 of the vehicle that is calling to the meeting place and the position information 300 of the user, and the position of the vehicle and the user that are acquired by the position acquisition unit 100. Based on the information 300, the user location calculation unit 210 that calculates the location of the user displayed in the vehicle external image data acquired from the external camera 26 of the vehicle, and the location of the user calculated by the user location calculation unit 210 It is an information processing apparatus provided with the image presentation part 120 shown on external image data, It is characterized by the above-mentioned.
With this configuration, it is possible to make it easier for the driver of the automobile to recognize the user, as in the second embodiment described above. In this case, since it is not necessary to confirm the car that the user has called, the burden on the user can be reduced.

<Fourth embodiment>
Next, an information processing system Xd according to a fourth embodiment of the present invention will be described with reference to FIG. Also in FIG. 10, the same code | symbol is attached | subjected to the structure similar to the above-mentioned 1st thru | or 3rd embodiment.
In the first to fourth embodiments described above, an example in which the server 3 includes the map data 340 has been described. However, in the present embodiment, for example, the same processing as in the second embodiment described above is performed only by the user terminal 1d and the automobile terminal 2d as in PtP.
In the example of FIG. 10, the control unit 10d and the storage unit 11d of the user terminal 1d and the control unit 20d of the automobile terminal 2d have the same configuration as the control unit 10b, the storage unit 11b, and the control unit 20b of the second embodiment. is there. On the other hand, the storage unit 21d of the automobile terminal 2d stores map data 340 in addition to the data of the storage unit 21b of the second embodiment.

With this configuration, the user and the driver can confirm each other without using the server 3. In addition, the car terminal 2d stores the map data 340 for the dedicated navigation, and collates based on the map data 340, thereby preventing a shift of the map data 340 between the user terminal 1d and the car terminal 2d, and the location of the user and the vehicle. It becomes easy to calculate.
In this example, the map data 340 is described as being stored in the automobile terminal 2d, but may be the user terminal 1d. In addition, a navigation device that can be connected to the automobile terminal 2b is provided separately, and the map data 340 may be acquired from the navigation device. Furthermore, you may make it recognize a vehicle or a user's location directly, without using the map data 340. FIG.

<Fifth embodiment>
Next, an information processing system Xe according to a fifth embodiment of the present invention will be described with reference to FIG. Also in FIG. 11, the same reference numerals are given to the same configurations as those in the first to fourth embodiments described above.
In the above-described first to fourth embodiments, the vehicle or user location is calculated from the image data at each terminal. However, like the information processing system Xe, the server 3e may calculate these points. In this case, for example, the server 3e includes a user location calculation unit 210 and a vehicle location calculation unit 110 in the control unit 30e. In addition to the map data 340, the storage unit 31e acquires the position information 300 and the user image data 310 from the user terminal 1e and stores them. Further, the position information 302, the outside image data 320, and the in-house image data are acquired from the automobile terminal 2e and stored in the storage unit 31d. Then, each image data is distributed from the server 3e to each terminal by streaming or the like.
In the case of this example, the control unit 10e of the user terminal 1e includes a position acquisition unit 100, an image presentation unit 120, and a camera control unit 130. The control unit 20e of the automobile terminal 2e includes a position acquisition unit 200 and an image presentation unit 220.
By comprising in this way, the processing burden of the user terminal 1e and the motor vehicle terminal 2e can be reduced. Further, by accumulating each image data in the server 3e and recognizing each location, it is possible to efficiently learn and to improve the accuracy of collation with the map data 340 and each recognition.

Furthermore, a configuration in which the vehicle outside image data 320 and the like are stored in the storage unit 31e of the server 3e for a specific period is possible.
Thereby, in the user terminal 1e, it is possible to search where the vehicle was running along the time series, thereby making it easier to find the vehicle.

In addition, according to FIG. 12, by accumulating the outside image data 320 of each vehicle using the vehicle calling service, the user can view the outside image data 320 collated with the map data 340. Thus, the user can set the destination.
For example, as shown in a screen example 520 in FIG. 12A, the user sets a destination with a pin on the map. In this case, as shown in the screen example 530 of FIG. 12B, the corresponding outside-vehicle image data 320 can be displayed. Thereby, in the user terminal 1e, a destination can be set by an image of a location where the vehicle has actually moved.
Furthermore, the user terminal 1e can also perform destination setting “I want to go to that neighborhood” from the collation result between the map data 340 and the landmark of the user image data 310.

With this configuration, even when the user does not know the arrival point, the destination can be easily set. For this reason, it is possible to cope with a case where it is difficult to explain an address at a travel destination or a place visited for the first time. That is, it is possible to easily search for a destination that the user remembers vaguely as “that neighborhood”. In addition, the user can reliably tell the driver where to go by showing the image.
By accumulating the outside image data 320 collated with the map data 340 in the server 3e, it is possible to view the destination image data after the user actually gets on the vehicle. For this reason, even when the user does not know the arrival place only by the address, the user can explain to the driver that he / she wants to go to “that neighborhood” by browsing the destination image.

  In addition, a 360 ° image can be created from the stored outside-vehicle image data 320 and a VR (Virtual Reality) moving image can be distributed in real time.

Furthermore, a configuration in which in-vehicle image data 330 and the like are stored in the storage unit 31e of the server 3e for a specific period is possible.
Thereby, it is possible for the user to easily determine from the user terminal 1e whether or not to use the calling service based on the past in-vehicle situation and how many people are on the board or whether there is a possibility of getting on.

<Other embodiments>
In the above-described embodiment, only the location of the user calling the vehicle is displayed in the outside image data 320.
However, you may make it display all the locations of the user who is starting the application in the image data 320 outside a vehicle. In addition, on the map data 340, the user data used in the past, the flow and number of people by mobile radio waves, etc. may be shown.
With this configuration, it is possible to determine whether or not the user or driver is the most suitable meeting place, and in some cases, the meeting place can be changed. This can prevent troubles from occurring.

In the above-described embodiment, the user image data 310 is described so as to present only the location of the vehicle.
However, even after the user gets on the vehicle, it is also possible to display the introduction of the historical building, the history and news of the town, the weather, and the like while collating with the map data 340 during the movement. At this time, automatic translation or the like may be performed, or information for AR display may be selected by estimating the user's emotion.
In this way, it is possible to generate impression by matching the knowledge with the experience on the spot. Users may also be idle on the move and do not want to talk to the driver. Even in this case, the user can be provided with convenience by viewing the AR display.

In the first to fifth embodiments of the present invention described above, the features of the present invention have been described focusing on the main configuration. However, the combination of the structures of the functional units in these embodiments is arbitrary. That is, any one of the configurations of the first to fifth embodiments, a configuration of any combination, or a configuration including all functional units may be used.
Further, the user terminal 1 may further include a functional unit that is not described in the first to fifth embodiments.

  In addition, the configuration and operation of the above embodiment are examples, and it is needless to say that the configuration and operation can be appropriately changed and executed without departing from the gist of the present invention.

1, 1a, 1b, 1c, 1d, 1e user terminal, 2, 2a, 2b, 2c, 2d, 2e automobile terminal, 3, 3e server, 4 network, 10, 10a, 10b, 10d, 10e, 20, 20b, 20c, 20d, 20e, 30, 30e control unit, 11, 11a, 11b, 11d, 21, 21b, 21c, 21d, 31, 31e storage unit, 12, 22 input unit, 13, 23 display unit, 14, 24, 34 connection unit, 15, 25 sensor group, 16 camera, 26 external camera, 27 in-vehicle camera, 18, 28 voice input / output unit, 100, 200 position acquisition unit, 110 vehicle location calculation unit, 120, 220 image presentation unit , 130 Camera control unit, 210 User location calculation unit, 300, 302 Position information, 310 User image data, 320 Outside vehicle image Data, 330 in-vehicle image data, 340 map data, 350 appearance image data, 500, 510, 512, 520, 530 screen example, 610 display column, 700 vehicle location, 710 user location, 720, 730 button, D dashboard, H holder, U, P pointer, X, Xa, Xb, Xc, Xd, Xe Information processing system

Claims (9)

A position acquisition unit (100) for acquiring the position information (302) of the vehicle calling to the meeting place and the position information (300) of the user;
The location of the vehicle displayed in the user image data (310) captured by the user's camera (16) is calculated based on the vehicle location information and the user location information acquired by the location acquisition unit. A vehicle location calculator (110);
An information processing apparatus comprising: an image presentation unit (120) that presents the location of the vehicle calculated by the vehicle location calculation unit on the user image data. The location of the user displayed in the outside image data (320) captured by the external camera (26) of the vehicle based on the position information of the vehicle and the position information of the user acquired by the position acquisition unit. A user location calculation unit (210) for calculating,
The image presentation unit
The information processing apparatus according to claim 1, wherein the user location calculated by the user location calculation unit is presented on the image data outside the vehicle. The vehicle location calculator
The information processing apparatus according to claim 1 or 2, wherein the vehicle location is calculated by acquiring the vehicle appearance image data (350) and recognizing the vehicle from the appearance image data. The image presentation unit
The information processing apparatus according to claim 3, wherein the position of the vehicle is displayed with a pointer on the acquired map data (340), and the appearance image data is displayed in the vicinity of the pointer. The image presentation unit
The information processing apparatus according to any one of claims 1 to 4, further presenting in-vehicle image data (330) captured by the in-vehicle camera (27) of the vehicle. The information processing apparatus according to any one of claims 2 to 5, further comprising a camera control unit (130) that controls the external camera and / or the in-vehicle camera of the vehicle. A position acquisition unit (100) for acquiring the position information (302) of the vehicle calling to the meeting place and the position information (300) of the user;
A user who calculates the location of the user displayed in the vehicle external image data (320) acquired from the external camera (26) of the vehicle based on the vehicle and user position information acquired by the position acquisition unit. A location calculation unit (210);
An information processing apparatus comprising: an image presentation unit (120) that presents the user location calculated by the user location calculation unit on the vehicle external image data. An information processing program executed by the information processing apparatus (1),
The position acquisition unit (100) is allowed to acquire the position information (320) of the vehicle calling to the meeting place and the position information (300) of the user,
Displayed in the vehicle location calculation unit (110) in the user image data (310) captured by the user's camera (16) based on the vehicle location information and the user location information acquired by the location acquisition unit. To calculate the location of the vehicle to be
A program for causing an image presentation unit (120) to present the vehicle location calculated by the vehicle location calculation unit on the user image data. An information processing method executed by the information processing apparatus (1),
Obtain the location information (302) and the location information (300) of the vehicle calling to the meeting place,
Based on the acquired position information of the vehicle and the position information of the user, the location of the vehicle displayed in the user image data (310) captured by the user's camera (16) is calculated,
An information processing method for presenting the calculated location of the vehicle on the user image data.

Images