WO2023084810A1 - Information processing apparatus, information processing method, and information processing program - Google Patents

Abstract

An information processing apparatus 10 that is communicably connected to a time-and-space database 20 for processing data on time and space in real time and provides information to a passenger of a moving body includes: a collection unit 11 that collects, from a plurality of moving bodies, pieces of image data obtained by imaging performed by the plurality of moving bodies at a plurality of locations at a plurality of dates and times, pieces of time data at the imaging, and pieces of position data of the moving bodies during the imaging and transmits the collected plurality of pieces of image data, the plurality of pieces of time data, and the plurality of pieces of image data to the time-and-space database; a reception unit 12 that receives, from a specific moving body, the position data of the specific moving body; a processing unit 13 that searches an image data group of the time-and-space database for an object in the image data corresponding to the position data of the specific moving body; and a transmission unit 14 that transmits, to an augmented reality display apparatus of the specific moving body, object display information to allow the object to be displayed in an augmented reality by being superimposed on a scene of the specific moving body at the same display position as a display position where the object has been displayed in the past.

Description

Information processing device, information processing method, and information processing program

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, an augmented reality display technology called AR (Augmented Reality) display that superimposes the virtual world on the real world is known (see Patent Document 1 and Non-Patent Document 1).

U.S. Pat. No. 8,687,021 JP 2020-13539 A

However, conventional AR displays have the problem of being uninteresting because they simply superimpose virtual objects in the real environment.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique capable of providing attractive information to passengers of mobile objects.

An information processing apparatus according to one aspect of the present invention is an information processing apparatus that is communicably connected to a spatio-temporal database that processes spatio-temporal data in real time and provides information to a passenger of a mobile object. A plurality of collected image data obtained by collecting from a mobile object image data taken by the plurality of mobile objects at a plurality of locations at a plurality of dates and times, time data at the time of shooting, and position data of each mobile object at the time of shooting. a collection unit that transmits a plurality of time data and a plurality of location data to the spatio-temporal database; a reception unit that receives location data of the predetermined mobile unit from a predetermined mobile unit; and a position of the predetermined mobile unit. a processing unit for retrieving an object in image data corresponding to the data from the image data group of the spatio-temporal database; and a transmitting unit configured to transmit object display information for displaying augmented reality superimposed on the scene of the moving object to the augmented reality display device of the predetermined moving object.

An information processing method according to one aspect of the present invention is an information processing method that is communicably connected to a spatio-temporal database that processes spatio-temporal data in real time and provides information to a passenger of a mobile object. A device collects, from a plurality of mobile bodies, image data taken by the plurality of mobile bodies at a plurality of locations at a plurality of dates and times, time data at the time of shooting, and position data of each mobile body at the time of shooting. a step of transmitting a plurality of image data, a plurality of time data, and a plurality of position data to the spatio-temporal database; a step of receiving position data of the predetermined moving body from a predetermined moving body; retrieving an object in image data corresponding to body position data from the image data group of the spatio-temporal database; and transmitting object display information to be superimposed on the scene of the predetermined moving body and displayed in augmented reality to the augmented reality display device of the predetermined moving body.

An information processing program according to one aspect of the present invention causes a computer in a communication network to function as the information processing device.

An information processing program according to one aspect of the present invention causes a computer in a mobile object to function as the information processing device.

According to the present invention, it is possible to provide technology capable of providing attractive information to passengers of mobile objects.

FIG. 1 is a diagram illustrating a configuration example of an information processing system. FIG. 2 is a diagram showing a processing flow of data collection operation. FIG. 3 is a diagram showing a processing flow of operations such as AR display of an object. FIG. 4 is a diagram illustrating a hardware configuration example of an information processing apparatus.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are denoted by the same reference numerals, and the description thereof is omitted.

[Summary of Invention]
An object of the present invention is to provide attractive information to a passenger of a mobile object. In order to achieve this object, the present invention performs an AR display in which the past real world is superimposed on the current real world, instead of the conventional AR display in which the virtual world is superimposed on the real world. In other words, the present invention gives the feeling that real value is added to the scene from the moving object by AR displaying not the virtual object but the real object that actually existed in the past.

However, in order to perform the AR display of the present invention, it is necessary to process a huge amount of past real-world spatio-temporal data generated at multiple locations on multiple dates and times at ultra-high speed. Therefore, the present invention provides a spatio-temporal database that can store a large amount of spatio-temporal data all at once, and that can search and analyze past spatio-temporal data and objects that occurred at a given place on a given date and time in real time. (See Patent Document 2 and Non-Patent Document 2).

In other words, the present invention uses a real-time spatio-temporal database for processing such as searches to retrieve information about actual objects that existed in a predetermined location in the past and events that occurred in the past in a predetermined location. An AR display superimposed on the current environment of the moving object located at the same place as the predetermined place is performed.

By performing the AR display of the present invention, it is possible to provide interesting, timeless and attractive information. For example, it is possible to allow a passenger of a mobile object to experience an advanced travel experience. Specifically, by displaying in AR the bronze statues of historical figures that were installed along the road on which the car travels, it is possible to provide social education to the children of the passengers. The AR display of past firework displays at the place where the vehicle is running can make the driver feel like he or she is enjoying a trip. In addition, the present invention can also improve driving safety, such as reducing driver distraction and reducing boredom experienced while driving on lightly traffic roads.

[Configuration example of information processing system]
FIG. 1 is a diagram showing a configuration example of an information processing system 1 according to this embodiment. This information processing system 1 includes an information processing device 10, a spatio-temporal database 20, and a plurality of automobiles 30, which are interconnected via a communication network so as to be able to communicate with each other.

The information processing device 10 is an information processing device that provides attractive information to passengers of the automobile 30 (driver, fellow passenger, operator, passenger, etc.). The information processing device 10 may be configured by a computer (server device, etc.) in the communication network, or may be configured by a computer (in-vehicle equipment, etc.) in the vehicle 30 .

The information processing device 10 includes a collection unit 11, a reception unit 12, a processing unit 13, and a transmission unit 14, as shown in FIG.

The collection unit 11 collects, from a plurality of automobiles 30, image data photographed by the plurality of automobiles 30 at a plurality of locations on a plurality of dates and times, recorded audio data, various types of measurement data measured, at the time of photographing, recording It has a function of collecting position data and time data of each vehicle 30 at the time of measurement.

The collection unit 11 transmits a plurality of image data, a plurality of audio data, a plurality of various measurement data, a plurality of position data, and a plurality of time data to the spatio-temporal database 20, and collects the image data, the audio data, and the various measurement data. , position data, and time data are associated with each other and stored in the spatio-temporal database 20 .

The collection unit 11 has a function of adding an image data tag indicating the image content of the image data to the image data. Similarly, the collection unit 11 has a function of adding an audio data tag indicating the audio content of the audio data to the audio data.

The receiving unit 12 has a function of receiving position data and time data of a predetermined automobile 30' from the predetermined automobile 30'.

The receiving unit 12 has a function of receiving profile information of passengers of the predetermined automobile 30' from the predetermined automobile 30'.

The processing unit 13 has a function of searching the image data group and the audio data group of the spatio-temporal database 20 for an object and audio data in the image data corresponding to the position data of the predetermined automobile 30'. Similarly, the processing unit 13 selects an object in the image data corresponding to both the position data and the time data of the predetermined vehicle 30' and the audio data in the image data group and the audio data group in the spatio-temporal database 20. Equipped with a function to search from

The processing unit 13 has a function of searching for an object in image data and audio data corresponding to the position data of a predetermined automobile 30' based on the image data tag and the audio data tag. Similarly, the processing unit 13 has a function of retrieving an object in image data and audio data corresponding to both position data and time data of a predetermined automobile 30' based on an image data tag and an audio data tag. Prepare.

The processing unit 13 has a function of retrieving objects and audio data in image data related to profile information of passengers of a predetermined automobile 30'.

The processing unit 13 has a function of searching for objects in image data and audio data that affect driving of a predetermined automobile 30'.

The transmitting unit 14 transmits the object in the image data searched by the processing unit 13 to the scene of the predetermined automobile 30' at the same display position as the display position where the object was displayed in the past in the image data of the object. It has a function of transmitting object display information to be superimposed and displayed in AR (augmented reality display) to an output device 35 of a predetermined automobile 30'. Similarly, the transmission unit 14 has a function of transmitting audio output information for outputting the audio data retrieved by the processing unit 13 to the output device 35 of the predetermined automobile 30'.

The spatio-temporal database 20 accumulates spatio-temporal data such as image data transmitted simultaneously by a plurality of automobiles 30, while storing spatio-temporal data corresponding to a predetermined time at a predetermined place and a predetermined time at a predetermined place. It is an ultra-high-speed spatio-temporal data management technology that includes high-speed spatio-temporal data search technology that searches and analyzes dynamic and static objects located at the time in real time. It is a database technology that enables search and classification analysis.

This spatio-temporal database 20 is disclosed in Patent Document 2 and Non-Patent Document 2. For example, as disclosed in Non-Patent Document 2, the spatio-temporal database 20 uses the spatio-temporal code as a key of a distributed key-value store to improve the efficiency of multidimensional information retrieval, etc. It enables fast retrieval of data and objects.

A plurality of cars 30 are connected cars connected to a communication network. A plurality of automobiles 30 each include a camera 31 , a position sensor 32 , a clock 33 , an input device 34 and an output device 35 . The automobile 30 may further include a speed sensor, an acceleration sensor, a steering wheel angle sensor, a line-of-sight position measurement sensor, and the like.

The camera 31 is an imaging device with a sound pickup function installed in the automobile 30 . The camera 31 has a function of photographing the front, rear, left, right, bottom, top, and oblique directions of the automobile 30 and picking up sounds generated around the automobile 30 .

The position sensor 32 has a function of acquiring position data indicating the geographical two-dimensional coordinate position of the automobile 30 from the global positioning system. The position sensor 32 is, for example, a positioning function built in a navigation system.

The clock 33 has a function of measuring the time data of the automobile 30. The clock 33 may measure the time itself, or may refer to the time via a communication network using a network time protocol. The clock 33 may be omitted if the camera 31 or the position sensor 32 has a built-in time measurement management function.

The input device 34 is a computer installed in the automobile 30. The input device 34 is, for example, a tablet terminal, an in-vehicle device, or the like.

The output device 35 is an AR display device capable of AR-displaying objects and an audio output device capable of outputting audio data. The output device 35 is, for example, a computer that outputs video and audio from the camera 31, a navigation system, a tablet terminal, or the like. The output device 35 may be, for example, a projector that outputs an image to the front window of the automobile 30 and outputs sound, a head-mounted display, a head-up display, or the like. That is, the output device 35 is a conventionally known general screen or display, VR goggles or spectacles type display device, a window installed in a moving object, a projector, or the like.

[Example of operation of information processing device]
Next, an operation example of the information processing device 10 will be described.

[Data collection operation]
First, the operation of the information processing device 10 collecting image data and the like from a plurality of automobiles 30 will be described. FIG. 2 is a diagram showing a processing flow of data collection operation.

Step S101;
The collection unit 11 constantly collects image data and the like regarding actual objects existing around the plurality of automobiles 30 that are running or paused, and actual events occurring around the plurality of automobiles 30 . Specifically, the collection unit 11 collects images captured by the cameras 31 of the automobiles 30 at a plurality of locations on a plurality of dates and times from the plurality of automobiles 30 that are running or at a temporary stop. Data, recorded voice data, position data of each car 30 obtained by the position sensor 32, and time data of each car 30 measured by the clock 33 are collected all the time.

For example, the collection unit 11 collects image data about cars, roads, signs, people, animals, trees, rivers, lawns, houses, weather, the sun, clouds, rainbows, lightning, and other potentially interesting human and natural events. are always collected. When the vehicle 30 is also equipped with a speed sensor, an acceleration sensor, a steering wheel angle sensor, a line-of-sight position measurement sensor, etc., the collection unit 11 constantly collects various measurement data measured by each sensor at the same time.

Step S102;
Next, the collection unit 11 uses the image data, sound data, various measurement data, position data, and time data collected from the plurality of automobiles 30 to determine the category to which the image data and sound data belong, An image data tag indicating the category is added to the image data, and an audio data tag indicating the category is added to the audio data.

Regarding the means for determining categories, for example, a machine learning model for classification tasks that has been built in advance to determine the categories to which image data and audio data belong is used. The collection unit 11 inputs image data or audio data to the machine learning model, and determines the inference result of the category output from the machine learning model as the category to which it belongs. It should be noted that the category classification accuracy of the machine learning model can be improved by inputting a plurality of types of image data and audio data to the machine learning model and repeatedly learning the model.

Step S103;
Finally, the collection unit 11 transmits a plurality of image data, a plurality of audio data, a plurality of various measurement data, a plurality of position data, a plurality of time data, and a plurality of tag data to the spatio-temporal database 20, and The data are associated with each other and stored in the spatio-temporal database 20 .

The information processing apparatus 10 executes steps S101 to S103 every hour, every day, every month, and every year. As a result, the spatio-temporal database 20 accumulates a huge amount of image data and the like regarding things that existed at arbitrary dates and times in various locations around the world and events that occurred at that time.

The spatio-temporal database 20 uses position data, time data, image data tags, and audio data tags as search keys, and stores image data, audio data, various measurement data, and tag data so as to be searchable with the search keys. are stored in association with each other. Due to real-time search and analysis of spatio-temporal data and objects possessed by the spatio-temporal database 20, the information processing apparatus 10 can search desired image data and audio data in real time at a later date.

[Category tag example]
An example of the image data tag and the audio data tag of the category described in step S102 will be described.

The collection unit 11 assigns category tags related to "seasons". The collection unit 11 collects the colors of flowers, the types of flowers, the colors of mountains, whether there is snow in the image data, the chirping of cicadas in the audio data, and the chirping of cicadas in the audio data, so that the season of the driving place of the automobile 30 can be determined at a later date. , time data, etc., the season of the image data and audio data is determined, and tags such as spring, summer, autumn, winter, early winter, etc. are added.

The collection unit 11 assigns category tags related to "time". The collection unit 11 determines the time of the image data based on how the sunset and sunrise appear in the image data, how the full moon appears in the case of nighttime, and the like, and attaches a tag for that time. The collection unit 11 may add the time data collected at the same time as the image data as it is.

The collection unit 11 assigns category tags related to "weather". The collection unit 11 determines whether the image data includes the sun, rain, many clouds, the color of the clouds, the sound of rain in the audio data, the sound of thunder, etc. , determines the weather in the image data or the audio data, and assigns tags such as sunny, cloudy, raining, snowing, and the like.

The collection unit 11 assigns category tags related to "landscape". The collection unit 11 extracts the scenery in the image data based on whether there is a rainbow, bridge, paddy field, mountain, sea, tower, etc. in the image data. Decide and assign a tag for that scenery.

The collection unit 11 assigns category tags related to "animals". The collection unit 11 identifies animals in the image data and the audio data based on whether the image data includes cows and birds, whether the audio data includes sheep barks, horse barks, and the like. Decide and give the animal a tag. The name of the tag may be a generic name such as dog or bird, or an individual name such as pug dog or beagle dog. An existing object recognition function can be used to determine whether an object such as a cow exists in the image data.

The collection unit 11 assigns category tags related to "vehicles". The collection unit 11 determines whether the image data contains a car or a train, whether the sound data contains a running sound of a train, or whether a running sound of a personal watercraft is included in the sound data. Determine the vehicles in the data and tag them. An existing image recognition function can be used to determine whether or not there is an image of an automobile or the like in the image data.

The collection unit 11 attaches category tags related to "people" and "human activities". The collection unit 11 attaches a rice harvesting tag when there is a state of rice harvesting in the image data. The collection unit 11 attaches a tag of a fireworks display when the image data includes the scenery of the fireworks display or fireworks in the sky. An existing image recognition function can also be used to determine whether or not an event such as rice harvesting is included in the image data.

The collection unit 11 assigns category tags related to "safe operation of automobiles". Based on the measurement data of the speed sensor, the acceleration sensor, and the steering wheel angle sensor, the collection unit 11 attaches a safe operation tag to the image data when the driving of the automobile 30 is as smooth as a manual operation. . On the other hand, when the acceleration or the steering wheel angle changes sharply in a short time, a dangerous operation tag is added to the image data.

The collection unit 11 assigns category tags related to "driver's interest". Based on the line-of-sight position measurement sensor, the collection unit 11 attaches a driver's interest tag to the image data when the line-of-sight of the driver is at a position other than the area in the traveling direction of the vehicle.

The category tags mentioned above are only examples. Other categories are also available to help determine which images and sounds should be selected for future display and sound output to occupants of vehicles passing through the location where the picture was taken. .

[AR Display Operation of Object]
Next, the information processing device 10 uses the past image data or the like accumulated in the spatio-temporal database 20 to AR-display a predetermined object to a passenger of a predetermined automobile 30', and displays a predetermined voice. The operation of outputting data will be explained. FIG. 3 is a diagram showing a processing flow of operations such as AR display of an object. The predetermined vehicle 30 ′ may be a vehicle 30 capable of transmitting image data and the like to the information processing device 10 or a vehicle not transmitting image data and the like to the information processing device 10 .

Step S201;
The receiving unit 12 receives the position data acquired by the position sensor 32 and the time data measured by the clock 33 of the predetermined automobile 30' currently running or stopped. When a passenger of a predetermined automobile 30' inputs his/her profile information to the input device 34 of the automobile 30', the receiving section 12 also receives the profile information. The profile information includes, for example, the age, sex, height, weight, hobby, job, personality, clothes, goals, etc. of the input person.

Step S202;
Next, when the processing unit 13 receives the profile information of the passenger, the processing unit 13 inputs the profile information into a machine learning model built in advance to determine the user's preferences from the profile, and from the machine learning model The output inference result is determined as the passenger's preference information.

In addition, the processing unit 13 inputs the position data and time data of the predetermined automobile 30' into a machine learning model constructed in advance for safe driving, and also inputs the passenger's profile information into the machine learning model. Then, the inference result output from the machine learning model is determined as the safe driving information regarding the safe driving of the passenger.

The above two types of machine learning models improve passenger preference information and safe driving information by inputting passenger profile information, vehicle location data and time data into the machine learning model and learning repeatedly. can do. In addition, since the taste information and safe driving information of passengers differ from person to person, the above two types of machine learning models may be customized for each passenger so as to learn the interests of each passenger. .

Step S203;
Next, the processing unit 13 accesses the spatio-temporal database 20, searches for an image data group that matches the position data of the predetermined vehicle 30', and selects an image that matches the passenger's preference information from the image data group. Search for image data with data tags attached. Similarly, the processing unit 13 also searches for audio data. For example, if the passenger likes birds, the processing unit 13 searches for images and sounds of birds that flew at a time zone different from the current date and time at a predetermined travel location of the automobile 30'.

In addition, the processing unit 13 accesses the spatio-temporal database 20, searches for an image data group that matches both the position data and the time data of the predetermined vehicle 30', and selects the passenger's preference from the image data group. Search for image data to which an image data tag that matches the information is added. Similarly, the processing unit 13 also searches for audio data. For example, when the passenger likes birds, the processing unit 13 searches for images and sounds of birds that flew at the same time zone as the current date and time at the predetermined travel location of the automobile 30'.

Similarly, the processing unit 13 also searches for image data and audio data that match the safe driving information of the passenger. For example, the processing unit 13 searches for an image of a vehicle that was in a traffic jam during a different time zone or the same time zone as the current date and time at a predetermined travel location of the vehicle 30', and for sounds during construction.

Note that the processing unit 13 may use all of the plurality of search methods described in step S203, or may use some of the search methods.

Step S204;
Next, the processing unit 13 selects one or more image data and audio data suitable for the passenger of the predetermined automobile 30' from among the searched plurality of image data and the plurality of audio data. After that, the processing unit 13 extracts the object and audio data included in the image data from the spatio-temporal database 20 . In the case of the above example, the processing unit 13 extracts, for example, a bird object and a bird's song from among the bird image, the bird's song, the image of the car in the traffic jam, and the sound during construction.

There are various choices for suitable image data and audio data.

For example, the processing unit 13 infers a plurality of image data that are interesting to the passenger based on the age of the passenger of the predetermined automobile 30', demographics obtained from the Internet, etc., and performs educational and entertainment activities according to the age. Extract target objects (walking horses, rainbows, etc.).

For example, the processing unit 13 extracts dynamic objects (automobiles, road construction scenes, etc.) that have traveled in the past under conditions that are very similar to the current weather and time at the travel location of the predetermined automobile 30'. do.

For example, when a predetermined vehicle 30' is traveling at night or in foggy conditions, the processing unit 13 may determine the state of traffic traffic including vehicles seen ahead at approximately the same traveling speed, especially the matching of the traveling speed. Extract the appearance of high traffic for a long period of time.

For example, the processing unit 13 receives the line-of-sight position data of the driver measured by the line-of-sight position measurement sensor of the predetermined automobile 30', and detects that the driver's line-of-sight position and line-of-sight movement are not fixed, and that the driver is tired or bored with driving. If so, select the image data that matches the tag of the category regarding "driver's interest".

For example, in principle, the processing unit 13 gives priority to the preferences of the passengers of the predetermined automobile 30', and exceptionally , give priority to safe driving information.

Step S205;
Next, the processing unit 13 determines the timing of outputting the object and audio data extracted from the spatio-temporal database 20 .

Step S206;
Finally, the transmission unit 14 provides object display information for AR-displaying the object extracted from the spatio-temporal database 20 at the above timing, and audio output information for outputting the audio data extracted from the spatio-temporal database 20 at the above timing. are output to the output device 35 of the predetermined automobile 30'.

After that, the output device 35 displays the object output from the information processing device 10 in an AR manner to the passenger of the predetermined automobile 30', and outputs the audio data as audio.

For example, if the output device 35 is a tablet terminal that outputs the video and audio from the camera 31, the output device 35 may display the image of the landscape taken by the camera 31 while traveling in the past at that location. The bird object that was flying is AR-displayed at the same position as the past flight position, and the bird's cry is output as voice.

For example, in the case where the output device 35 is a projector, the output device 35 reproduces a bird object that has flown in the past at the place of travel with respect to the actual scenery seen from the front window of the predetermined automobile 30'. An AR display is displayed on the front window at the same position as the flight position, and the bird's song is output from a speaker.

In this way, the objects displayed in AR are not virtual objects like conventional ones, but actual things that existed in the past or actual events that happened in the past. Therefore, it is possible to give the feeling of adding genuine value to the passengers of the predetermined automobile 30'. Passengers see realistic objects, so they can enjoy a more enjoyable trip, and they can drive safely without distracting their attention from the road while maintaining their awareness. As a result, attractive information can be provided to the passengers of the predetermined automobile 30'.

Note that when a plurality of objects are displayed on the output device 35 due to a plurality of selection results of the optimum image data and sound data, the passenger of the predetermined automobile 30' can perform a touch operation on the screen. , swipe operation, voice input, or other gestures may be used to select one of them. The selected object information is fed back to the user's preference machine learning model described in step S202, and learned as the passenger's preference.

[About object display information]
The object display information that the information processing device 10 transmits to the predetermined automobile 30' is to display the object to be displayed in AR at the same display position as the display position in which the object was displayed in the past in the image data of the object. This is display information for AR display superimposed on a scene seen from a predetermined automobile 30'.

That is, the information processing apparatus 10 maintains the display position where the object was displayed in the past image data without changing the display position of the object, that is, the attribute information of the object extracted from the past image data. (position, size, color, shape, vector, etc.) is used as is to generate object display information.

The output device 35 outputs the object to be displayed as it is to the display position in the image data in which the object was captured in the past. Since the given car 30' is driving in the same place where the object was photographed, the object can be seen against the surrounding background without processing to change the position or size of the object. are superimposed without a sense of incongruity.

In general, AR display has very low computational efficiency due to its extremely high computational cost. On the other hand, since automobiles tend to travel in the same lane, the actual scene seen from the automobile and the past image have the same angle of view. Therefore, the computational power required for the synchronization process to align them with each other is much less than before, and since complex calculations do not have to be performed, attractive information can be provided at high speed.

For example, consider a bird that flies from the branch of a particular tree out of sight of the car occupant. The photographed image data is image data photographed in the past by a vehicle running at a predetermined speed, at a predetermined position, and at a predetermined time period. The information processing apparatus 10 displays the bird object included in the image data on the output device 35 of the automobile currently traveling in the same place as the place. The AR-displayed bird object is displayed as seen from the car's camera. Therefore, in order to AR-display such a bird object, it is only necessary to place the bird object in a frame such as a tablet terminal or a front window at an appropriate timing. Since the bird object appears at the same angle and position, there is no need to calculate how it will look from different angles.

[Object description information]
Up to this point, the information processing apparatus 10 outputs object display information for AR-displaying an object and audio output information for outputting audio data of the object to a vehicle, and the vehicle can display a landscape image or a real scene. , the object is AR-displayed and the audio data of the object is output.

On the other hand, the information processing apparatus 10 not only superimposes and outputs the object itself, but also provides commentary information for explaining the object, that is, an actual object that existed in a predetermined place in the past, or an object that has existed in a predetermined place in the past. It is also possible to simultaneously output commentary information about an actual event that occurred in the past to the automobile, further display the commentary information of the object in AR, and further output audio data of the commentary information.

For example, the processing unit 13 of the information processing apparatus 10 converts the search key used when searching the spatio-temporal database 20 for image data and audio data in step S203 as commentary information about the object of the image data. to generate Specifically, the processing unit 13 extracts the object based on the position data, the time data, the image data tag, the audio data tag, etc. used for searching the object from the image data group of the spatio-temporal database. Generates descriptive information for .

Then, the transmitting unit 14 transmits object commentary information output information for AR display and audio output of the commentary information of the object to a predetermined automobile 30'. The object commentary information output information includes commentary information of the object, voice data of the commentary information, an AR display command of the commentary information, an output command of the voice data, and the like.

After that, the output device 35 of the predetermined automobile 30' displays the commentary information of the object included in the object commentary information output information output from the information processing device 10 to the passenger of the predetermined automobile 30' in AR, The voice data of the commentary information is output as voice.

As a result, in the predetermined automobile 30', in addition to the AR display/audio output of the object, the explanation information of the object is also AR displayed/audio output at the same time. For example, the state of the car that caused the traffic accident is displayed and voice-outputted in AR, and the commentary information "A car accident occurred at this place on the date and time of the year △ month ◇" is displayed and voiced in AR. output.

As a result, it is possible to provide more attractive information to car passengers by displaying the commentary information of the object in AR.

[Application example]
In the present embodiment, an automobile has been described as an example of a moving body. The mobile body of the present invention can be applied to various vehicles such as trains, monorails, aircraft, airships, boats, submarines, motorcycles, bicycles, and ropeways. For example, it can be applied to a maritime safety patrol boat, and an AR display can be made of the scene of past water accident rescues while moving on the water. Since there are no running sections such as roads on the water, there is a possibility that the position cannot be synchronized even if the object is output as it is to the display position in the image data where the object was shot in the past. It is effective when it is sufficient to be able to grasp at an approximate position.

[Effects of Embodiment]
According to the present embodiment, in the information processing device 10 that is communicably connected to the spatio-temporal database 20 that processes spatio-temporal data in real time and that provides information to passengers of a predetermined automobile 30', a plurality of image data photographed at a plurality of locations at a plurality of dates and times by the plurality of automobiles 30, the time data at the time of photographing, and the position data of each moving body at the time of photographing are collected from the plurality of collected images. data, a plurality of time data, and a plurality of position data to the spatio-temporal database 20; A processing unit 13 for searching the image data group of the spatio-temporal database 20 for an object in the image data corresponding to the position data of the automobile 30', A transmission unit 14 for transmitting object display information to be superimposed on a scene of a predetermined automobile 30' and displayed in AR at a display position to an output device 35 of the predetermined automobile 30'.

That is, the information processing apparatus 10 according to the present embodiment can reproduce an actual object that existed in a predetermined place in the past or an actual event that occurred in the past in a predetermined place. AR display is superimposed on the current environment of the moving object. For those who want to see things and events in the real world instead of virtual objects, it gives a sense of adding real value, so providing attractive information to mobile passengers can be done.

In particular, the information processing apparatus 10 according to the present embodiment transmits object display information for AR display in which the searched object is superimposed on the scene from the predetermined automobile 30' at the same display position as the display position of the object. Since there is no processing to render objects according to the passenger's position, it requires much less computing power than general AR display, and can provide attractive information to passengers of mobile vehicles at high speed. can.

Further, according to the present embodiment, the information processing apparatus 10 retrieves objects related to the profile of the passenger of the predetermined automobile 30', so that it is possible to present objects and events that remain in the passenger's taste and memory. can.

[others]
The invention is not limited to the above embodiments. The present invention can be modified in many ways within the scope of the gist of the present invention.

The information processing apparatus 10 of the embodiment described above includes, for example, a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in FIG. It can be realized using a general-purpose computer system. Memory 902 and storage 903 are storage devices. In the computer system, each function of the information processing apparatus 10 is realized by the CPU 901 executing a predetermined program loaded on the memory 902 .

The information processing device 10 may be implemented by one computer. The information processing device 10 may be implemented by multiple computers. The information processing device 10 may be a virtual machine implemented in a computer. A program for the information processing device 10 can be stored in computer-readable recording media such as HDD, SSD, USB memory, CD, and DVD.

The program for the information processing device 10 may be installed and executed on a computer (server device, etc.) within the communication network in order to perform sensing and control from a remote location via the communication network. The program can also be distributed on a recording medium and distributed over a communication network. The program may be executed by installing a distributed or distributed program in a computer (in-vehicle device or the like) in the vehicle 30 .

[Appendix]
The following additional remarks are disclosed regarding the above embodiments.

(Appendix 1)
memory;
at least one processor connected to the memory;
including
The processor
An information processing device that is communicatively connected to a spatio-temporal database that processes spatio-temporal data in real time and provides information to a passenger of a mobile object,
Image data taken by the plurality of mobile bodies at a plurality of locations on a plurality of dates and times, time data at the time of shooting, and position data of each mobile body at the time of shooting are collected from the plurality of mobile bodies, and the collected plurality of transmitting image data, a plurality of time data, and a plurality of position data to the spatio-temporal database;
receiving position data of a given mobile object from a given mobile object;
retrieving an object in the image data corresponding to the position data of the predetermined mobile object from the image data group of the spatio-temporal database;
Augmented reality display of the predetermined moving body is object display information for displaying the object in augmented reality by superimposing the object on the scene of the predetermined moving body at the same display position as the display position where the object was displayed in the past. send to the device,
Information processing equipment.

(Appendix 2)
A non-temporary device that is communicatively connected to a spatio-temporal database that processes spatio-temporal data in real time and that stores a program executable by a computer to perform information processing that provides information to passengers of a mobile object a storage medium,
The information processing includes:
Image data taken by the plurality of mobile bodies at a plurality of locations on a plurality of dates and times, time data at the time of shooting, and position data of each mobile body at the time of shooting are collected from the plurality of mobile bodies, and the collected plurality of transmitting image data, a plurality of time data, and a plurality of position data to the spatio-temporal database;
receiving position data of a given mobile object from a given mobile object;
retrieving an object in the image data corresponding to the position data of the predetermined mobile object from the image data group of the spatio-temporal database;
Augmented reality display of the predetermined moving body is object display information for displaying the object in augmented reality by superimposing the object on the scene of the predetermined moving body at the same display position as the display position where the object was displayed in the past. send to the device,
Non-transitory storage media.

1: information processing system 10: information processing device 11: collection unit 12: reception unit 13: processing unit 14: transmission unit 20: spatio-temporal database 30: automobile 31: camera 32: position sensor 33: clock 34: input device 35: Output device 901: CPU
902: Memory 903: Storage 904: Communication device 905: Input device 906: Output device

Claims (10)

1. An information processing device that is communicatively connected to a spatio-temporal database that processes spatio-temporal data in real time and provides information to passengers of a mobile object,
   Image data taken by the plurality of mobile bodies at a plurality of locations on a plurality of dates and times, time data at the time of shooting, and position data of each mobile body at the time of shooting are collected from the plurality of mobile bodies, and the collected plurality of a collection unit that transmits image data, a plurality of time data, and a plurality of position data to the spatio-temporal database;
   a receiving unit that receives position data of a predetermined moving body from a predetermined moving body;
   a processing unit that searches for an object in image data corresponding to the position data of the predetermined mobile object from the image data group of the spatio-temporal database;
   Augmented reality display of the predetermined moving body is object display information for displaying the object in augmented reality by superimposing the object on the scene of the predetermined moving body at the same display position as the display position where the object was displayed in the past. a transmitter for transmitting to the device;
   Information processing device.
2. The processing unit is
   2. The information processing apparatus according to claim 1, wherein an object in image data corresponding to both position data and time data of said predetermined moving body is searched.
3. The processing unit is
   2. The information processing apparatus according to claim 1, wherein said object is retrieved based on a tag indicating image content of each image data attached to said plurality of image data.
4. The processing unit is
   2. The information processing apparatus according to claim 1, wherein said object related to a profile of a passenger of said predetermined mobile body is retrieved.
5. The processing unit is
   2. The information processing apparatus according to claim 1, wherein the object that affects driving of the predetermined moving body is searched.
6. The transmission unit
   2. The information processing apparatus according to claim 1, wherein object commentary information output information for augmented reality display or audio output of the commentary information of the object is transmitted to the augmented reality display device.
7. The explanatory information includes:
   7. The information processing apparatus according to claim 6, wherein the object is generated based on a search key for searching the image data group of the spatio-temporal database.
8. In an information processing method for providing information to a passenger of a mobile object by being communicatively connected to a spatio-temporal database that processes spatio-temporal data in real time,
   The information processing device
   Image data taken by the plurality of mobile bodies at a plurality of locations on a plurality of dates and times, time data at the time of shooting, and position data of each mobile body at the time of shooting are collected from the plurality of mobile bodies, and the collected plurality of transmitting image data, a plurality of time data, and a plurality of position data to the spatio-temporal database;
   receiving from a given mobile object location data of said given mobile object;
   a step of searching the image data group of the spatio-temporal database for an object in the image data corresponding to the position data of the predetermined moving object;
   Augmented reality display of the predetermined moving body is object display information for displaying the object in augmented reality by superimposing the object on the scene of the predetermined moving body at the same display position as the display position where the object was displayed in the past. transmitting to the device;
   Information processing method that performs
9. An information processing program that causes a computer in a communication network to function as the information processing apparatus according to claim 1.
10. An information processing program that causes a computer in a moving body to function as the information processing device according to claim 1.

Images