JP2023135615A - Bird's eye view data generation device, learning device, bird's eye view data generation program, bird's eye view data generation method, and robot - Google Patents

Abstract

To make it possible to generate, from two-dimensional observation information observed from a viewpoint of an observation moving body mounted with an observation device in a dynamic environment, a movement locus of the observation moving body on the ground and bird's eye view data representing a movement locus of each of the moving bodies on the ground even in a situation in which a static landmark is not detected.SOLUTION: A bird's eye view data generation device includes: an acquisition unit 22 that acquires time series data of two-dimensional observation information representing at least one moving body observed from a viewpoint from an observation moving body mounted with an observation device in a dynamic environment; and a generation unit 26 that, using a learned model for estimating a movement of the observation moving body on the ground and a movement of each of the moving bodies on the ground, generates a movement locus of the observation moving body on the ground, which is acquired from the time-series data of the two-dimensional observation information when the observation moving body is observed from a bird's eye view position, and bird's-eye view data representing a movement locus of each of the moving bodies on the ground.SELECTED DRAWING: Figure 1

Description

There is an application for exception to loss of novelty.

The present invention relates to an overhead view data generation device, a learning device, an overhead view data generation program, an overhead view data generation method, and a robot.

BACKGROUND ART Conventionally, there has been known a technique for estimating a person's position distribution from an overhead perspective based on a human skeleton observed in a video shot from a first-person perspective (Non-Patent Document 1).

Furthermore, a technique is known in which a moving object is added to the optimization target of static landmark-based self-position estimation (SLAM) and sequential optimization is performed (Non-Patent Document 2).

Furthermore, a technique for estimating a position using GNSS (Global Navigation Satellite System) is known (Non-Patent Document 3).

Furthermore, a technique for estimating the shooting position of a first-person video in an overhead view video is known (Patent Document 1). In this technique, motion features extracted from both the bird's-eye view and the first-person view are compared for estimation.

"MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation", Internet search <URL: https://arxiv. org/abs/1906.06059>, Jun 2019 "CubeSLAM: Monocular 3D Object SLAM", Internet search <URL: https://arxiv. org/abs/1806.00557>, Jun 2018 "Current status and prospects of field robotics", Internet search <URL: https://committees. jsce. or. jp/opcet_sip/system/files/0130_01. pdf＞

JP2021-77287A

However, with the technique described in Non-Patent Document 1, it is not possible to restore the movement of the observation camera or the movement trajectory of surrounding moving objects.

Further, the technique described in Non-Patent Document 2 can only be applied in an environment where static landmarks can be stably observed together with moving objects. Furthermore, the motion model of the moving body is limited to simple rigid body motion, and cannot support the movement of the moving body in consideration of interaction.

Further, the technique described in Non-Patent Document 3 is only intended to restore the self-position of the device equipped with GNSS, and cannot restore the positions of surrounding moving objects. Furthermore, in an environment where there is shielding from a skyscraper or the like, reception of GPS (Global Positioning System) radio waves becomes unstable, resulting in inaccurate position restoration results.

Further, the technique described in Patent Document 1 cannot be applied when an overhead view video is not available.

The present invention has been made in view of the above points, and even in a situation where static landmarks are not detected, it can be observed from the perspective of an observation vehicle equipped with an observation device in a dynamic environment. A bird's-eye view data generation device, a learning device, and a bird's-eye view data generation program capable of generating bird's-eye view data representing a movement trajectory on the ground of an observed moving object and a movement trajectory on the ground of each moving object from two-dimensional observation information, The purpose of this invention is to provide a bird's-eye view data generation method and a robot.

A first aspect of the disclosure is a bird's-eye view data generation device that generates time-series data of two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. and a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground, from the time series data of the two-dimensional observation information. a generation unit that generates bird's-eye view data representing a movement trajectory of the observed moving object on the ground and a movement trajectory of each of the moving objects on the ground obtained when the observed moving object is observed from a bird's-eye view position; include.

In the first aspect, the generation unit uses a trained model that estimates the movement of the observation moving object on the ground and the distribution of the movement of each of the moving objects on the ground to estimate the two-dimensional observation information. A movement trajectory representing the position distribution of the observed moving object on the ground, obtained when the observed moving object is observed from a bird's-eye view position, and a position distribution of each of the moving objects on the ground, obtained from time-series data. Bird's-eye view data representing the movement trajectory may be generated.

In the first aspect, tracking each of the moving objects is tracked from time-series data of the two-dimensional observation information, and the position and size of each of the moving objects at each time on the two-dimensional observation information is acquired. The generation unit receives the position and size at each time of each of the moving objects on the two-dimensional observation information as input, and generates the movement of the observed moving object on the ground and each of the moving objects. The bird's-eye view data may be generated using the learned model that estimates movement on the ground.

In the first aspect, the learned model includes a first encoder that receives as input the position and size of each of the moving objects at the target time and outputs a vector, and a first encoder that receives the position and size of each of the moving objects at the target time and outputs a vector, and a second encoder that takes as input the movement on the ground of The decoder may also include a decoder that receives the vector as an input and outputs the movement of the observation moving object on the ground at the target time and the movement of each of the moving objects on the ground.

A second aspect of the disclosure is a learning device that learns about at least one moving object on two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment. Obtaining a combination of time-series data of the position and size at each time, movement of the observed moving object on the ground, and time-series data of the movement of each of the moving objects on the ground as training data. and the movement of the observed moving object on the ground, and the movement of each of the moving objects on the ground based on the teacher data and the position and size of each of the moving objects on the two-dimensional observation information at each time as input. a learning unit that learns a model for estimating movement on the ground.

In the second aspect, the model includes a first encoder that receives as input the position and size of each of the moving objects at a target time and outputs a vector, and a ground surface of the observed moving object obtained one time ago. a second encoder that inputs the above movement and the movement of each of the moving objects on the ground and outputs a vector; the vector output by the first encoder; and the vector output by the second encoder. The decoder may also include a decoder that receives as input and outputs the movement of the observed moving object on the ground at the target time and the movement of each of the moving objects on the ground.

A third aspect of the disclosure is an overhead data generation device that generates time-series data of two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. from the time-series data of the two-dimensional observation information using an acquisition unit that acquires the movement of the observed moving object on the ground and a trained model that predicts the movement of each of the moving objects on the ground. a generation unit that generates a prediction result of bird's-eye view data representing a movement trajectory of the observed moving object on the ground, and a movement trajectory of each of the moving objects on the ground, obtained when the observed moving object is observed from a bird's-eye view position; and, including.

In the third aspect, the generation unit uses a trained model that predicts the movement of the observation moving object on the ground and the distribution of the movement of each of the moving objects on the ground to predict the two-dimensional observation information. A movement trajectory representing the position distribution of the observed moving object on the ground, obtained when the observed moving object is observed from a bird's-eye view position, and a position distribution of each of the moving objects on the ground, obtained from time-series data. A prediction result of bird's-eye view data representing a movement trajectory may be generated.

A fourth aspect of the disclosure is a bird's-eye view data generation program that causes a computer to generate two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. an acquisition step of acquiring time-series data; and a learned model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground. A generation step of generating bird's-eye view data representing a movement trajectory of the observed moving object on the ground, and a movement trajectory of each of the moving objects on the ground, obtained when the observed moving object is observed from a bird's-eye view position. This is a program for executing processing including.

A fifth aspect of the disclosure is a bird's-eye view data generation method, in which a computer generates two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. an acquisition step of acquiring time-series data; and a learned model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground. A generation step of generating bird's-eye view data representing a movement trajectory of the observed moving object on the ground, and a movement trajectory of each of the moving objects on the ground, obtained when the observed moving object is observed from a bird's-eye view position. Execute processing including.

A sixth aspect of the disclosure is a robot, and an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of a robot equipped with an observation device in a dynamic environment. Then, using a trained model that estimates the movement of the robot on the ground and the movement of each of the moving objects on the ground, the robot is observed from a bird's-eye view position based on the time series data of the two-dimensional observation information. a generation unit that generates bird's-eye view data representing a movement trajectory of the robot on the ground and a movement trajectory of each of the moving bodies on the ground, obtained when the robot moves autonomously; A control unit that controls the autonomous traveling unit so that the robot moves to a destination using bird's-eye view data.

A seventh aspect of the disclosure is a bird's-eye view data generation program that causes a computer to generate two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. an acquisition step of acquiring time-series data, and a learned model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground, to obtain the time-series data of the two-dimensional observation information. From this, a prediction result of bird's-eye view data representing the movement trajectory of the observed moving object on the ground and the movement trajectory of each of the moving objects on the ground obtained when the observed moving object is observed from a bird's-eye view position is generated. This is a program for executing processing including a generation process.

An eighth aspect of the disclosure is a bird's-eye view data generation method, in which a computer generates two-dimensional observation information representing at least one moving object observed from the perspective of an observation moving object equipped with an observation device in a dynamic environment. an acquisition step of acquiring time-series data, and a learned model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground, to obtain the time-series data of the two-dimensional observation information. From this, a prediction result of bird's-eye view data representing the movement trajectory of the observed moving object on the ground and the movement trajectory of each of the moving objects on the ground obtained when the observed moving object is observed from a bird's-eye view position is generated. A process including a generation process is executed.

A ninth aspect of the disclosure is a robot, and an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of a robot equipped with an observation device in a dynamic environment. Then, using a trained model that predicts the movement of the robot on the ground and the movement of each of the moving objects on the ground, the robot is observed from a bird's-eye view position based on the time series data of the two-dimensional observation information. a generation unit that generates a prediction result of bird's-eye view data representing a movement trajectory of the robot on the ground and a movement trajectory of each of the moving objects on the ground, obtained when and a control unit that controls the autonomous traveling unit so that the robot moves to a destination using the prediction result of the bird's-eye view data.

According to the present invention, even in a situation where static landmarks are not detected, the observation vehicle can be detected from two-dimensional observation information observed from the viewpoint of the observation vehicle equipped with an observation device in a dynamic environment. It is possible to generate overhead view data representing a movement trajectory on the ground and a movement trajectory of each moving object on the ground.

FIG. 1 is a diagram showing a schematic configuration of a robot according to a first embodiment. FIG. 3 is a diagram showing an example of an image taken by a camera. It is a figure which shows an example of the result of detecting a person from an image. FIG. 3 is a diagram showing an example of a trained model. FIG. 3 is a diagram showing an example of bird's-eye view data. FIG. 2 is a block diagram showing the hardware configuration of an overhead view data generation device and a learning device according to the first and second embodiments. FIG. 1 is a diagram showing a schematic configuration of a learning device according to first and second embodiments. It is a flowchart showing the flow of learning processing by the learning device according to the first and second embodiments. 2 is a flowchart showing the flow of overhead view data generation processing by the overhead view data generation device according to the first and second embodiments. FIG. 2 is a diagram showing a schematic configuration of an information processing terminal according to a second embodiment. FIG. 3 is a diagram showing an example of bird's-eye view data. FIG. 7 is a diagram showing another example of bird's-eye view data. It is a figure which shows an example of the result of detecting a person from an image.

An example of an embodiment of the present invention will be described below with reference to the drawings. In addition, the same reference numerals are given to the same or equivalent components and parts in each drawing. Further, the dimensional ratios in the drawings may be exaggerated for convenience of explanation and may differ from the actual ratios.

[First embodiment]
FIG. 1 is a diagram showing a schematic configuration of a robot 100 according to a first embodiment of the present invention. As shown in FIG. 1, the robot 100 includes a camera 10, an overhead view data generation device 20, a notification section 50, and an autonomous traveling section 60. The bird's-eye view data generation device 20 includes an acquisition section 22 , a tracking section 24 , a generation section 26 , a model storage section 27 , and a control section 28 . Note that the robot 100 is an example of an observation moving object, and the camera 10 is an example of an observation device.

The camera 10 photographs the surroundings of the robot 100 at predetermined intervals until it moves from the start point to the destination, and outputs the photographed images to the acquisition unit 22 of the bird's-eye view data generation device 20. Note that the image is an example of two-dimensional observation information.

For example, an image representing at least one person observed from the viewpoint of the robot 100 in a dynamic environment is captured by the camera 10 (see FIG. 2).

As the camera 10, a perspective projection RGB camera, a fisheye camera, or a 360-degree camera may be used.

The acquisition unit 22 acquires time-series data of images captured by the camera 10.

The tracking unit 24 tracks each person from the time series data of the acquired images, and acquires the position and size of each person on the image at each time.

For example, as shown in Figure 3, for each person on an image, a bounding box representing the person is detected and tracked, and the center position (center position of the bounding box) and height (center position of the bounding box) of the person on the image are detected and tracked. height) at each time.

The generation unit 26 uses a learned model that estimates the movement of the robot 100 on the ground and the movement of each person on the ground to calculate the time of each person on the image obtained from the time series data of the image. From the position and size, overhead view data representing the movement trajectory of the robot 100 on the ground and the movement trajectory of each person on the ground, obtained when the robot 100 is observed from a bird's-eye view position, is generated.

Specifically, the generation unit 26 receives the position and size of each person on the image at each time as input, and is trained to estimate the movement of the robot 100 on the ground and the movement of each person on the ground. Generate bird's-eye view data using the model.

Here, the trained model includes a first encoder that inputs the position and size of each target time of the person and outputs a vector, and the movement of the robot 100 on the ground obtained one time ago and the person. a second encoder that inputs the movement on the ground of each of the robots 100 and outputs a vector; and a second encoder that inputs the movement on the ground of each of and a decoder that outputs the movement of each person on the ground.

More specifically, as shown in FIG. 4, the learned model 70 includes a first encoder 72, a second encoder 74, and a decoder 76.

The first encoder 72 inputs the position and size of each person observed by the robot 100 from a first-person viewpoint, takes self-attention between the people, and outputs the obtained vector.

Specifically, a vector representing the position and size of each person on the image at time t is input into a multilayer perceptron (MLP) 720, and the obtained vector is used as the input vector of the first encoder 72. shall be.

The multi-head self-attention layer 722 of the first encoder 72 receives the input vectors of the first encoder 72 as each of Query, Key, and Value, takes self-attention, and outputs the vectors.

The first normalization layer 724 of the first encoder 72 adds the input vector of the first encoder 72 and the output vector of the multi-head self-attention layer 722, performs normalization, and outputs the vector.

The forward propagation neural network 726 inputs the output vector of the first normalization layer 724 and outputs the vector.

The second normalization layer 728 adds the output vector of the first normalization layer 724 and the output vector of the forward propagation neural network 726, performs normalization, outputs the vector, and sends this to the first encoder 72. Let be the output vector of . This output vector represents the first-person perspective embedding.

The second encoder 74 inputs the movement of the robot 100 on the ground and the movement of each person on the ground obtained one time ago, and encodes the relative position and velocity of each person with respect to the position of the robot 100. , output the obtained vector.

Specifically, a vector representing the movement of each person on the ground with respect to the position of the robot 100 is calculated from the movement of the robot 100 on the ground and the movement of each person on the ground obtained at time t-1. This vector is input to the multilayer perceptron 740, and the obtained vector is used as the input vector to the second encoder 74.

The multi-head self-attention layer 742 of the second encoder 74 receives the input vectors of the second encoder 74 as each of Query, Key, and Value, takes self-attention, and outputs the vectors.

The normalization layer 744 of the second encoder 74 adds the input vector of the second encoder 74 and the output vector of the multi-head self-attention layer 742, then performs normalization and outputs the vector. This vector represents the embedding of the bird's-eye view.

The decoder 76 performs cross-attention between the output vector of the first encoder 72 and the output vector of the second encoder 74, and outputs a vector obtained as a result of the cross-attention. This vector represents the result of multi-head prediction of the movement of the robot 100 on the ground and the movement of each person on the ground.

Specifically, the output vector of the first encoder 72 and the output vector of the second encoder 74 are input to the decoder 76.

The multi-head cross-attention layer 760 of the decoder 76 receives the output vector of the first encoder 72 as a key and value, receives the output vector of the second encoder 74 as a query, takes cross-attention, and outputs the vector. .

The first normalization layer 762 of the decoder 76 adds the output vector of the second encoder 74 and the output vector of the multi-head cross-attention layer 760, performs normalization, and outputs the vector.

The forward propagation neural network 764 inputs the output vector of the first normalization layer 762 and outputs the vector.

The second normalization layer 766 adds the output vector of the first normalization layer 762 and the output vector of the forward propagation neural network 764, performs normalization, outputs the vector, and outputs the vector from the decoder 76. Let it be a vector.

The forward propagation neural network 768 inputs the output vector of the decoder 76 and outputs a vector representing the movement of the robot 100 at time t.

Further, the forward propagation neural network 770 receives the output vector of the decoder 76 as input, and outputs a vector representing each movement of the person at time t.

Here, the vector representing movement is, for example, a vector representing relative position and relative velocity with respect to one time ago. Note that the vector representing movement may be a vector representing a relative position with respect to one time ago, or a vector representing relative velocity with respect to one time ago.

In this embodiment, the generation unit 26 generates a vector representing the position and size of each person on the image at time t, a vector representing the movement of the robot 100 on the ground obtained at time t-1, and Using the learned model 70, find a vector representing the movement of the robot 100 on the ground and a vector representing the movement of each person on the ground at time t from vectors representing the movement of each person on the ground. By repeating this for each time t, overhead view data is generated.

The generation unit 26 generates bird's-eye view data as shown in FIG. 5, for example. FIG. 5 shows an example in which a line connecting black circles indicates the locus of movement of the robot 100 on the ground, and a broken line indicates the locus of movement of the person on the ground.

The control unit 28 uses the bird's-eye view data to control the autonomous traveling unit 60 so that the robot 100 moves to the destination. For example, the control unit 28 specifies the moving direction and speed of the robot 100, and controls the autonomous traveling unit 60 to move in the specified moving direction and speed.

In addition, if the control unit 28 determines that intervention is necessary using the bird's-eye view data, the control unit 28 controls the notification unit 50 to output a message such as "Please clear the road" or to sound a warning sound. do.

Next, the hardware configuration of the bird's-eye view data generation device 20 of the robot 100 will be described.

As shown in FIG. 6, the bird's-eye view data generation device 20 includes a CPU (Central Processing Unit) 61, a ROM (Read Only Memory) 62, a RAM (Random Access Memory) 63, a storage 64, and a communication interface (I/F) 65. has. Each component is communicably connected to each other via a bus 66.

In this embodiment, the storage 64 stores an overhead view data generation program. The CPU 61 is a central processing unit that executes various programs and controls each component. That is, the CPU 61 reads the program from the storage 64 and executes the program using the RAM 63 as a work area. The CPU 61 controls each of the above components and performs various arithmetic operations according to programs recorded in the storage 64.

The ROM 62 stores various programs and various data. The RAM 63 temporarily stores programs or data as a work area. The storage 64 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The communication interface 65 is an interface for communicating with other devices, and uses, for example, a standard such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

The trained model 70 is trained in advance by the learning device 120 shown in FIG. This learning device 120 will be explained below.

FIG. 7 is a diagram showing a schematic configuration of the learning device 120 according to the first embodiment of the present invention. As shown in FIG. 7, the learning device 120 includes a teacher data storage section 122, an acquisition section 124, a learning section 126, and a model storage section 128.

The teacher data storage unit 122 stores time series data of the position and size of each person on the image at each time observed from the viewpoint of the robot 100 in a dynamic environment, the movement of the robot 100 on the ground, A plurality of combinations of the time-series data of the movement of each person on the ground and the movement of each person on the ground are stored as teacher data.

The acquisition unit 124 acquires a plurality of pieces of teacher data from the teacher data storage unit 122.

The learning unit 126 has the same configuration as the learned model 70 when inputting time series data of the position and size of each person on the image of the teacher data at each time based on a plurality of pieces of teacher data. The parameters of the model are learned so that the model outputs time-series data of the movement of the robot 100 on the ground and the movement of each person on the ground as teacher data.

The model storage unit 128 stores the learning results obtained by the learning unit 126 as a learned model.

Next, the hardware configuration of the learning device 120 will be explained.

As shown in FIG. 6 above, the learning device 120 includes a CPU 61, a ROM 62, a RAM 63, a storage 64, and a communication interface 65, similarly to the bird's-eye view data generation device 20. Each component is communicably connected to each other via a bus 66. In this embodiment, the storage 64 stores a learning program.

Next, the operation of the learning device 120 will be explained.

First, the learning device 120 is provided with time series data of the position and size at each time of each person on the image observed from the viewpoint of the robot 100 in a dynamic environment, the movement of the robot 100 on the ground, and A plurality of combinations of time-series data of each person's movement on the ground are input as teacher data and stored in the teacher data storage unit 122.

FIG. 8 is a flowchart showing the flow of learning processing by the learning device 120. The learning process is performed by the CPU 61 reading the learning program from the storage 64, loading it onto the RAM 63, and executing it.

In step S100, the CPU 61, as the acquisition unit 124, acquires a plurality of pieces of teacher data from the teacher data storage unit 122.

In step S102, when the CPU 61, as the learning unit 126, inputs time series data of the position and size of each person on the image of the teacher data at each time based on a plurality of pieces of teacher data, The parameters of a model having a configuration similar to the model 70 are learned so that the model outputs time-series data of the movement of the robot 100 on the ground and the movement of each person on the ground as teacher data.

The learning result by the learning unit 126 is then stored in the model storage unit 128 as a trained model.

Next, the operation of the robot 100 will be explained.

First, the learned model learned by the learning device 120 is stored in the model storage unit 27 of the bird's-eye view data generation device 20.

Then, when the robot 100 moves to the destination by the autonomous traveling unit 60, the camera 10 takes pictures of the surroundings of the robot 100 at predetermined intervals, and the bird's-eye view data generation device 20 periodically takes pictures of the surroundings of the robot 100 as shown in FIG. Bird's-eye view data is generated by the bird's-eye view data generation process shown in FIG.

FIG. 9 is a flowchart showing the process of generating bird's-eye view data by the bird's-eye view data generating device 20. The CPU 61 reads the bird's-eye view data generation program from the storage 64, expands it to the RAM 63, and executes it, thereby performing the bird's-eye view data generation process.

In step S110, the CPU 61, as the acquisition unit 22, acquires time-series data of images photographed by the camera 10.

In step S112, the CPU 61, as the tracking unit 24, tracks each person from the time series data of the acquired image, and acquires the position and size of each person on the image at each time.

In step S114, the CPU 61, as the generation unit 26, generates a vector representing the movement of the robot 100 on the ground and the movement of each person on the ground one hour before the first time of the time series data of the acquired image. Set the initial value for the vector representing . Further, the first time of the time series data of the image is assumed to be time t.

In step S116, the CPU 61, as the generation unit 26, generates a vector representing the position and size of each person on the image at time t, and a vector representing the movement of the robot 100 on the ground obtained at time t-1. , and a vector representing the movement of each person on the ground, using the learned model 70, a vector representing the movement of the robot 100 on the ground and a vector representing the movement of each person on the ground at time t. Estimate.

In step S118, the CPU 61, acting as the generation unit 26, determines whether a predetermined repetition end condition is satisfied. For example, reaching the final time of the time-series data of an image may be used as the repetition termination condition. If the repetition end condition is satisfied, the CPU 61 moves to step S120. On the other hand, if the repetition end condition is not satisfied, the CPU 61 returns to step S116, sets the next time to time t, and repeats the process.

In step S120, the CPU 61, as the generation unit 26, generates a vector for the robot 100 at each time from a vector representing the movement of the robot 100 on the ground and a vector representing the movement of each person on the ground. 100 on the ground, the observation direction of the camera 10, and the position of each person on the ground are generated and output to the control unit 28, and the bird's-eye view data generation process is completed.

Using the generated bird's-eye view data, the control unit 28 specifies the moving direction and speed of the robot 100 so that the robot 100 moves to the destination, and autonomously runs the robot 100 so that the robot 100 moves in the specified moving direction and speed. 60. In addition, if the control unit 28 determines that intervention is necessary using the bird's-eye view data, the control unit 28 controls the notification unit 50 to output a message such as "Please clear the road" or to sound a warning sound. do.

In this way, in this embodiment, using a trained model that estimates the movement of the robot 100 on the ground and the movement of each person on the ground, the robot 100 is estimated from a bird's-eye view position based on time-series data of images. Overhead view data representing the movement trajectory of the robot 100 on the ground and the movement trajectory of each person on the ground, obtained when observed, is generated. As a result, even in a situation where static landmarks are not detected, the movement trajectory of the robot 100 on the ground and the person It is possible to generate bird's-eye view data representing the movement locus on the ground of each of the ground planes.

In addition, since it can be realized by calculation using a trained model, the amount of calculation is reduced and it is possible to generate bird's-eye view data in real time.

Further, since time-series data of the position and size of each person on the image at each time is used as the training data, there is no need to use an actual image. This reduces the burden of creating teacher data.

[Second embodiment]
Next, a bird's-eye view data generation device according to a second embodiment will be described. Note that parts having the same configuration as those in the first embodiment are given the same reference numerals and detailed explanations will be omitted.

In the second embodiment, an example will be described in which an information processing terminal held by a user includes an overhead view data generation device.

FIG. 10 is a diagram showing a schematic configuration of an information processing terminal 200 according to the second embodiment of the present invention. As shown in FIG. 10, the information processing terminal 200 includes a camera 10, an overhead data generation device 220, and an output unit 250. The bird's-eye view data generation device 220 includes an acquisition section 22 , a tracking section 24 , a generation section 26 , and a model storage section 27 . Note that the user is an example of an observation moving object, and the camera 10 is an example of an observation device.

The information processing terminal 200 is held directly by the user, or is mounted on a holding object (for example, a suitcase) held by the user.

The camera 10 photographs the surroundings of the user at predetermined intervals, and outputs the photographed images to the acquisition unit 22 of the bird's-eye view data generation device 220.

The generation unit 26 uses a trained model that estimates the movement of the user on the ground and the movement of each person on the ground to calculate the position of each person on the image at each time obtained from the time series data of the image. and the size, it generates bird's-eye view data representing the movement trajectory of the user on the ground and the movement trajectory of each person on the ground, obtained when observing the user from a position overlooking the user, and outputs it to the output unit 250. .

The model storage unit 27 stores learned models for estimating the movement of the user on the ground and the movements of each person on the ground, which are learned by the learning device 120 in the same manner as in the first embodiment. There is.

The output unit 250 presents the generated bird's-eye view data to the user and transmits the bird's-eye view data to a server (not shown) via the Internet.

Further, as shown in FIG. 6, the bird's-eye view data generation device 220 has the same hardware configuration as the bird's-eye view data generation device 20 of the first embodiment.

Note that the other configurations and operations of the bird's-eye view data generation device 220 are the same as those in the first embodiment, and therefore description thereof will be omitted.

Further, as shown in FIG. 7 above, the learning device 120 according to the second embodiment includes a teacher data storage section 122, an acquisition section 124, a learning section 126, and a model storage section 128.

The teacher data storage unit 122 stores time-series data of the position and size of each person on the image at each time observed from the user's viewpoint in a dynamic environment, the movement of the user on the ground, and the person. A plurality of combinations of each movement on the ground with time series data are stored as teacher data.

The learning unit 126 has the same configuration as the learned model 70 when inputting time series data of the position and size of each person on the image of the teacher data at each time based on a plurality of pieces of teacher data. The parameters of the model are learned so that the model outputs time-series data of the user's movement on the ground and the movement of each person on the ground in the training data.

Note that the other configurations and functions of the learning device 120 are the same as those in the first embodiment, so their explanations will be omitted.

In this way, in this embodiment, the user's movement is estimated from the time-series data of images using a trained model that estimates the movement of the user holding the information processing terminal 200 on the ground and the movement of each person on the ground. Bird's-eye view data representing the user's movement trajectory on the ground and the movement trajectory of each person on the ground, obtained when observed from a bird's-eye view position, is generated. As a result, even in a situation where static landmarks are not detected, the movement of the user on the ground can be determined based on the image observed from the viewpoint of the user holding the information processing terminal 200 having the camera 10 in a dynamic environment. It is possible to generate overhead view data representing the trajectory and the movement trajectory of each person on the ground.

The present invention can also be applied to autonomous vehicles. In this case, the observation moving object is a self-driving vehicle, the observation device is a camera, a laser radar, a millimeter wave radar, and the moving object is another vehicle, a motorcycle, a pedestrian, etc.

[Third embodiment]
Next, a bird's-eye view data generation device according to a third embodiment will be described. Note that the bird's-eye view data generation device according to the third embodiment has the same configuration as the first embodiment, so the same reference numerals are given and detailed explanation will be omitted.

The third embodiment differs from the first embodiment in that the distribution of the user's movement on the ground and the distribution of the movement of each person on the ground are predicted.

The generation unit 26 of the bird's-eye view data generation device 20 according to the third embodiment generates a time series of images using a trained model that predicts the distribution of the movement of the robot 100 on the ground and the movement of each person on the ground. From the position and size at each time of each person on the image acquired from the data, the movement trajectory of the robot 100 on the ground and the position of each person on the ground obtained when observing the robot 100 from a bird's-eye view position. A prediction result of bird's-eye view data representing a movement trajectory representing the position distribution of is generated.

Specifically, the generation unit 26 inputs the position and size of each person on the image at each time, and calculates the movement of the robot 100 on the ground one time ahead and the movement of each person on the ground. Generate prediction results for bird's-eye view data using a trained model that predicts distribution.

Here, the trained model includes a first encoder that inputs the position and size of each target time of the person and outputs a vector, and a first encoder that outputs a vector, the movement of the robot 100 on the ground obtained at the target time, and the person's A second encoder takes as input the distribution of movement on each ground and outputs a vector, and takes as input the vector output by the first encoder and the vector output by the second encoder, and calculates the time one time ahead from the target time. The decoder outputs the movement of the robot 100 on the ground, and the distribution of the movement of each person on the ground.

More specifically, the first encoder 72 of the trained model 70 inputs the position and size of each person observed by the robot 100 from a first-person viewpoint, takes self-attention between the people, and outputs the obtained vector. do.

The second encoder 74 inputs the distribution of the movement of the robot 100 on the ground and the movement of each person on the ground obtained at the target time, and the distribution of the relative position and velocity of each person with respect to the position of the robot 100. encodes the distribution of and outputs the obtained vector.

Specifically, from the distribution of the movement of the robot 100 on the ground and the movement of each person on the ground obtained at time t, the distribution of the movement of each person on the ground with respect to the position of the robot 100 is expressed. A vector is determined, and this vector is input to the multilayer perceptron 740, and the obtained vector is used as the input vector to the second encoder 74.

The decoder 76 performs cross-attention between the output vector of the first encoder 72 and the output vector of the second encoder 74, and outputs a vector obtained as a result of the cross-attention. This vector represents the result of multi-head prediction of the movement of the robot 100 on the ground and the distribution of the movement of each person on the ground.

Here, the vector representing the distribution of motion is, for example, a vector representing a Gaussian distribution (average and variance) of relative position with respect to the target time and a Gaussian distribution (average and variance) of relative velocity. Note that the vector representing the distribution of motion may be a vector representing a Gaussian distribution (average and variance) of relative positions to the target time, or a vector representing a Gaussian distribution (average and variance) of relative velocities to the target time.

In this embodiment, the generation unit 26 generates a vector representing the position and size of each person on the image at time t, a vector representing the movement of the robot 100 on the ground obtained at time t, and a vector representing the movement of the robot 100 on the ground obtained at time t. Using the learned model 70, a vector representing the movement of the robot 100 on the ground and a vector representing the distribution of the movement of each person on the ground at time t+1 are obtained from vectors representing the distribution of movement on the ground. By repeating the calculation for each time t, a prediction result of the bird's-eye view data is generated.

The generation unit 26 generates a prediction result of bird's-eye view data as shown in FIG. 11A, for example. FIG. 11A shows the movement trajectory of the robot 100 on the ground by a line connecting black circles indicating positions determined from relative positions. In addition, in FIG. 11A, the movement locus of each person on the ground is shown by a line connecting x marks indicating the average position found from the average relative position, and the ellipse around the x mark indicates the position found from the distribution of relative positions. An example showing the distribution of is shown. The ellipse indicating the distribution may be a circle, or may be displayed using contour lines or color-coded lines indicating the height distribution. Furthermore, since the position of the robot 100 includes errors in the control of the robot 100 and sensors that specify the position, it may be calculated including the uncertainty distribution and displayed together with the distribution. , overhead view data representing the distribution of the positions of each person on the ground at the next time may be generated. In FIG. 11B, the vertical and horizontal axes represent distance, and an example of an overhead view including the distribution of the positions of robots (inverted triangles) and people is shown. The contour ellipse shows the position of the person with the distribution of uncertainty, and the dotted line shows the field of view of the camera of the robot 100. The example in FIG. 11B is a diagram showing information within the robot 100, so the position of the robot is fixed (there is no uncertainty distribution), and only the person has an uncertainty distribution.

The control unit 28 uses the bird's-eye view data to control the autonomous traveling unit 60 so that the robot 100 does not collide with a person and the robot 100 moves to the destination. For example, the control unit 28 specifies the moving direction and speed of the robot 100, and controls the autonomous traveling unit 60 to move in the specified moving direction and speed. At this time, by specifying the moving direction and speed of the robot 100 so as to avoid the elliptical range of the bird's-eye view data in FIG. 11, a collision between the robot 100 and the person can be further avoided.

The teacher data storage unit 122 of the learning device 120 according to the third embodiment stores time-series data of the position and size of each person on the image at each time observed from the viewpoint of the robot 100 in a dynamic environment. A plurality of combinations of the time-series data of the movement of the robot 100 on the ground, and the movement of each person on the ground are stored as teacher data. Here, the training data includes the position and size at the relevant time of each person on the image observed from the user's viewpoint in a dynamic environment, the user's movement on the ground at the next time, and the position and size of each person on the image at the next time. It is associated with the movement on the ground at the next time.

The acquisition unit 124 acquires a plurality of pieces of teacher data from the teacher data storage unit 122.

The learning unit 126 has the same configuration as the learned model 70 when inputting time series data of the position and size of each person on the image of the teacher data at each time based on a plurality of pieces of teacher data. The model outputs time-series data of movement corresponding to the movement of the robot 100 on the ground in the teacher data, and time-series data of movement distribution corresponding to the movement of each person in the teacher data on the ground. , learn the parameters of the model.

The model storage unit 128 stores the learning results obtained by the learning unit 126 as a learned model.

Note that the other configurations and operations of the bird's-eye view data generation device 20 and the learning device 120 according to the third embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted.

As described above, according to the present embodiment, the learned model that predicts the movement of the robot 100 on the ground at the next time and the movement of each person on the ground at the next time is used to predict the movement of the robot 100 on the ground at the next time. , a prediction result of bird's-eye view data representing the movement trajectory of the robot 100 on the ground and the movement trajectory of each person on the ground obtained when the robot 100 is observed from a bird's-eye view position is generated. As a result, even in a situation where static landmarks are not detected, the moving trajectory of the robot 100 on the ground and the person It is possible to generate a prediction result of bird's-eye view data representing the movement trajectory on the ground of each of the ground planes.

[Example]
An example in which bird's-eye view data is generated from time-series data of images by the bird's-eye view data generation device 20 of the first embodiment will be described.

As a comparative example, the posterior distribution is expressed using the relative position of the person from the robot and the movement model at each time, and the position of the robot and the person on the ground one time ago, as well as the current position. A method of generating bird's-eye view data is used to maximize the posterior distribution of the positions of the robot and the person on the ground, given the position and size of each person on the image at the time. .

The amount of calculation was measured for databases of different scenes: "Hotel", "ETH", and "Students". Further, in the comparative example, a CPU was used, and in the example, the amount of calculation was measured using a CPU and a GPU. Table 1 shows the results of measuring the amount of calculation.

As shown in Table 1, it was found that the amount of calculation was smaller in the example (ViewBirdiformer) than in the comparative example (GeoVB). Furthermore, it was found that the amount of calculation can be further reduced by using a GPU as the device.

[Modified example]
In addition, although the above-mentioned embodiment explained the case where the robot 100 and the information processing terminal 200 were equipped with the bird's-eye view data generation devices 20 and 220, the functions of the bird's-eye view data generation devices 20 and 220 may be provided in an external server. In this case, the robot 100 and the information processing terminal 200 transmit time-series data of images captured by the camera 10 to an external server. The external server generates bird's-eye view data from the time series data of the transmitted images and transmits it to the robot 100 and the information processing terminal 200.

Further, under conditions where a static landmark is detected from the image captured by the camera 10, the generation unit 26 may generate the bird's-eye view data using the static landmark represented by the image. For example, the technique described in Non-Patent Document 2 mentioned above may be used. In this case, under the condition that a static landmark is detected from the image photographed by the camera 10, the static landmark represented by the image is used to generate overhead view data, and the image photographed by the camera 10 is Under conditions where static landmarks are not detected (for example, in a crowded environment), overhead view data may be generated using the method described in the above embodiment. Further, the bird's-eye view data generated using the static landmark represented by the image and the bird's-eye view data generated by the method described in the above embodiment may be integrated.

The tracking unit 24 also detects and tracks a bounding box representing each person on the image, and determines the center position (center position of the bounding box) and height (height of the bounding box) of the person on the image. Although the explanation has been given using an example where the data is acquired at each time, the present invention is not limited to this. For example, the tracking unit 24 detects and tracks a human skeleton representing each person on the image, and determines the center position (center position of the human skeleton) and height (height of the human skeleton) of the person on the image. ) may be acquired at each time. Further, as shown in FIG. 12, the tracking unit 24 detects and tracks a line indicating the height of each person on the image, and detects and tracks the center position of the person on the image (the center position of the line). ) and height (line height) may be acquired at each time.

Further, although the two-dimensional observation information has been described using an example of an image, the present invention is not limited to this. For example, if the observation device is an event camera, data having a pixel value corresponding to the movement of each pixel may be used as the two-dimensional observation information.

Further, although the case where the moving object represented by the bird's-eye view data is a person has been described as an example, the present invention is not limited to this. For example, the moving object represented by the bird's-eye view data may be a personal mobility object such as a bicycle or a vehicle.

Furthermore, in the first embodiment described above, similarly to the third embodiment described above, two-dimensional observation information is obtained using a trained model that estimates the distribution of the movement of the robot on the ground and the movement of each person on the ground. From the time series data, overhead view data representing the movement trajectory of the robot on the ground and the movement trajectory representing the position distribution of each person on the ground obtained when the robot is observed from a bird's-eye view position is generated. You can.

Further, in each of the above embodiments, the CPU reads and executes the software (program), and the bird's-eye view data generation process and the learning process may be executed by various processors other than the CPU. In this case, the processor includes a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing, such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Cipher). rcuit) to execute specific processing such as An example is a dedicated electric circuit that is a processor having a specially designed circuit configuration. Additionally, the bird's-eye view data generation process and the learning process may be executed by one of these various processors, or by a combination of two or more processors of the same type or different types (for example, multiple FPGAs and CPUs). It may also be executed in combination with FPGA, etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit that is a combination of circuit elements such as semiconductor elements.

Further, in each of the embodiments described above, a mode has been described in which the bird's-eye view data generation program and the learning program are stored in advance in the storage 64, but the present invention is not limited thereto. The program can be stored on recording media such as CD-ROM (Compact Disc Read Only Memory), DVD-ROM (Digital Versatile Disc Read Only Memory), and USB (Universal Serial Bus) memory. It may also be provided in recorded form. Further, the program may be downloaded from an external device via a network.

Regarding the above embodiments, the following additional notes are further disclosed.

[Additional note 1]
an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation unit that generates bird's-eye view data representing the
A bird's-eye view data generation device including:

[Additional note 2]
The generation unit is
Using a trained model that estimates the movement of the observed moving object on the ground and the distribution of the movement of each of the moving objects on the ground,
A movement trajectory of the observed moving object on the ground, obtained when the observed moving object is observed from a bird's-eye view position, and a position distribution of each of the moving objects on the ground, from the time series data of the two-dimensional observation information. The bird's-eye view data generation device according to supplementary note 1, which generates bird's-eye view data representing a movement trajectory.

[Additional note 3]
further comprising a tracking unit that tracks each of the moving objects from the time series data of the two-dimensional observation information and obtains the position and size of each of the moving objects at each time on the two-dimensional observation information,
The generation unit inputs the position and size at each time of each of the moving objects on the two-dimensional observation information, and generates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground. The bird's-eye view data generation device according to supplementary note 1 or 2, which generates the bird's-eye view data using the learned model that estimates the bird's-eye view data.

[Additional note 4]
The trained model is
a first encoder that receives as input the position and size of each target time of the moving body and outputs a vector;
a second encoder that receives as input the movement of the observation moving object on the ground and the movement of each of the moving objects on the ground obtained one time ago, and outputs a vector;
The vector output by the first encoder and the vector output by the second encoder are input, and the movement of the observation moving object on the ground at the target time and the movement of each of the moving objects on the ground are calculated. 3.

[Additional note 5]
Time-series data of the position and size of each of the moving objects at each time on two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. and an acquisition unit that acquires a combination of the movement of the observed moving object on the ground and time-series data of the movement of each of the moving objects on the ground as training data;
Based on the teacher data, the position and size at each time of each of the moving objects on the two-dimensional observation information are input, and the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground are calculated. a learning unit that learns a model for estimating the movement of the
learning devices including;

[Additional note 6]
The model is
a first encoder that receives as input the position and size of each target time of the moving body and outputs a vector;
a second encoder that receives as input the movement of the observation moving object on the ground and the movement of each of the moving objects on the ground obtained one time ago, and outputs a vector;
The vector output by the first encoder and the vector output by the second encoder are input, and the movement of the observation moving object on the ground at the target time and the movement of each of the moving objects on the ground are calculated. The learning device according to supplementary note 5, including a decoder that outputs motion.

[Additional note 7]
an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation unit that generates a prediction result of bird's-eye view data representing;
A bird's-eye view data generation device including:

[Additional note 8]
The generation unit is
Using a trained model that predicts the movement of the observed moving object on the ground and the distribution of the movement of each of the moving objects on the ground,
A movement trajectory of the observed moving object on the ground, obtained when the observed moving object is observed from a bird's-eye view position, and a position distribution of each of the moving objects on the ground, from the time series data of the two-dimensional observation information. 8. The bird's-eye view data generation device according to supplementary note 7, which generates a prediction result of bird's-eye view data representing a movement trajectory.

[Additional note 9]
to the computer,
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating bird's-eye view data representing the
A bird's-eye view data generation program for executing processing including.

[Additional note 10]
The computer is
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating bird's-eye view data representing the
A bird's-eye view data generation method that performs processing including.

[Additional note 11]
an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from the viewpoint of a robot equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the robot on the ground and the movement of each of the moving objects on the ground,
Bird's-eye view data representing the movement trajectory of the robot on the ground and the movement trajectory of each of the mobile objects on the ground, obtained when the robot is observed from a bird's-eye view position from the time series data of the two-dimensional observation information. a generation unit that generates
an autonomous traveling unit that causes the robot to autonomously travel;
a control unit that uses the bird's-eye view data to control the autonomous traveling unit so that the robot moves to a destination;
including robots.

[Additional note 12]
to the computer,
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating a prediction result of bird's-eye view data representing the
A bird's-eye view data generation program for executing processing including.

[Additional note 13]
The computer is
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating a prediction result of bird's-eye view data representing the
A bird's-eye view data generation method that performs processing including.

[Additional note 14]
an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from the viewpoint of a robot equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the robot on the ground and the movement of each of the moving objects on the ground,
Bird's-eye view data representing the movement trajectory of the robot on the ground and the movement trajectory of each of the mobile objects on the ground, obtained when the robot is observed from a bird's-eye view position from the time series data of the two-dimensional observation information. a generation unit that generates a prediction result of
an autonomous traveling unit that causes the robot to autonomously travel;
a control unit that controls the autonomous traveling unit so that the robot moves to a destination using a prediction result of the bird's-eye view data;
including robots.

10 Camera 20 Overhead data generation device 22 Acquisition unit 24 Tracking unit 26 Generation unit 28 Control unit 50 Notification unit 60 Autonomous traveling unit 70 Learned model 72 First encoder 74 Second encoder 76 Decoder 100 Robot 120 Learning device 122 Teacher data storage unit 124 Acquisition unit 126 Learning unit 200 Information processing terminal 220 Overhead view data generation device

Claims (14)

an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation unit that generates bird's-eye view data representing the
A bird's-eye view data generation device including: The generation unit is
Using a trained model that estimates the movement of the observed moving object on the ground and the distribution of the movement of each of the moving objects on the ground,
A movement trajectory of the observed moving object on the ground, obtained when the observed moving object is observed from a bird's-eye view position, and a position distribution of each of the moving objects on the ground, from the time series data of the two-dimensional observation information. The bird's-eye view data generation device according to claim 1, wherein the bird's-eye view data generating device generates bird's-eye view data representing a movement trajectory. further comprising a tracking unit that tracks each of the moving objects from the time series data of the two-dimensional observation information and obtains the position and size of each of the moving objects at each time on the two-dimensional observation information,
The generation unit inputs the position and size at each time of each of the moving objects on the two-dimensional observation information, and generates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground. The bird's-eye view data generation device according to claim 1, wherein the bird's-eye view data is generated using the learned model that estimates the bird's-eye view data. The trained model is
a first encoder that receives as input the position and size of each target time of the moving body and outputs a vector;
a second encoder that receives as input the movement of the observation moving object on the ground and the movement of each of the moving objects on the ground obtained one time ago, and outputs a vector;
The vector output by the first encoder and the vector output by the second encoder are input, and the movement of the observation moving object on the ground at the target time and the movement of each of the moving objects on the ground are calculated. 4. The bird's-eye view data generation device according to claim 3, further comprising a decoder that outputs a motion of the object. Time-series data of the position and size of each of the moving objects at each time on two-dimensional observation information representing at least one moving object observed from the viewpoint of an observation moving object equipped with an observation device in a dynamic environment. and an acquisition unit that acquires a combination of the movement of the observed moving object on the ground and time-series data of the movement of each of the moving objects on the ground as training data;
Based on the teacher data, the position and size at each time of each of the moving objects on the two-dimensional observation information are input, and the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground are calculated. a learning unit that learns a model for estimating the movement of the
learning devices including; The model is
a first encoder that receives as input the position and size of each target time of the moving body and outputs a vector;
a second encoder that receives as input the movement of the observation moving object on the ground and the movement of each of the moving objects on the ground obtained one time ago, and outputs a vector;
The vector output by the first encoder and the vector output by the second encoder are input, and the movement of the observation moving object on the ground at the target time and the movement of each of the moving objects on the ground are calculated. The learning device according to claim 5, further comprising a decoder that outputs motion. an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation unit that generates a prediction result of bird's-eye view data representing;
A bird's-eye view data generation device including: The generation unit is
Using a trained model that predicts the movement of the observed moving object on the ground and the distribution of the movement of each of the moving objects on the ground,
A movement trajectory of the observed moving object on the ground, obtained when the observed moving object is observed from a bird's-eye view position, and a position distribution of each of the moving objects on the ground, from the time series data of the two-dimensional observation information. 8. The bird's-eye view data generation device according to claim 7, which generates a prediction result of bird's-eye view data representing a movement trajectory. to the computer,
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating bird's-eye view data representing the
A bird's-eye view data generation program for executing processing including. The computer is
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating bird's-eye view data representing the
A bird's-eye view data generation method that performs processing including. an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from the viewpoint of a robot equipped with an observation device in a dynamic environment;
Using a trained model that estimates the movement of the robot on the ground and the movement of each of the moving objects on the ground,
Bird's-eye view data representing the movement trajectory of the robot on the ground and the movement trajectory of each of the mobile objects on the ground, obtained when the robot is observed from a bird's-eye view position from the time series data of the two-dimensional observation information. a generation unit that generates
an autonomous traveling unit that causes the robot to autonomously travel;
a control unit that uses the bird's-eye view data to control the autonomous traveling unit so that the robot moves to a destination;
including robots. to the computer,
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating a prediction result of bird's-eye view data representing the
A bird's-eye view data generation program for executing processing including. The computer is
an acquisition step of acquiring time-series data of two-dimensional observation information representing at least one moving object observed from a viewpoint of an observation moving object equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the observed moving object on the ground and the movement of each of the moving objects on the ground,
A movement trajectory on the ground of the observed moving object, obtained when the observed moving object is observed from a bird's-eye view position, and a moving trajectory on the ground of each of the moving objects, from the time series data of the two-dimensional observation information. a generation step of generating a prediction result of bird's-eye view data representing the
A bird's-eye view data generation method that performs processing including. an acquisition unit that acquires time-series data of two-dimensional observation information representing at least one moving object observed from the viewpoint of a robot equipped with an observation device in a dynamic environment;
Using a trained model that predicts the movement of the robot on the ground and the movement of each of the moving objects on the ground,
Bird's-eye view data representing the movement trajectory of the robot on the ground and the movement trajectory of each of the mobile objects on the ground, obtained when the robot is observed from a bird's-eye view position from the time series data of the two-dimensional observation information. a generation unit that generates a prediction result of
an autonomous traveling unit that causes the robot to autonomously travel;
a control unit that controls the autonomous traveling unit so that the robot moves to a destination using a prediction result of the bird's-eye view data;
including robots.