JP2020163509A - Simulation system, simulation program and learning device - Google Patents

Abstract

To provide a simulation system which can be used for a sensor system arranged in the real world.SOLUTION: A simulation device of a robot system generates, by a person generation module (110), a pedestrian in a three-dimensional shape from 3D model database (111), and places it in a virtual simulation space. An objective to be executed is selected from an objective model (116), simulation parameters are determined, and a gesture is selected for each object from a gesture database (113), and set to the objective. A movement module (114) updates the pedestrian. According to a social force model (118), the position of the pedestrian is updated, and according to the set gesture, a gesture of the pedestrian is reproduced. As the pedestrian is updated, a data set is output which has a true value label and sensor data applied with a noise model, combined together.SELECTED DRAWING: Figure 5

Description

The present invention relates to a simulation system, and more particularly to a simulation system, a simulation program, and a learning device that simulates a robot or a human agent in a coexistence environment between a robot such as a communication robot and a human.

Conventional simulation devices of this type are disclosed in Non-Patent Documents 1 and 2. In these simulation devices of Non-Patent Documents 1 and 2, by simulating and visualizing sensors and actuators, it is possible to more easily simulate the adaptation of a robot in a coexistence environment with humans.

As a calibration technique for a sensor network installed in an environment, Patent Document 1 and Patent Document 2 are known, which specify the positional relationship between sensors by moving a person or an object in the environment and recording them at the same time. ..

One way to improve the recognition accuracy of people's orientation and gestures in sensor networks is to collect a large amount of behavior data of many people with different genders, heights, physical constitutions, etc., and adjust the parameters of the recognition algorithm. To carry out. However, the sensor arrangement in the sensor network is fixed in the actual field environment, and it varies depending on the introduction situation, so it is necessary to collect a large amount of data in the field in order to collect these behavior data. , It costs a lot of money.

On the other hand, an approach of generating a large amount of data by simulation and using it for learning is known in Non-Patent Document 3 and Non-Patent Document 4.

JP-A-2017-9681 (G01S 7/497 G01S 17/42) Patent No. 6447706 (G01B 11/00 G06T 7/70 G06T 7/80)

https://www.openrobots.org/morse/doc/stable/morse.html http://gazebosim.org/ David Silver et al., Mastering the game of Go with deep neural networks and tree branch, Nature, Vol.529, 484-503, 2016 Mnih V et al., Human-level control through deep reinforcement learning, Nature, Vol.518, No.7540, 529-533, 2015

Approaches such as Non-Patent Document 3 and Non-Patent Document 4 are aimed at objects that can be easily reproduced in a virtual world such as Go (Non-Patent Document 3) and computer games (Non-Patent Document 4), and human behavior. It is difficult to apply recognition to a real-world sensing system.

Therefore, the main object of the present invention is to provide a novel simulation system.

Another object of the present invention is to provide a simulation system capable of generating human data that can be used in real-world sensing systems.

The present invention has adopted the following configuration in order to solve the above problems. The reference numerals and supplementary explanations in parentheses indicate the correspondence with the embodiments described later in order to assist the understanding of the present invention, and do not limit the present invention in any way.

The first invention is a human agent generation means for inputting a human agent into a virtual simulation space simulating a real world in which a plurality of three-dimensional distance image sensors are arranged, an update means for updating a human agent in the virtual simulation space, and a human agent. It is provided with a 3D model database having a plurality of 3D models indicating the three-dimensional shape of the human agent, and an objective model database having an objective of the human agent, and the human agent generation means is a person having a shape according to a 3D model selected from the 3D model database. The agent is generated, the update means moves the person agent according to the objective, and the true value label that creates the true value label of the person agent is created. When the person agent is moved by the update means, multiple 3D distance image sensors are installed. It is a simulation system including a sensor data acquisition means for acquiring simulated center data and an output means for outputting a data set in which a true value label and sensor data are combined.

In the first invention, the simulation system (10: reference code illustrating the corresponding portion in the embodiment; the same applies hereinafter) includes a human agent generating means (26, 110), and the human agent generating means is a 3D model. A human agent is generated according to a 3D model selected from a plurality of 3D models set in the database (111) and put into the depopulated agent simulation space (22). The update means (26, 114) updates the position of the human agent according to the objective selected from the projective model database (116). The sensor data acquisition means (S33) acquires center data that simulates a plurality of three-dimensional distance image sensors when the human agent is moved by the update means. The true value label creating means (S31) creates the true value label (position, orientation, etc. of the human agent) of the agent, and the output means (S35) outputs a data set in which the true value label and the sensor data are combined. If this data set is input to the learner, the 3D distance image sensor can be calibrated.

According to the first invention, the simulation system can efficiently output a large amount of data sets.

The second invention is subordinate to the first invention, and the human agent generating means is a simulation system that generates different types of human agents, each having a shape according to a 3D model.

In the second invention, the human agent generating means (110) generates different types of human agents, such as adults, children, women, and men, respectively, according to a 3D model.

According to the second invention, since various types of human agents can be input to the virtual simulation space, various data sets can be output.

The third invention further comprises a gesture database that is subordinate to the first or second invention and sets different gestures for the human agent, and the updating means causes the human agent to make a gesture selected from the gesture database. It is a simulation system.

In the third invention, the gesture database (113) has a plurality of gestures, and the updating means causes a human agent to make a gesture selected from the gesture database.

According to the third invention, since the human agent is made to make a gesture, it is possible to output various data sets close to the real world.

The fourth invention is subordinate to any one of the first to third inventions and further comprises a social force model database (118) that sets a social force model, and the updating means is that of a human agent according to the social force model. It is a simulation system that updates the position.

According to the fourth invention, since the human agent is moved according to the social force model, it is possible to output various data sets closer to the real world.

A fifth invention is a simulation system that executes a simulation in a virtual simulation space that imitates a real world in which a plurality of three-dimensional distance image sensors are arranged. The simulation system is a plurality of simulation systems that indicate a three-dimensional shape of a human agent. A 3D model database having a 3D model and an objective model database having an objective of a human agent are provided, and a computer of a simulation system is introduced into a virtual simulation space with a human agent having a shape according to a 3D model selected from the 3D model database. The human agent is moved by the human agent generation means, the update means that moves the human agent according to the objective in the virtual simulation space and updates the human agent, the true value label creation means that creates the true value label of the human agent, and the update means. This is a simulation program that functions as a sensor data acquisition means for acquiring center data obtained by simulating a plurality of three-dimensional distance image sensors and an output means for outputting a data set in which a true value label and sensor data are combined.

The same effect as that of the first invention can be expected by the fifth invention.

The sixth invention is a sensor calibration or recognition model learning device including a learning device for learning a data set output by the output means of any one of the first to fifth inventions.

According to the sixth invention, since a large amount of data set can be efficiently obtained from the output means, sensor calibration and learning of the recognition model can be efficiently performed.

According to the present invention, a large amount of various sensor data can be acquired by reproducing various people (pedestrians) in a virtual simulation space. Therefore, if the large amount of sensor data is input to the learner, sensor calibration and recognition model learning can be performed efficiently.

The above-mentioned object, other object, feature and advantage of the present invention will become more apparent from the detailed description of the following examples made with reference to the drawings.

FIG. 1 is a functional block diagram showing a simulation system according to an embodiment of the present invention. FIG. 2 is an illustrated diagram showing an example of a robot to which the present invention is applied. FIG. 3 is a block diagram showing an electrical configuration of the robot shown in FIG. FIG. 4 is an illustrated diagram showing an example of the virtual simulation space of FIG. FIG. 5 is a functional block diagram showing details of the human simulator of FIG. FIG. 6 is an illustrated diagram showing an example of a basic objective model of a human agent. FIG. 7 is a schematic diagram showing another example of the basic objective model of a human agent. FIG. 8 is a schematic diagram showing another example of an objective model of a human agent. FIG. 9 is an illustrated diagram showing an example of the movement of a human agent in a part of the virtual simulation space of FIG. FIG. 10 is a flow chart showing an example of a basic simulation operation in the embodiment of FIG. FIG. 11 is a flow chart showing in detail an example of a pedestrian (person) simulation operation in the embodiment of FIG. FIG. 12 is a flow chart showing an example of an operation of generating a pedestrian in the embodiment of FIG. FIG. 13 is a flow chart showing an example of an operation of updating a pedestrian in the embodiment of FIG. FIG. 14 shows an example of sensing data in the embodiment of FIG. 11, FIG. 14 (A) shows the sensing data (distance image) of each sensor AJ, and FIG. 14 (B) shows the sensing data in which they are integrated. .. FIG. 15 is an illustrated diagram showing an arrangement example of a three-dimensional distance image sensor in a real-world environment corresponding to the virtual simulation space of the embodiment.

With reference to FIG. 1, the simulation system 10 of this embodiment includes a robot 12, an operating device 14 that remotely operates the robot 12, a robot agent that simulates the robot 12, and a human agent that simulates a person (pedestrian). A simulation device 16 for simulating is included. In the simulation system 10, it is realized by one or two or more computers (processors) except for the robot 12.

The robot 12 of this embodiment is a robot that operates in an environment 200 that coexists with humans, as shown in FIG. 15, such as the communication robot shown in FIG. Since such a robot 12 is affected by the behavior of a person existing in the environment, the simulation device 16 of this embodiment is used when simulating the behavior of the robot 12 according to a robot application program (robot behavior determination program). It also supports the development of safe robot application programs for robot 12 by simulating human behavior.

In an environment (real world) 200 where various people come and go, such as a shopping mall, where the robot 12 is active, as shown in FIG. 15, a plurality of sensors 202 are located at a relatively high place such as a ceiling. Is placed. In this embodiment, each sensor 202 assumes a three-dimensional distance image sensor.

Here, with reference to FIGS. 2 and 3, the configuration of the robot 12 will be described to the extent necessary for understanding the present invention. The robot 12 includes a carriage 30, and two wheels 32 and one trailing wheel 34 for moving the robot 12 are provided on the lower surface of the carriage 30. The two wheels 32 are independently driven by a wheel motor 36 (see FIG. 3), and the carriage 30, that is, the robot 12 can be moved in any direction in the front-rear, left-right, and left-right directions.

A cylindrical sensor mounting panel 38 is provided on the trolley 30, and a large number of distance sensors 40 are mounted on the sensor mounting panel 38. These distance sensors 40 measure the distance to an object (such as a person or an obstacle) around the robot 12 by using, for example, infrared rays or ultrasonic waves.

A body 42 is provided upright on the sensor mounting panel 38. Further, the above-mentioned distance sensor 40 is further provided in the upper front center of the body 42 (position corresponding to the chest of a person), and measures the distance in front of the robot 12 mainly to a person. Further, the body 42 is provided with a support column 44 extending from substantially the center of the upper end portion on the side surface side thereof, and an omnidirectional camera 46 is provided on the support column 44. The omnidirectional camera 46 captures the surroundings of the robot 12 and is distinguished from the eye camera 70 described later. As the omnidirectional camera 46, a camera using a solid-state image sensor such as a CCD or CMOS can be adopted.

Upper arms 50R and upper arms 50L are provided at the upper ends of both side surfaces of the body 42 (positions corresponding to human shoulders) by the shoulder joints 48R and shoulder joints 48L, respectively. Although not shown, the shoulder joint 48R and the shoulder joint 48L each have three orthogonal degrees of freedom. That is, the shoulder joint 48R can control the angle of the upper arm 50R around each of the three orthogonal axes. One axis (yaw axis) of the shoulder joint 48R is an axis parallel to the longitudinal direction (or axis) of the upper arm 50R, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is the axis to do. Similarly, the shoulder joint 48L can control the angle of the upper arm 50L around each of the three orthogonal axes. One axis (yaw axis) of the shoulder joint 48L is an axis parallel to the longitudinal direction (or axis) of the upper arm 50L, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is the axis to do.

Further, an elbow joint 52R and an elbow joint 52L are provided at the tips of the upper arm 50R and the upper arm 50L, respectively. Although not shown, the elbow joint 52R and the elbow joint 52L each have a degree of freedom of one axis, and the angles of the forearm 54R and the forearm 54L can be controlled around the axis of this axis (pitch axis).

A hand 56R and a hand 56L corresponding to a human hand are provided at the tips of the forearm 54R and the forearm 54L, respectively. Although not shown in detail, these hands 56R and 56L are configured to be openable and closable so that the robot 12 can grip or hold an object using the hands 56R and 56L. However, the shapes of the hands 56R and 56L are not limited to the shapes of the embodiments, and the hands may have shapes and functions that closely resemble those of human hands.

Although not shown, the front surface of the trolley 30, the portion corresponding to the shoulder including the shoulder joint 48R and the shoulder joint 48L, the upper arm 50R, the upper arm 50L, the forearm 54R, the forearm 54L, the hand 56R and the hand 56L, respectively. , A contact sensor 58 (shown comprehensively in FIG. 3) is provided. The contact sensor 58 on the front surface of the trolley 30 detects the contact of a person or other obstacle with the trolley 30. Therefore, when the robot 12 comes into contact with an obstacle during its own movement, it can detect the contact and immediately stop the driving of the wheels 32 to suddenly stop the movement of the robot 12. In addition, the other contact sensor 58 detects whether or not the respective parts have been touched.

A neck joint 60 is provided at the upper center of the body 42 (a position corresponding to a person's neck), and a head 62 is provided above the neck joint 60. Although not shown, the neck joint 60 has three degrees of freedom, and the angle can be controlled around each of the three axes. A certain axis (yaw axis) is an axis directly above (vertically upward) of the robot 12, and the other two axes (pitch axis and roll axis) are orthogonal axes in different directions.

A speaker 64 is provided on the head 62 at a position corresponding to a person's mouth. The speaker 64 is used by the robot 12 to communicate with people around it by voice. Further, a microphone 66R and a microphone 66L are provided at positions corresponding to human ears. Hereinafter, the right microphone 66R and the left microphone 66L may be collectively referred to as a microphone 66. The microphone 66 captures ambient sounds, especially human voice, which is the object of communication.

Further, a right eyeball portion 68R and a left eyeball portion 68L are provided at positions corresponding to the human eye. The right eyeball portion 68R and the left eyeball portion 68L include a right eye camera 70R and a left eye camera 70L, respectively. Hereinafter, the right eyeball portion 68R and the left eyeball portion 68L may be collectively referred to as an eyeball portion 68. Further, the right eye camera 70R and the left eye camera 70L may be collectively referred to as an eye camera 70.

The eye camera 70 captures a face of a person approaching the robot 12, other parts or objects, and captures a corresponding video signal. In this embodiment, the robot 12 detects the line-of-sight directions (vectors) of the left and right eyes of the person by the video signal from the eye camera 70.

Further, as the eye camera 70, the same camera as the omnidirectional camera 46 described above can be used. For example, the eye camera 70 is fixed in the eyeball portion 68, and the eyeball portion 68 is attached to a predetermined position in the head 62 via an eyeball support portion (not shown). Although not shown, the eyeball support portion has two degrees of freedom, and the angle can be controlled around each of these axes. For example, one of these two axes is an axis (yaw axis) in the upward direction of the head 62, and the other is orthogonal to one axis and orthogonal to the front side (face) of the head 62. It is the axis (pitch axis) in the direction of By rotating the eyeball support portion around each of these two axes, the tip (front) side of the eyeball portion 68 or the eye camera 70 is displaced, and the camera axis, that is, the line-of-sight direction is moved. The installation positions of the speaker 64, the microphone 66, and the eye camera 70 are not limited to the relevant parts, and may be provided at appropriate positions.

As described above, the robot 12 of this embodiment has independent 2-axis drive of the wheels 32, 3 degrees of freedom of the shoulder joint 48 (6 degrees of freedom on the left and right), 1 degree of freedom of the elbow joint 52 (2 degrees of freedom on the left and right). It has a total of 17 degrees of freedom, including 3 degrees of freedom for the neck joint 60 and 2 degrees of freedom for the eyeball support (4 degrees of freedom on the left and right).

FIG. 3 is a block diagram showing an electrical configuration of the robot 12. With reference to FIG. 3, the robot 12 includes one or more processor 80s. The processor 80 is connected to the memory 84, the motor control board 86, the sensor input / output board 88, and the voice input / output board 90 via the bus 82.

The memory 84 includes a ROM, an HDD, and a RAM, although not shown. Various programs are stored in advance in the ROM and HDD.

The motor control board 86 is composed of, for example, a DSP, and controls the drive of each axis motor such as each arm, neck joint 60, and eyeball portion 68. That is, the motor control board 86 receives control data from the processor 80 and controls the angles of the two axes of the right eyeball portion 68R (in FIG. 3, collectively referred to as “right eyeball motor 92”). Control the rotation angle of. Similarly, the motor control board 86 receives control data from the processor 80 and controls the angles of the two axes of the left eyeball portion 68L (in FIG. 3, collectively referred to as the "left eyeball motor 94"). The rotation angle of) is controlled.

Further, the motor control board 86 receives control data from the processor 80, and is a total of three motors that control the angles of the three orthogonal axes of the shoulder joint 48R and one motor that controls the angles of the elbow joint 52R. The rotation angles of the four motors (collectively referred to as "right arm motor 96" in FIG. 3) are controlled. Similarly, the motor control board 86 receives control data from the processor 80, and has three motors that control the angles of the three orthogonal axes of the shoulder joint 48L and one motor that controls the angles of the elbow joint 52L. The rotation angles of a total of four motors (collectively referred to as "left arm motor 98" in FIG. 3) are controlled.

Further, the motor control board 86 receives control data from the processor 80 and controls the angles of the three orthogonal axes of the neck joint 60 (in FIG. 3, collectively referred to as “head motor 100”). ) Rotation angle is controlled. Then, the motor control board 86 receives control data from the processor 80 and controls the rotation angles of the two motors (collectively referred to as “wheel motors 36” in FIG. 3) that drive the wheels 32.

A hand actuator 101 is further coupled to the motor control board 86, and the motor control board 86 receives control data from the processor 80 and controls the opening and closing of the hands 56R and 56L.

Similar to the motor control board 86, the sensor input / output board 88 is composed of a DSP, and takes in signals from each sensor and gives them to the processor 80. That is, data regarding the reflection time from each of the distance sensors 40 is input to the processor 80 through the sensor input / output board 88. Further, the video signal from the omnidirectional camera 46 is input to the processor 80 after being subjected to predetermined processing on the sensor input / output board 88 as needed. The video signal from the eye camera 70 is also input to the processor 80 in the same manner. Further, signals from the plurality of contact sensors 58 (collectively referred to as “contact sensor 58” in FIG. 3) described above are given to the processor 80 via the sensor input / output board 88. Similarly, the voice input / output board 90 is also composed of a DSP, and a voice or a voice according to the voice synthesis data given from the processor 80 is output from the speaker 64. Further, the voice input from the microphone 66 is given to the processor 80 via the voice input / output board 90.

Further, the processor 80 is connected to the communication LAN board 102 via the bus 82. The communication LAN board 102 is composed of, for example, a DSP, and gives transmission data given by the processor 80 to the wireless communication module 104, and the wireless communication module 104 transmits transmission data to a server (not shown) or the like via a network. To do. Further, the communication LAN board 102 receives data via the wireless communication module 104, and gives the received data to the processor 80.

The robot to which the simulation system 10 of the embodiment can be applied is not limited to the robot 12 shown in FIGS. 2 and 3, and this simulation device can also be applied to communication robots of other types and structures. ..

Returning to FIG. 1, the operating device 14 is basically composed of one or more computers or processors, and the robot application program 18 is set in the memory (not shown) of the processor, and the robot application is set. The program is given from the computer, for example, wirelessly to the robot 12 or the simulation device 16. The robot application program 18 is an application program for the robot 12. For example, when the robot 12 is operated as a guide robot, the robot 12 executes the operation for that purpose.

Environmental data 20 is further stored in the memory of the operating device 14 in advance. The environment data 20 mainly includes the map data of the virtual simulation space 22 shown in FIG. 4 and the appearance rate data of the human agent in the virtual simulation space 22. The environment and the objects in the environment are represented as 3D model data and visualized as 3D virtual space. This three-dimensional virtual simulation space 22 reproduces the real world 200 described above.

The environmental data 20 also includes the position, orientation, scanning range (sensing range) of each sensor 202, such as a three-dimensional distance image sensor, set in the real world 200 (FIG. 15).

The virtual simulation space 22 shown in FIG. 4 assumes a three-dimensional closed space having an entrance / exit 22a indicated by a dotted line rectangle in the figure. In this virtual simulation space 22, a passage 22b indicated by a thick black line is set, and the robot 12 and the human agent pass through the passage 50, the open space, and the like, or exist there. These passages 22b and the plaza are divided by objects such as buildings and walls, and branch or merge at the places indicated by circles in the figure.

The simulation device 16 is basically composed of one or two or more processors, and includes a robot simulator 24 that simulates a robot 12 and a human simulator 26 that simulates a human agent such as a pedestrian, and the simulation results of each are included. Is input to the physics engine 28.

The robot simulator 24 is, for example, the MORSE simulator exemplified as Non-Patent Document 1 above, and this MORSE simulator provides many models of sensors, actuators and robots similar to the robot 12, and for such models. API (Application Programming Interface) can also be used.

For example, in the sensor simulation executed in step S33 of FIG. 11, for example, in the case of a distance image sensor, an optical calculation is performed from the camera viewpoint to generate image information for each camera viewpoint. In the case of a laser rangefinder, the distance is measured from the sensor in each direction by imitating a scan by a laser, and the result is saved as a sensor value.

When developers prepare robot application programs for robots, they access these APIs to send sensor data (eg, read distance from a laser range meter) and commands to actuators (eg, read). Movement speed).

The human simulator 26 periodically generates new pedestrians, updates their positions, and removes them as they leave the simulated environment (virtual simulation space 22). A pedestrian or person is represented as a three-dimensional object, placed in the virtual simulator space 22, and its walking motion is animated using an animation engine.

However, since the human simulator 26 is given the environmental data 20 as shown in FIG. 1, the human simulator 26 periodically generates new pedestrians with a predetermined probability called the appearance rate included in the environmental data. To do. Then, the pedestrian is assigned to one of the doorways 22a shown in FIG. 4 and is set to go to the other one.

It should be noted that pedestrians are often generated as groups (like families and couples), and when they are displayed in groups, the size is defined by the distribution of group members, assuming that all members share the same purpose.

The physics engine 28 calculates what kind of movement path a robot or a human agent follows based on a physical force, a law, or the like. More specifically, the physics engine 28 executes a process of executing the movement intended by each agent (including the robot) from the robot simulator 24 and the human simulator 26 in the virtual simulation space 22 (FIG. 4). At this time, if there is no interference with other obstacles, structures or other agents, the intended movement is generated. However, if there is interference, it processes the actual movement of each agent according to the laws of physics (laws of mechanics). Then, the integrated simulation data is output from the physics engine 28.

The simulation device 16 is provided with a display 105, and the display 105 displays, for example, an image of a virtual simulation space as shown in FIG. 4 together with a simulated robot or a human agent existing therein. .. For example, a human agent is represented as a three-dimensional object, placed in a virtual simulation space 22, and its walking motion is animated using an animation engine. The developer of the robot application program 18 creates and modifies the action determination program (application program) of the robot 12 while checking the image displayed on the display 105.

Further, the simulation device 16 is provided with an input device 107. The input device 107 may be, for example, a keyboard or a touch panel. In the case of a touch panel, the input device 107 may be incorporated in the display 105, and the display 105 may be configured as a touch display.

The human simulator 26 will be described in detail with reference to FIG. Like the robot simulator 24, the human simulator 26 is a function realized by a computer or a processor of the simulation device 16, and includes a human generation module 110. As described above, the human generation module 110 represents a new pedestrian (human agent) according to the appearance rate data 108 included in the environmental data 20 in the virtual simulation space 22 shown in FIG. 4 represented by the map data 106. , Generate periodically.

The appearance rate data 108 is a database in which the types (patterns) of people such as adults and children and the appearance rate of each person are preset for the person agent to be input to the virtual simulation space 22 of the simulation device 16. Yes, a human agent with a set shape is generated with a set probability.

A three-dimensional (3D) model database 111 is attached to the human generation module 110. The 3D model database 111 holds a plurality of different shapes, such as gender, height, body shape, etc., for each of the above types of persons. Therefore, the human generation module 110 generates a human agent in the virtual simulation space 22 that follows the instructions of the appearance rate data 108 and that follows the 3D model of the type randomly selected from the 3D model database 111.

The appearance rate data 108 may hold different data sets for each day of the week and time so that the simulation can be performed by designating the day of the week and time. Then, you can simulate various things such as what happens when you move the robot at a certain time.

However, if the appearance rate data 108 indicates that the generated human agents form a group, the human generation module 110 is a plurality of groups having a member configuration according to the group member data 112 set as simulation parameters. Generate a person agent. Again, their body shape follows the 3D model selected from the 3D model database 111.

The movement module 114 moves the human agent generated by the human generation module 110 in the virtual simulation space 22 (FIG. 4) based on the objective preset in the objective model database 116. Here, the objective is the way in which the human agent reacts when the robot agent (not shown) enters the field of view of the human agent (not shown) in the virtual simulation space 22, that is, the person It is an instruction set that defines the action patterns for robots that can actually be taken in real space and the action patterns that can be performed in the environment (even when there is no robot) (for example, stopping in front of a product shelf such as a convenience store). The instruction set includes, for example, a movement target and a movement speed as parameters when a person walks.

In this embodiment, four typical reaction modes (objectives) described later are set as a model, and when the human agent is moved, the movement module 114 is moved according to the objective indicated by the model. That is, the objective 116 functions as a first instruction set that causes a human agent to perform a reaction action to the existence of a robot in a virtual simulation space. These objectives are stored in the first memory (not shown) of the simulation system 10.

It should be noted that the available objectives preset in the objective model 116 are known to the developer who intends to debug, for example, by reading from the first memory and displaying a list thereof on a display 105 or the like. Can be done. The developer can select any objective among them and change (update) the objective with a script (122) as described later. At this time, the developer can change the part requiring change by using an input device (not shown) such as a keyboard provided in the simulation system 10. Then, the changed (updated) objective is also stored in the first memory. That is, since the objective model 116 is designed to have extensibility, it is effective that the objective updated by someone else can be known as list information.

For example, as an action pattern (objective) that a person takes when a robot enters his / her field of view, there are a pattern of interacting with the robot and a pattern of observing the robot, as shown in FIGS. 6 and 7.

FIG. 6 illustrates an objective in which when a person (indicated by “i”) finds a robot 12 in his / her field of view, he / she stops around the robot 12 at a stop distance Dstop from the robot 12 and interacts with the robot 12. are doing.

In FIG. 7, when a person (indicated by “i”) finds the robot 12 in his / her field of view, the robot stops around the robot 12 with an observation distance Dobserve slightly larger than the stop distance Dstop. The objective of observing 12 is illustrated.

Other objectives include an objective that passes by the robot and an objective that slows down by the robot and passes by while looking at it. In the movement module 114, the human agent moves according to one of the registered objectives. To simulate.

However, the movement module 114 simulates the movement of the human agent in consideration of the respective model data from the Social Force Model database 118 and the Interaction with robot model database 120 included in the simulation parameters. To. The social force model is a model showing how pedestrians affect each other and change their behavior, as shown in FIG. 8 described later. For example, it is known as repulsive force or reaction force.

In the virtual simulation space 22, the behavior of a human agent changes under the influence of other agents (for example, avoiding people from colliding with each other, approaching a robot when they see it, etc.).

For example, as shown in FIG. 8, two people i and j may influence each other to determine the movement position in the next step, because the person agent i originally moves in the direction of di, j. However, since the human agent j is trying to move in the direction of the angle θi, j with respect to the human agent i at a velocity vi, j, the human agent i will eventually move in the direction of d'i, j. ..

The interaction model 120 is a model in which a pattern of interaction with a human robot is preset.

FIG. 9 schematically shows a part of the virtual simulation environment (space) shown in FIG. 4, and the virtual simulation space 22 assumes a closed space partitioned by an object 124 such as a wall provided with an entrance / exit 22a. There is. However, all the objects 124 are written in the map data 106 as obstacles.

The robot simulator 24 simulates the behavior of the robot in the next step according to the robot application program 18 based on the data such as the position of the robot (robot agent) included in the environment data 20. Here, the robot application program 18 is a program that controls each motor, that is, an actuator, and determines the behavior of the robot according to the sensor output input to the sensor input / output board 88 of FIG.

In the example of FIG. 9, the robot application program 18 is set to move according to the passage 22b from one entrance / exit 22a until the robot 12 enters the virtual simulation space 22 and exits from the other entrance / exit 22a, for example. Has been done. In FIG. 9, reference numeral 126 indicates a human agent put into the virtual simulation space 22 for simulation.

The human simulator 26 further performs the simulation according to the script 122 instead of the objective 116, if necessary. As described above, the objective 116 is a preset fixed robot reaction pattern, but the script 122 is an instruction to cause the simulation device 16, that is, the human simulator 26 to perform an atypical operation. Here, the script is a simple program (instruction set) that can be easily interpreted and executed by omitting the conversion work into a language that can be understood by a computer. When giving instructions to a computer, such as in software design, it is usually necessary to convert from a language that humans can normally read and write to a language that can be understood by a computer, the so-called machine language. It is a simple program that can be easily interpreted and executed by omitting the conversion work. These scripts correspond to the second instruction set, but they are stored in the second memory (not shown) of the simulation system 10. However, the first memory and the second memory may be different storage locations of the same memory.

The human simulator 26 is further attached with the gesture model database 113. A plurality of types of gestures (movements) that a person (pedestrian) can take are registered in this gesture model database 113. The gestures include, for example, looking at things, walking while looking at things, picking things (right and / or left hands), walking (normally or slowly, fast), stopping, crouching, and so on. Gesture data may include data generated using a motion capture system in order to generate data similar to human gestures that occur in the real world.

Therefore, the movement module 114 of the person simulator 26 moves the person agent performing the gesture randomly selected from the gesture model database 113 in the movement pattern or movement path according to the objective model 116 or the script 122.

Here, for the purpose of facilitating the understanding of the embodiment of the present invention shown in FIG. 11, first, an example of the operation in the simulation device 16 shown in FIG. 1 according to the objective model 116 will be described with reference to FIG. .. However, in the simulation shown in FIG. 10, a plurality of steps (S3-S17) are executed as one step. In practical operation, one step of the simulation can be executed in units of 0.1 milliseconds at the fastest, but here it is set in units of 10 milliseconds, and the execution of the robot simulation program 18 can be executed in units of 100 milliseconds. It is possible. In the case of this example, the robot application program 18 is executed once every 10 steps executed by the simulation device 16. That is, in this embodiment, it is assumed that the cycle of the robot application program is longer than the execution cycle of one step of the simulation.

In the first step S1, as an example, the robot application program 18 set to move on the passage 22b shown in FIG. 4 (and FIG. 9) is read, and the environment data 20 is read. That is, a parameter (appearance frequency) related to the generation of a human agent to be input to the virtual simulation space 22 indicated by the map data 106 and the appearance rate data 108 of the virtual simulation space 22 as shown in FIG. 4 (and FIG. 9) that simulates the behavior of the robot 12. , Movement route, number of groups, ratio of adults and children and men and women, height, etc.) and position data of each agent (including robot 12). The virtual simulation space 22 is displayed on the display 105 together with the robot 12 and the human agent.

Further, in this step S1, the simulation is initialized. That is, the execution speed is set to s = 1.0, and the number of basic execution steps fps is set to fps = 30.0. However, fps is a numerical value indicating how many steps of simulation are executed per second, and the initial setting of this embodiment is set so as to execute the simulation of 30 steps per second.

In the next step S3, one or two in the virtual simulation space 22 according to the above parameters and data of the person pattern from the group regarding the generation of the person agent presented by the environment data 20 by the person generation module 110 of the person simulator 26. Create and deploy the above person agents.

In step S5, the human simulator 26 performs the behavior of the human agent in the next step based on the current positions of the human agent and the robot 12 shown by the environmental data 20 and based on the anti-robot behavior pattern shown by the objective model 116. decide. The basic list of the above-mentioned four objectives is provided in a form that can be freely accessed by the developer (operator of the operating device 14). In addition, other objective models can be adopted.

In step S7, the robot simulator 24 determines the action of the robot 12 in the next step according to the robot application program 18 based on the current positions of the human agent and the robot 12 indicated by the environmental data 20.

Specifically, in step S7, when the robot 12 is a robot whose main purpose is to go to the destination, such as a transfer robot or a boarding robot, it is possible to consider another person from the current position and movement speed of other people. Performs a process of calculating a movement route that does not interfere with the movement of. Further, when the robot 12 is a robot whose purpose is to provide services to people, for example, a person who is stopped near the robot in order to approach a person who may be interested in the robot. Choose to go towards that person, choose a place where people tend to gather, wait for people to come in such a place, or get too close to a place where people pass too much to avoid congestion. It performs processing such as preventing it from occurring.

In the case of this step S7, since the action in the next step of the human agent in step S5 is determined, the above-mentioned process is executed in consideration of it. Therefore, it is possible to judge in the virtual simulation space whether or not the robot application program that operates in the real space coexisting with humans is appropriate.

After that, in step S9, the action of each agent is realized in the physics engine 28. More specifically, in step S9, as described above, the process of executing the movement intended by each agent (including the robot) in the virtual simulation space 22 is performed, and other obstacles, structures and other agents are performed. If there is no interference with, each agent works as intended. For example, if the agent tries to take one step forward, the agent takes one step forward as intended.

However, when there is interference, the actual movement of each agent is processed according to the laws of physics (law of mechanics). In a simple case, for example, if you try to move toward a wall, it will collide with the wall, you will not be able to move forward any further, and you will receive a reaction force according to a constant coefficient of restitution. In the case of a collision between agents, the forward movement and the movement due to the reaction force overlap, for example, considering a simple example, if a collision occurs from the side while moving forward, it will actually move diagonally forward. ..

In the next step S11, the environment data, the sensor data, and the like are updated.

Then, it is determined whether or not to end the simulation in step S13, and if the simulation is continued, the process returns to the previous step S3 again. For example, if the action determination program is terminated, "YES" is determined and the simulation is terminated.

In this way, the robot behavior can be simulated by performing simulation in the simulation device 16 so as to reproduce the behavior of people around the robot according to the objective 116 and confirming the state on the display 105 (FIG. 1). It is possible to efficiently develop a robot application program 18 that operates in a real space that coexists with humans without actually moving the robot in the real space.

FIG. 11 is a flow chart mainly showing the operation of the human simulator 26 in this embodiment, and in the first step S21 of FIG. 11, the computer determines whether or not the simulation is being executed. If "NO", it ends as it is.

The features of this embodiment are the generation of a human agent by the human simulator 26 in step S3 of FIG. 10, the input to the virtual simulation space 22, the movement of the human agent in the virtual simulation space 22 in step S5, the gesture, and the like. It is to acquire the simulation data corresponding to the three-dimensional distance image sensor 202 (FIG. 15) at the time of action. This will be described below.

When "YES" is determined in step S21, in the next step S23, the computer determines whether or not the pedestrian (person) generation condition is satisfied. When the number of pedestrians existing in the virtual simulation space 22 is less than a predetermined maximum number, the pedestrians are generated with a certain probability according to the appearance rate data 108 (FIG. 5). Therefore, the pedestrian generation condition is a condition that the appearance rate is reached.

If "YES" is determined in step S23, a pedestrian is generated by the human generation module 110 in step S25. Specifically, a pedestrian is generated according to the flow chart shown in FIG.

In step S41 of FIG. 12, the computer or human generation module 110 randomly selects a pedestrian model from the 3D model database 111 or according to a predetermined rule.

Then, in step S43, the objective (behavior type) is randomly selected from the objective model 116 or according to a predetermined rule, and the simulation parameters are deterministically or predetermined according to the group member data 112, the social force model 118, and the interaction model 120. Determined according to the law of. In this step S43, a plurality of objectives can be selected, and in that case, the objectives are executed in the selected order.

In step S45, for each objective, the gesture to be executed is randomly selected from the gesture database 113 or according to a predetermined rule and set as the objective. Gestures include "taking things" as explained earlier. A plurality of gestures may be selected for each objective. Then return.

In this way, in step S25 of FIG. 11, a human agent (pedestrian) can be generated in the virtual simulation space 22 (FIG. 4). Therefore, this step S25 and the computer that executes it can be said to be a human agent generation means.

In step S27 of FIG. 11, a loop for each pedestrian Pi is executed, and in step S29, the computer, that is, the movement module 114 updates the pedestrian Pi. Specifically, it follows the flow chart shown in FIG.

In step S51 of FIG. 13, the computer determines whether the objective has ended. If "YES", the pedestrian Pi is deleted (erased) from the environment, that is, the virtual simulation space 22, and the pedestrian Pi is returned. In other words, since the movement sequence is set for each pedestrian, it disappears from the environment after that.

On the other hand, when it is determined as "NO" in step S51, the position of the pedestrian is updated in step S55 according to the social force model previously set in step S43. Then, in step S57, the gesture of the pedestrian Pi is reproduced according to the gesture set in step S45, and then the gesture is returned.

When a plurality of objectives exist, the update process of step S27 sequentially executes the update process of the next objective. Since the update in step S27, that is, step S29 updates the position of the human agent by applying the movement sequence of the human agent in the virtual simulation space, the steps S27 and S29 and the computer that executes them are referred to as update means. be able to.

When the update process of each pedestrian Pi is completed, the true value label of each pedestrian is created in step S31 of FIG. The true value label is the appearance of each pedestrian (person) from each sensor, and includes, for example, the position of the pedestrian, the direction of the pedestrian, the gesture, and the like. Since the pedestrian movement sequence is simulated by the human simulator 26, the computer can easily obtain such a true value label from each model and parameter.

Then, in step S33, the computer creates sensor data. The sensor data is data simulating the appearance of the pedestrian Pi from each sensor when the pedestrian Pi is simulated as described above, for example, a distance image. When a three-dimensional distance image sensor is used as the sensor 202 (FIG. 15), each distance image of the sensors AJ is created as shown in FIG. 14 (A). By integrating the distance images for each sensor AJ in FIG. 14 (A), the distance image as shown in FIG. 14 (B) can be obtained. Since sensor noise exists in real-world sensing, a sensor noise model for reproducing these sensor noises is applied when creating distance image data. That is, the result of adding (or subtracting) probabilistic sensor noise (Gaussian distribution, etc.) to the distance measured in the simulation is generated as a distance image. A noise model of such a sensor is preset in, for example, environmental data 20 (FIG. 1) together with data of a three-dimensional distance image sensor arranged in the real world.

In the next step S35, the computer outputs a data set in which the true value label data acquired in step S31 and the distance image as shown in FIG. 14 created in step S33 are combined.

By inputting this data set to a learner (not shown), it is possible to calibrate the sensor and learn the recognition model. However, sensor calibration means determining the calculation method of sensor data indicating what kind of calculation should be performed for each sensor 202 to improve the accuracy of sensing, and for learning by the learner. A large number of data sets, for example one million or more, are required.

Attempting to acquire an extremely large number of data sets in the real world requires a large amount of data that behaves in various places (environments) and patterns, and a large amount of the label sets, which requires a great deal of cost.

On the other hand, if a pair of true value label and sensor data is generated by simulation as in this embodiment, various human behaviors are generated by a human (pedestrian) behavior model in the simulation, and those are generated. By combining a person's three-dimensional model, human motion (gesture) data, and sensor noise model with behavior, it is possible to acquire accuracy improvement data that can be used in a real-world sensor system at low cost and in a short time.

Apart from the learner that learns the parameters of such a recognition model, the approach of creating a large data set of this example by simulation can also be applied to deep learning (so-called deep learning). Deep learning is, so to speak, a learning device that learns the recognition model itself. Therefore, the data set obtained by the embodiment can be used for sensor calibration and a learning device (deep learning) aiming at acquisition of a recognition model.

In the above embodiment, the human agent to be input to the virtual simulation space is shaped according to the 3D model, moved according to the objective, and further motions according to the gesture model, so that the human agent is actually Since a simulation that closely resembles a person (pedestrian) in the world can be executed, the sensing accuracy of the 3D distance image sensor that senses such a person agent can be reproduced with the same accuracy as that in the real world. Therefore, the accuracy of the learner (not shown) for learning with the data set output in step S35 is also improved.

However, if sufficient accuracy can be achieved by simply shaping the human agent to be input into the virtual simulation space according to the 3D model and moving it according to the objective, it is possible to prevent the human agent from making gestures in the virtual simulation space. Is.

Similarly, it is possible to learn without giving simulation parameters to human agents.

10 ... Simulation system 12 ... Robot 14 ... Operating device 16 ... Simulation device 18 ... Robot application program 22 ... Virtual simulation space 24 ... Robot simulator 26 ... Human simulator 28 ... Physics engine 105 ... Display 202 ... Distance image sensor

Claims (6)

A human agent generation means that puts a human agent into a virtual simulation space that imitates the real world in which multiple 3D distance image sensors are arranged.
An update means for updating the person agent in the virtual simulation space,
A 3D model database having a plurality of 3D models indicating the three-dimensional shape of the human agent and an objective model database having the objective of the human agent are provided.
The human agent generating means generates a human agent having a shape according to a 3D model selected from the 3D model database, the updating means moves the human agent according to the objective, and further, the true value label of the human agent is displayed. True value label creation method to be created,
When the person agent is moved by the update means, the sensor data acquisition means for acquiring the center data simulating the plurality of three-dimensional distance image sensors, and the data set in which the true value label and the sensor data are combined are output. A simulation system with output means. The simulation system according to claim 1, wherein the human agent generating means generates different types of human agents, each having a shape according to a 3D model. Further equipped with a gesture database that sets different gestures for the person agent,
The simulation system according to claim 1 or 2, wherein the updating means causes the person agent to perform a gesture selected from the gesture database. Further equipped with a social force model database that sets a social force model,
The simulation system according to any one of claims 1 to 3, wherein the updating means updates the position of the person agent according to the social force model. A simulation system that executes simulations in a virtual simulation space that imitates the real world in which multiple 3D distance image sensors are arranged.
The simulation system includes a 3D model database having a plurality of 3D models indicating the three-dimensional shape of the human agent, and an objective model database having the objective of the human agent, and the computer of the simulation system is used.
A human agent generation means for inputting a human agent having a shape according to a 3D model selected from the 3D model database into the virtual simulation space.
An update means for moving the person agent according to the objective in the virtual simulation space and updating the person agent.
A true value label creation means for creating a true value label for the person agent,
When the person agent is moved by the update means, the sensor data acquisition means for acquiring the center data simulating the plurality of three-dimensional distance image sensors, and the data set in which the true value label and the sensor data are combined are output. A simulation program that functions as an output means. A sensor calibration or recognition model learning device comprising a learning device that learns a data set output by the output means according to any one of claims 1 to 5.