JP6919882B2 - Person estimation system and estimation program - Google Patents

Description

The present invention relates to a human estimation system and an estimation program, and in particular, a human estimation system and an estimation program that estimate a person's position using three-dimensional scan data acquired by, for example, a three-dimensional laser range finder mounted on a mobile robot.

In order for robots and other mobile machines to operate in a coexisting environment with humans, it is important to know their own position and the positions and characteristics of those around them. When using a general camera as a robot sensor, privacy may be asked, but a 3D rangefinder (3D laser rangefinder) that measures only the distance does not have to worry about that, so it is possible to interact with people. It is a sensor suitable for coexistence.

Patent Document 1 discloses a method of detecting a person using a three-dimensional laser range finder and estimating the position thereof.

Patent No. 5935484 [G01S 17/88]

The background technique of Patent Document 1 is based on the premise that all moving objects are human beings, and false detections associated therewith are unavoidable.

Therefore, the main object of the present invention is to provide a novel human estimation system and estimation program.

Another object of the present invention is to provide a human estimation system and an estimation program capable of suppressing false positives as much as possible.

The present invention has adopted the following configuration in order to solve the above problems. The reference numerals and supplementary explanations in parentheses indicate the correspondence with the embodiments described later in order to help the understanding of the present invention, and do not limit the present invention in any way.

The first invention is a moving robot, a three-dimensional laser distance meter mounted on the robot, a three-dimensional distance image data from the three-dimensional laser distance meter, and a three-dimensional environment map data, and the three-dimensional distance of a moving object is compared. Extraction unit that extracts image data, 3D distance image data of moving objects Clustering unit that generates 3D point groups of moving objects, 2D distance image data of moving objects by projecting 3D point groups It is equipped with a projection unit that generates a 2D distance image data and a deep neural network that inputs 2D distance image data. A human estimation system that determines the pixel value of each pixel based on the average distance of a point from the plane of projection, and a deep neural network distinguishes between people and objects.

In the first invention, the human estimation system includes a moving robot (10: a reference reference numeral exemplifying a corresponding portion in the embodiment; the same applies hereinafter), and the robot is equipped with a three-dimensional laser range finder (26). Will be done. The extraction unit (60, 94, S1) compares the 3D distance image data from the 3D laser range finder with the 3D environment map data and extracts the 3D distance image data of the moving object. The clustering unit (60, 94, S3) clusters the three-dimensional distance image data of the extracted moving object to generate a three-dimensional point cloud of the moving object. The projection unit (60, 94, S5) generates two-dimensional distance image data of a moving object by projecting a three-dimensional point cloud, and the two-dimensional distance image data is input to the deep neural network (S7). Specifically, the projection unit sets a projection plane perpendicular to the line from the 3D laser rangefinder, and calculates the pixel value of each pixel based on the average distance from the projection plane of all the points that enter each pixel. decide. Then, in the deep neural network, a person and an object are discriminated based on the two-dimensional distance image data. In addition, deep neural networks can also identify human characteristics (eg, gender, orientation, etc.).

According to the first invention, by extracting a three-dimensional distance image of an object that is not on the three-dimensional environment map, it is possible to suppress erroneous recognition of a part of the environment (fixed object) as a moving object. Will be done. Moreover, a recognition program that detects whether a person or an object is detected from the shape of a three-dimensional moving object can be realized by a method such as deep learning, and it is possible to detect only a person while ignoring the object.

The second invention is subordinate to the first invention and is a human estimation system in which the deep neural network is a convolutional neural network.

According to the second invention, since the convolutional neural network is used as the deep neural network (DNN), human estimation can be easily performed.

The third invention is an estimation program executed by a computer of a human estimation system including a mobile robot equipped with a three-dimensional laser distance meter, in which the estimation program makes the computer three-dimensional from the three-dimensional laser distance meter. An extraction unit that compares the distance image data with the 3D environment map data to extract the 3D distance image data of the moving object, and clusters the 3D distance image data of the moving object to generate a 3D point group of the moving object. The clustering unit functions as a projection unit that generates 2D distance image data of a moving object by projecting a 3D point group, and a deep neural network that inputs 2D distance image data, and the projection unit is a 3D laser. Set a plane of projection perpendicular to the line from the distance meter, determine the pixel value of each pixel based on the average distance from the plane of projection of all points that enter each pixel, and a deep neural network connects people to objects. An estimation program that identifies.

The fourth invention is an estimation method executed by a computer of a human estimation system including a mobile robot equipped with a 3D laser distance meter, and is a 3D distance image data from a 3D laser distance meter and a 3D environment. Extraction step to extract 3D distance image data of moving object by comparing with map data, clustering step to generate 3D point group of moving object by clustering 3D distance image data of moving object, 3D point group projection step of generating a 2-dimensional range image data of the moving object by projecting, and the 2-dimensional range image data viewed including the step of introducing the deep neural network, in the projection step, the lines of the three-dimensional laser rangefinders An estimation method that sets a vertical plane of projection, determines the pixel value of each pixel based on the average distance from the plane of projection of all points that enter each pixel, and a deep neural network distinguishes between people and objects. be.

According to the present invention, it is possible to prevent misidentification that a part of the environment is a moving object, so that false detection can be suppressed as much as possible.

The above-mentioned object, other object, feature and advantage of the present invention will become more apparent from the detailed description of the following examples made with reference to the drawings.

FIG. 1 is an illustrated diagram showing an example of a robot used in a human estimation system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the electrical configuration of the robot according to the embodiment of FIG. FIG. 3 is an illustrated diagram showing an example of the moving space of the robot according to the embodiment of FIG. FIG. 4 is a block diagram showing an example of a remote control device for a robot according to an embodiment of FIG. FIG. 5 is an illustrated diagram showing an example of a memory map of the memory of the robot in the embodiment of FIG. FIG. 6 is an illustrated diagram showing a part of the moving space of the robot according to the embodiment of FIG. FIG. 7 is an illustrated diagram showing an example of actual scan data of an omnidirectional laser range finder mounted on a robot. FIG. 8 is a flow chart showing an example of the operation of the estimation system using the robot of the embodiment of FIG. FIG. 9 is an illustrated diagram showing an example of a three-dimensional point cloud of a moving object as a result of clustering three-dimensional distance image data of a portion not on the map in the embodiment of FIG. FIG. 10 is an illustrated diagram showing an example of a two-dimensional distance image in which a three-dimensional point cloud of the moving object shown in FIG. 9 is projected in the embodiment of FIG. FIG. 11 is an illustrated diagram showing an example of a method of projecting a three-dimensional point cloud of a moving object shown in FIG.

FIG. 1 shows an example of a robot 10 used in an estimation system according to an embodiment of the present invention. In this embodiment, the estimation system uses a three-dimensional range finder (three-dimensional omnidirectional laser range finder) mounted on the robot 10 to simultaneously estimate the self-position of the robot 10 and the position, orientation, or attribute of a person. It is possible.

The robot 10 includes a trolley 12, and two wheels 14 for moving the robot 10 and one trailing wheel 16 are provided on the lower surface of the trolley 12. The two wheels 14 are independently driven by a wheel motor 82 (see FIG. 2), and the dolly 12, that is, the robot 10 can be moved in any direction of front, back, left, and right.

A cylindrical sensor mounting panel 18 is provided on the trolley 12, and a large number of distance sensors 20 are mounted on the sensor mounting panel 18. These distance sensors 20 measure the distance to an object (such as a person or an obstacle) around the robot 10 by using, for example, infrared rays or ultrasonic waves.

A body 22 is provided upright on the sensor mounting panel 18. Further, the above-mentioned distance sensor 20 is further provided in the upper front center of the body 22 (position corresponding to the chest of a person), and measures the distance in front of the robot 10 mainly to a person. Further, the body 22 is provided with a support column 24 extending from substantially the center of the upper end portion on the side surface side thereof, and an omnidirectional laser rangefinder 26 is provided on the support column 24. The omnidirectional laser range finder 26 is, for example, a three-dimensional laser range finder having a detection angle (vertical viewing angle) of 40 ° (+ 30 ° −-10 °) up and down with reference to the horizontal. This three-dimensional omnidirectional laser range finder (hereinafter, may be simply referred to as "laser range finder") 26 rotates once every 0.1 seconds, for example, and measures a distance of up to about 100 m. In the experiment, an imaging unit LiDAR (HDL-32e) (trade name) manufactured by Velodine was used as the laser range finder 26.

Although not shown, the support column 24 may be further provided with an omnidirectional camera. The omnidirectional camera captures the surroundings of the robot 10 and is distinguished from the eye camera 50 described later. As the three omnidirectional cameras, for example, a camera using a solid-state image sensor such as a CCD or CMOS can be adopted.

The upper arm 30R and the upper arm 30L are provided by the shoulder joint 28R and the shoulder joint 28L, respectively, at the upper ends of both side surfaces of the body 22 (positions corresponding to the shoulders of a person). Although not shown, the shoulder joint 28R and the shoulder joint 28L each have three orthogonal degrees of freedom. That is, the shoulder joint 28R can control the angle of the upper arm 30R around each of the three orthogonal axes. One axis (yaw axis) of the shoulder joint 28R is an axis parallel to the longitudinal direction (or axis) of the upper arm 30R, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is the axis to do. Similarly, the shoulder joint 28L can control the angle of the upper arm 30L around each of the three orthogonal axes. One axis (yaw axis) of the shoulder joint 28L is an axis parallel to the longitudinal direction (or axis) of the upper arm 30L, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is the axis to do.

Further, an elbow joint 32R and an elbow joint 32L are provided at the tips of the upper arm 30R and the upper arm 30L, respectively. Although not shown, the elbow joint 32R and the elbow joint 32L each have a degree of freedom of one axis, and the angles of the forearm 34R and the forearm 34L can be controlled around the axis of this axis (pitch axis).

A hand 36R and a hand 36L corresponding to a human hand are provided at the tips of the forearm 34R and the forearm 34L, respectively. These hands 36R and 36L are configured to be openable and closable, although detailed illustration is omitted, so that the robot 10 can grasp or hold an object using the hands 36R and 36L. However, the shapes of the hands 36R and 36L are not limited to the shapes of the embodiments, and may have shapes and functions that closely resemble those of human hands.

Further, although not shown, the front surface of the carriage 12, the portion corresponding to the shoulder including the shoulder joint 28R and the shoulder joint 28L, the upper arm 30R, the upper arm 30L, the forearm 34R, the forearm 34L, the hand 36R and the hand 36L, respectively. , A contact sensor 38 (shown comprehensively in FIG. 2) is provided. The contact sensor 38 on the front surface of the dolly 12 detects the contact of a person or other obstacle with the dolly 12. Therefore, when the robot 10 comes into contact with an obstacle during its own movement, the robot 10 can detect the contact and immediately stop the driving of the wheels 14 to suddenly stop the movement of the robot 10. In addition, the other contact sensor 38 detects whether or not the respective parts have been touched.

A neck joint 40 is provided at the upper center of the body 22 (a position corresponding to a person's neck), and a head 42 is provided above the neck joint 40. Although not shown, the neck joint 40 has three degrees of freedom, and the angle can be controlled around each of the three axes. A certain axis (yaw axis) is an axis that goes directly above the robot 10 (vertically upward), and the other two axes (pitch axis, roll axis) are axes that are orthogonal to each other in different directions.

A speaker 44 is provided on the head 42 at a position corresponding to a person's mouth. The speaker 44 is used by the robot 10 to communicate with people around it by voice. Further, a microphone 46R and a microphone 46L are provided at positions corresponding to human ears. Hereinafter, the right microphone 46R and the left microphone 46L may be collectively referred to as a microphone 46. The microphone 46 captures ambient sounds, especially human voice, which is the object of communication.

Further, a right eyeball portion 48R and a left eyeball portion 48L are provided at positions corresponding to the human eye. The right eyeball portion 48R and the left eyeball portion 48L include a right eye camera 50R and a left eye camera 50L, respectively. Hereinafter, the right eyeball portion 48R and the left eyeball portion 48L may be collectively referred to as the eyeball portion 48. Further, the right eye camera 50R and the left eye camera 50L may be collectively referred to as an eye camera 50.

The eye camera 50 captures a face of a person approaching the robot 10, other parts or objects, and captures a corresponding video signal. In this embodiment, the robot 10 detects the line-of-sight directions (vectors) of the left and right eyes of a person by the video signal from the eye camera 50.

Further, as the eye camera 50, the same camera as the above-mentioned three-dimensional distance image meter 26 can be used. For example, the eye camera 50 is fixed in the eyeball portion 48, and the eyeball portion 48 is attached to a predetermined position in the head 42 via an eyeball support portion (not shown). Although not shown, the eyeball support portion has two degrees of freedom, and the angle can be controlled around each of these axes. For example, one of these two axes is an axis (yaw axis) in the upward direction of the head 42, and the other is orthogonal to one axis and orthogonal to the front side (face) of the head 42. It is the axis (pitch axis) in the direction of By rotating the eyeball support portion around each of these two axes, the tip end (front) side of the eyeball portion 48 or the eye camera 50 is displaced, and the camera axis, that is, the line-of-sight direction is moved. The installation positions of the speaker 44, the microphone 46, and the eye camera 50 are not limited to the relevant parts, and may be provided at appropriate positions.

As described above, the robot 10 of this embodiment has independent 2-axis drive of the wheels 14, 3 degrees of freedom of the shoulder joint 28 (6 degrees of freedom on the left and right), 1 degree of freedom of the elbow joint 32 (2 degrees of freedom on the left and right). It has a total of 17 degrees of freedom, including 3 degrees of freedom for the neck joint 40 and 2 degrees of freedom for the eyeball support (4 degrees of freedom on the left and right).

FIG. 2 is a block diagram showing an electrical configuration of the robot 10. With reference to FIG. 2, the robot 10 includes one or more computers 60. The computer 60 is connected to the memory 64, the motor control board 66, the sensor input / output board 68, and the voice input / output board 70 via the bus 62.

The memory 64 includes a ROM, an HDD, and a RAM, although not shown. As will be described later, various programs and data are stored in the ROM and HDD in advance, and are also used as a buffer area or a working area for the computer 60.

The motor control board 66 is composed of, for example, a DSP, and controls the drive of each axis motor such as each arm, neck joint 40, and eyeball portion 48. That is, the motor control board 66 receives control data from the computer 60 and controls the angles of the two axes of the right eyeball portion 48R (in FIG. 2, collectively referred to as "right eyeball motor 72"). Control the rotation angle of. Similarly, the motor control board 66 receives control data from the computer 60 and controls the respective angles of the two axes of the left eyeball portion 48L (in FIG. 2, collectively referred to as "left eyeball motor 74"). The rotation angle of) is controlled.

Further, the motor control board 66 receives control data from the computer 60, and is a total of three motors that control the angles of the three orthogonal axes of the shoulder joint 28R and one motor that controls the angles of the elbow joint 32R. The rotation angles of the four motors (collectively referred to as "right arm motor 76" in FIG. 2) are controlled. Similarly, the motor control board 66 receives control data from the computer 60, and includes three motors that control the angles of the three orthogonal axes of the shoulder joint 28L and one motor that controls the angles of the elbow joint 32L. The rotation angles of a total of four motors (collectively referred to as "left arm motor 78" in FIG. 2) are controlled.

Further, the motor control board 66 receives control data from the computer 60 and controls the angles of the three orthogonal axes of the neck joint 40 (in FIG. 2, collectively referred to as "head motor 80"). ) Rotation angle is controlled. Then, the motor control board 66 receives the control data from the computer 60 and controls the rotation angles of the two motors (collectively referred to as "wheel motor 82" in FIG. 2) for driving the wheels 14.

Similar to the motor control board 66, the sensor input / output board 68 is composed of a DSP, and takes in signals from each sensor and gives them to the computer 60. That is, data regarding the reflection time from each of the distance sensors 20 is input to the computer 60 through the sensor input / output board 68.

The distance image signal from the three-dimensional distance image meter 26 is subjected to predetermined processing on the sensor input / output board 68 as needed, and then input to the computer 60 as distance image data.

The video signal from the eye camera 50 is also input to the computer 60 in the same manner. Further, signals from the plurality of contact sensors 38 described above (collectively referred to as “contact sensors 38” in FIG. 2) are given to the computer 60 via the sensor input / output board 68.

Similarly, the voice input / output board 70 is also composed of a DSP, and a voice or a voice according to the voice synthesis data given from the computer 60 is output from the speaker 44. Further, the voice input from the microphone 46 is given to the computer 60 via the voice input / output board 70.

Further, the computer 60 is connected to the communication LAN board 84 via the bus 62. The communication LAN board 84 is composed of, for example, a DSP, and supplies transmission data given by the computer 60 to the wireless communication module 86, and the wireless communication module 86 transmits the transmission data via a network to a remote control device (not shown) or the like. Send to. Further, the communication LAN board 84 receives data via the wireless communication module 86, and gives the received data to the computer 60.

Such a robot 10 moves along a movement path 88b between entrances and exits 88a of the movement space 88 shown in FIG. 3, for example, and provides a guidance service to a person existing in the movement space 88 such as a commercial facility. Provide services such as.

Although not shown, a plurality of three-dimensional omnidirectional laser range finders similar to those of the robot 10 are arranged in the moving space 88 as sensors on the environment side.

The robot 10 that moves in such a moving space 88 can basically move autonomously, but in preparation for a situation where the robot 10 cannot move autonomously, for example, the remote control device 90 as shown in FIG. 4 It is also possible to control by.

With reference to FIG. 4, the remote control device 90 of this embodiment can communicate with the robot 10 via the network 92 in order to control the operation and movement of the robot 10 as a mobile body.

The remote control device 90 includes a computer 94, and the operator operates an operation key or a joystick (not shown) provided on the console 96 connected to the computer 94, and the computer 94 responds to the operation input. Can control the movement of the wheels 14 of the robot 10, that is, the robot 10.

A memory 98 is connected to the computer 94, and a display 10 is connected to the computer 94. The memory 98 stores necessary data such as a program for robot control, two-dimensional map data of the moving space 88 (FIG. 3), and three-dimensional map data. In order to safely move the robot 10, the display 100 displays the moving state of the robot 10 and an image taken by an omnidirectional camera (not shown) and shows it to the operator.

A wireless communication module 102 is attached to the computer 94, and the computer 94 can perform wireless communication with the computer 60 via the wireless communication module 86 (FIG. 2) of the robot 10 via the network 92.

Here, the memory 64 of the computer 60 of the robot 10 of FIG. 1 includes a program storage area 104 and a data storage area 106, for example, as shown in FIG.

In the program storage area 104, in addition to necessary programs such as an OS, a movement control program 108 for controlling the movement of the robot 10, an estimation program 110 which will be described in detail later, and the like are set in advance. The estimation program 110 estimates the characteristics of a person moving in the moving space 88, and includes a deep neural network (for example, a convolutional neural network) for that purpose.

The data storage area 106 includes a three-dimensional distance image data storage area 112 for temporarily storing the three-dimensional distance image data (data set) obtained from the laser distance meter 26 (FIGS. 1 and 2), and a remote control device. The three-dimensional map data storage area 114 for storing the data of the three-dimensional map (three-dimensional environment map) of the moving space 88 acquired from the memory 98 of the 90 is included.

Further, the data storage area 106 further stores the data of the three-dimensional point cloud of the moving object, which will be described later, obtained in the processing process according to the estimation program 110, the three-dimensional point cloud data storage area 116, and the two-dimensional distance image data of the moving object. A two-dimensional distance image data storage area 118 for storing the data is formed.

Such a robot 10 acquires scan data by a laser range finder 28 while being moved at a place as shown in FIG. 3, for example. The place illustrated in FIG. 6 is a part of the second floor of the commercial facility used by the inventors for the experiment of the estimation system. In the experiment, the robot 10 moves about 530 m in this place of the commercial facility in about 25 minutes in one run, and outputs the position data (data set) of about 182 million point clouds.

FIG. 7 shows typical three-dimensional scan data 120 acquired by the laser range finder 28, and the arcuate line in FIG. 7 schematically shows how the laser range finder 28 is scanning. As shown in FIG. 7, scan data includes data from dynamic objects such as humans as well as static objects such as walls and floors. On the other hand, although it is not clear in FIG. 7, in the actual scan data, the low position is colored in red and the high position is represented in blue in the actual scan data. It is stored in the dimension distance image data storage area 112 (FIG. 5).

In step S1 shown in FIG. 8, the self-position is first identified using such three-dimensional distance image data and the three-dimensional environment map data stored in advance in the three-dimensional map data storage area 114 of the memory 64. .. Specifically, the distance between the position of the wall (indicated by "124" in the scan data 122 of FIG. 7) recorded as a fixed object (stationary object) on the three-dimensional environment map and the position of the robot 10 is (3). The current position (coordinate position) of the robot 10 can be specified by calculation (based on the principle of angular measurement).

However, the calculation for specifying the current position of the robot 10 in step S1 uses a plurality of three-dimensional omnidirectional laser range finders (not shown) distributed in the moving space 88 (FIG. 3). Can also be calculated.

Then, after specifying the current position of the robot 10, the 3D distance image data obtained from the 3D omnidirectional laser distance meter 26 of the robot 10 is compared with the 3D environment map data, and the data is not recorded on the map. Dimensional distance image data can be extracted. The object extracted by this is a movable object (a person or a movable object) in the moving space. This is because what is on the 3D environment map is a fixed object such as a wall, so what is not on the 3D environment map is a temporary moving object.

By clustering the three-dimensional distance image data showing such a moving object by the method shown in Japanese Patent No. 5935484 mentioned in the background technology (step S3), the three-dimensional points of the moving object as shown in FIG. 9 Can be divided into groups.

One way of thinking is to use such a 3D point cloud as an input for an estimation algorithm such as a neural network, and the output is the characteristics of the object corresponding to each point cloud (whether it is a person, gender, body orientation at that moment, etc.). Etc.) can be used. Neural networks can be learned if there is sufficient point cloud and feature data. For example, if you collect many scans of people who understand gender, you can use it for learning data of gender. However, learning data can also be created with a simulator.

However, in this embodiment, the three-dimensional point cloud is not directly input to the neural network.

First, in step S5, a projection is used to create a two-dimensional distance image as shown in FIG.

Specifically, as shown in FIG. 11, a square surface perpendicular to the line extending from the three-dimensional range finder 26 is used as the projection surface. However, the width and height of the projection surface are arbitrary (for example, 1 m × 2 m). Then, the center of the X-axis of the projection plane is aligned with the center of the point group (so that the image of the object is in the center). The distance of the projection surface from the three-dimensional rangefinder 26 is also the same as the average distance of the point cloud. When viewed from the 3D rangefinder 26, about half of the points are in front of the surface and half are behind the surface. However, the Y-axis position of the projection plane does not have to be considered very rigorously, but preferably the same Y-axis position is used for all point clouds for comparison. For example, the highest point of the projection surface should be 2 m so that the human head can enter. It is then divided into a grid of pixels on the projection plane (pixels are constant in width and height).

In this way, all points in the three-dimensional point cloud are projected onto the projection plane. Then, the value of that pixel is determined by calculating the average distance from the plane of all the points that entered a pixel. For example, the average distance of 0 m is 0.5, 0.5 m in the direction of the laser range finder 26 is "0", and 0.5 m in the opposite direction is "1". However, the value of the pixel that does not contain even one point becomes 0 (zero).

An example of a two-dimensional distance image in which a three-dimensional point cloud is projected by such a method is shown in FIG.

Then, in step S7, the convolutional neural network is used. The input to the convolutional neural network at this time is the two-dimensional distance image created in step S5, and the convolutional neural network used for the input of a normal camera may be used, and the configuration of the convolutional neural network can be avoided.

Then, the convolutional neural network into which the two-dimensional distance image is input can identify a person and an object and extract a person's characteristics such as body orientation and gender.

As described above, by using the latest machine learning method such as deep neural network (DNN) in this embodiment, it is possible to improve the accuracy and robustness of this process. At the same time as distinguishing between people and things, it reduces false positives and can estimate not only the position of people but also various other features. Furthermore, if this feature estimation result is used as feedback, the human detection accuracy can be improved. In addition, it is possible to improve the robustness of the robot's self-position estimation by using the human tracking results.

In addition, the conventional 3D rangefinder (3D omnidirectional laser rangefinder) human detection system generally uses the concept that all moving objects are humans, but this is not always correct. Especially in the case of a moving robot, there is a high possibility that a part of the environment is misjudged as a moving object due to an error in self-position estimation. On the other hand, as in the embodiment, after specifying the current position of the robot 10, a part of the environment (fixed object) is extracted by extracting a three-dimensional distance image of an object that is not on the three-dimensional environment map. False recognition as a moving object is suppressed as much as possible. Moreover, by a method such as deep learning, a recognition program that detects a person or an object from the shape of a three-dimensional moving object can be realized, and the object can be ignored and only the person can be detected and tracked.

Furthermore, with the conventional method, it was difficult to detect the characteristics of a person from the 3D distance image data of the 3D range sensor, but as described above, 3D measurement is performed using a machine learning method such as deep learning. The orientation of the body can be estimated from the 3D distance image data of the person captured by the area sensor. In addition, when the distance between the person and the sensor is sufficiently close, it is possible to estimate the orientation of the face. Furthermore, it is possible to estimate not only the orientation but also other characteristics of the person. For example, gender can be identified, or height, luggage, hairstyle, and clothing type (eg, skirt or trousers). However, these estimation accuracy depends on the resolution of the three-dimensional range sensor and the distance from a person.

In a person tracking system, while tracking a person, people and objects around him may become obstructive and invisible, so it is necessary to determine whether they are the same person when they are detected again. This re-recognition stage was difficult because human features could not be used with the conventional human detection algorithm using three-dimensional data, but by using the above feature identification results, it is possible to determine whether the person is the same person. Becomes more accurate. As a result, the robustness of human detection and tracking is improved, and the accuracy is improved.

In the conventional method, the general approach is for the robot to detect features in the environment and perform self-position estimation. This can be used stably in places where there are no or few people, but when there are many people around the robot, it interferes with the sensor and reduces the number of environmental features that can be detected, making self-position estimation difficult and unstable. May become. This is a common phenomenon especially in the case of service robots. On the other hand, if not only the characteristics of the environment but also the human detection and re-recognition results as in the above-described embodiment are used for self-position estimation, the stability of self-position estimation of the robot 10 is improved.

In the above-described embodiment, the convolutional neural network is used as the deep neural network (DNN) based on the idea that the embodiment is suitable for the problem to be solved, but other deep neural networks may be adopted.

Further, in the above-described embodiment, all the steps of FIG. 8 have been described as being executed by the computer 60 (FIG. 2) of the robot 10. However, the computer 94 of the remote control device 90 shown in FIG. 4 may perform all or part of the steps. In each case, it constitutes a human estimation system.

10 ... Robot 26 ... 3D omnidirectional laser rangefinder 60, 94 ... Computer 64, 98 ... Memory

Claims (4)

Moving robot,
A three-dimensional laser rangefinder mounted on the robot,
An extraction unit that compares the 3D distance image data from the 3D laser rangefinder with the 3D environment map data and extracts the 3D distance image data of a moving object.
A clustering unit that clusters the three-dimensional distance image data of the moving object to generate a three-dimensional point cloud of the moving object.
A projection unit that generates two-dimensional distance image data of the moving object by projecting the three-dimensional point group, and a deep neural network that inputs the two-dimensional distance image data are provided.
The projection unit sets a projection plane perpendicular to the line from the three-dimensional laser rangefinder, determines the pixel value of each pixel based on the average distance from the projection plane of all points entering each pixel. A person estimation system in which the deep neural network distinguishes between a person and an object. The person estimation system according to claim 1, wherein the deep neural network is a convolutional neural network. An estimation program executed by a computer of a human estimation system including a mobile robot equipped with a 3D laser distance meter, wherein the estimation program uses the computer as 3D distance image data from the 3D laser distance meter. Extractor that extracts 3D distance image data of moving objects by comparing with 3D environment map data,
A clustering unit that clusters the three-dimensional distance image data of the moving object to generate a three-dimensional point cloud of the moving object.
It functions as a projection unit that generates two-dimensional distance image data of the moving object by projecting the three-dimensional point group, and a deep neural network that inputs the two-dimensional distance image data.
The projection unit sets a projection plane perpendicular to the line from the three-dimensional laser rangefinder, determines the pixel value of each pixel based on the average distance from the projection plane of all points entering each pixel. An estimation program in which the deep neural network discriminates between a person and an object and also identifies the characteristics of the person. An estimation method performed by a computer in a human estimation system that includes a mobile robot equipped with a three-dimensional laser rangefinder.
An extraction step of comparing the 3D distance image data from the 3D laser rangefinder with the 3D environment map data to extract the 3D distance image data of a moving object.
A clustering step of clustering the three-dimensional distance image data of the moving object to generate a three-dimensional point cloud of the moving object.
Look including the step of introducing a projection step of generating a 2-dimensional range image data of the moving object, and the 2-dimensional range image data in the deep neural network by projecting the three-dimensional point group,
In the projection step, a projection plane perpendicular to the line from the three-dimensional laser rangefinder is set, and the pixel value of each pixel is determined based on the average distance from the projection plane of all points entering each pixel. and
An estimation method in which the deep neural network discriminates between a person and an object and also identifies the characteristics of the person.

Images