JP2018173707A - Person estimation system and estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a person estimation system which can suppress wrong detection as much as possible.SOLUTION: An estimation program includes steps of: extracting three dimensional distance image data of a moving object by comparing the three dimensional distance image data from a three dimensional laser distance meter mounted on a movable robot with three dimensional environmental map data, after identifying the current location of the movable robot (S1); generating a three dimensional point group of the moving object by clustering the three dimensional distance image data of the moving object (S3); generating two dimensional distance image data of the moving object by projecting the three dimensional point group (S5); and convoluting the two dimensional distance image data to supply the data to a neural network (S7).SELECTED DRAWING: Figure 8

Description

  The present invention relates to a human estimation system and an estimation program, and in particular, a human estimation system and an estimation program for estimating the position of a person using, for example, three-dimensional scan data acquired by a three-dimensional laser rangefinder mounted on a mobile robot.

  In order for robots and other mobile machines to operate in a coexisting environment with people, it is important to know their own position, the positions of surrounding people and their characteristics. When using a general camera as a robot sensor, privacy may be required, but the 3D range sensor (3D laser rangefinder) that measures only the distance does not have such a concern. This sensor is suitable for coexistence.

  Patent Document 1 discloses a technique for detecting a person using a three-dimensional laser distance meter and estimating the position thereof.

Patent No. 595484 [G01S 17/88]

  In the background art of Patent Document 1, it is based on the premise that all moving objects are people, and the erroneous detection associated therewith cannot be avoided.

  Therefore, a main object of the present invention is to provide a novel human estimation system and estimation program.

  Another object of the present invention is to provide a human estimation system and an estimation program that can suppress erroneous detection as much as possible.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate correspondence relationships with embodiments described later to help understanding of the present invention, and do not limit the present invention in any way.

  The first invention compares the three-dimensional distance image data from the moving robot, the three-dimensional laser rangefinder mounted on the robot, the three-dimensional laser rangefinder, and the three-dimensional environment map data to determine the three-dimensional distance of the moving object. Extraction unit for extracting image data, clustering unit for generating 3D point cloud of moving object by clustering 3D distance image data of moving object, and 2D distance image data of moving object by projecting 3D point cloud Is a human estimation system that includes a projecting unit that generates and a deep neural network that receives two-dimensional distance image data as input, and the deep neural network identifies a person and an object.

  In the first invention, the human estimation system includes a moving robot (10: reference numerals exemplifying corresponding parts in the embodiment; the same applies hereinafter), and the robot is equipped with a three-dimensional laser distance meter (26). Is done. The extraction unit (60, 94, S1) compares the three-dimensional distance image data from the three-dimensional laser rangefinder and the three-dimensional environment map data to extract the three-dimensional distance image data of the moving object. In the clustering unit (60, 94, S3), the extracted three-dimensional distance image data of the moving object is clustered to generate a three-dimensional point group of the moving object. The projection unit (60, 94, S5) generates two-dimensional distance image data of the moving object by projecting the three-dimensional point group, and the two-dimensional distance image data is input to the deep neural network (S7). In a deep neural network, a person and an object are identified based on two-dimensional distance image data. Furthermore, a deep neural network can also identify human characteristics (eg, gender, orientation, etc.).

  According to the first invention, by extracting a three-dimensional distance image of an object that is not on the three-dimensional environment map, it is possible to suppress erroneous recognition of a part of the environment (fixed object) as a moving object. Is done. In addition, a recognition program for detecting whether a person or an object from the shape of a three-dimensional moving object can be realized by a technique such as deep learning, so that only a person can be detected ignoring the object.

  A second invention is a human estimation system according to the first invention, wherein the deep neural network is a convolutional neural network.

  According to the second invention, since the convolutional neural network is used as the deep neural network (DNN), human estimation can be easily performed.

  A third invention is dependent on the first or second invention, and the projection unit sets a projection plane perpendicular to a line from the three-dimensional laser rangefinder, and from the projection plane of all points entering each pixel. It is a human estimation system that determines the pixel value of each pixel based on the average distance of.

  According to the third invention, the clustered three-dimensional point group can be efficiently projected as a two-dimensional distance image.

  A fourth invention is an estimation program executed by a computer of a human estimation system including a movable robot equipped with a three-dimensional laser rangefinder, the estimation program comprising: An extraction unit that compares the distance image data with the 3D environment map data and extracts the 3D distance image data of the moving object, and generates a 3D point cloud of the moving object by clustering the 3D distance image data of the moving object. The clustering unit functions as a projection unit that generates two-dimensional distance image data of a moving object by projecting a three-dimensional point cloud, and a deep neural network that receives the two-dimensional distance image data as input. It is an estimation program that identifies

  A fifth invention is an estimation method executed by a computer of a human estimation system including a movable robot equipped with a three-dimensional laser distance meter, and includes three-dimensional distance image data from the three-dimensional laser distance meter and a three-dimensional environment. Extraction step for extracting 3D distance image data of moving object by comparing with map data, Clustering step for generating 3D point cloud of moving object by clustering 3D distance image data of moving object, 3D point cloud Is a projection method for generating two-dimensional distance image data of a moving object by projecting and a step of inputting the two-dimensional distance image data into a deep neural network.

  According to the present invention, since it is possible to prevent misidentification that a part of the environment is a moving object, it is possible to suppress erroneous detection as much as possible.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing one example of a robot used in a human estimation system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the electrical configuration of the robot of FIG. 1 embodiment. FIG. 3 is an illustrative view showing one example of a movement space of the robot of FIG. 1 embodiment. FIG. 4 is a block diagram showing an example of the robot remote control apparatus of FIG. 1 embodiment. FIG. 5 is an illustrative view showing one example of a memory map of a memory of the robot in the embodiment of FIG. FIG. 6 is an illustrative view showing a part of the movement space of the robot of FIG. 1 embodiment. FIG. 7 is an illustrative view showing one example of actual scan data of the omnidirectional laser rangefinder mounted on the robot. FIG. 8 is a flowchart showing an example of the operation of the estimation system using the robot of FIG. 1 embodiment. FIG. 9 is an illustrative view showing one example of a three-dimensional point group of a moving object as a result of clustering three-dimensional distance image data of a portion not on the map in the embodiment of FIG. 10 is an illustrative view showing one example of a two-dimensional distance image obtained by projecting the three-dimensional point group of the moving object shown in FIG. 9 in the embodiment of FIG. FIG. 11 is an illustrative view showing one example of a method for projecting the three-dimensional point group of the moving object shown in FIG.

  FIG. 1 shows an example of a robot 10 used in an estimation system according to an embodiment of the present invention. In this embodiment, the estimation system uses the three-dimensional range sensor (three-dimensional omnidirectional laser rangefinder) mounted on the robot 10 to simultaneously estimate the self-position of the robot 10 and the position, orientation, or attribute of the person. It is possible.

  The robot 10 includes a carriage 12, and two wheels 14 and one slave wheel 16 for moving the robot 10 are provided on the lower surface of the carriage 12. The two wheels 14 are independently driven by a wheel motor 82 (see FIG. 2), and the carriage 12, that is, the robot 10 can be moved in any direction, front, back, left, and right.

  A cylindrical sensor mounting panel 18 is provided on the carriage 12, and a number of distance sensors 20 are mounted on the sensor mounting panel 18. These distance sensors 20 measure distances from objects (people, obstacles, etc.) around the robot 10 using, for example, infrared rays or ultrasonic waves.

  A body 22 is provided upright on the sensor mounting panel 18. Further, the above-described distance sensor 20 is further provided at the upper front upper portion of the body 22 (a position corresponding to a person's chest), and measures the distance mainly to the person in front of the robot 10. Further, the body 22 is provided with a support column 24 extending from substantially the center of the upper end of the side surface, and an omnidirectional laser rangefinder 26 is provided on the support 24. The omnidirectional laser rangefinder 26 is, for example, a three-dimensional laser rangefinder having a detection angle (vertical viewing angle) of 40 ° above and below (+ 30 ° −−10 °) with respect to the horizontal. The three-dimensional omnidirectional laser rangefinder (hereinafter, simply referred to as “laser rangefinder”) 26 measures a distance of up to about 100 m, for example, by making one revolution per 0.1 second. In the experiment, an imaging unit LiDAR (HDL-32e) (trade name) manufactured by Velodine was used as the laser distance meter 26.

  Although not shown, the support column 24 may be further provided with an omnidirectional camera. The omnidirectional camera captures the surroundings of the robot 10 and is distinguished from an eye camera 50 described later. As the three omnidirectional camera, for example, a camera using a solid-state imaging device such as a CCD or a CMOS can be adopted.

  An upper arm 30R and an upper arm 30L are provided at upper end portions on both sides of the body 22 (a position corresponding to a human shoulder) by a shoulder joint 28R and a shoulder joint 28L, respectively. Although illustration is omitted, each of the shoulder joint 28R and the shoulder joint 28L has three orthogonal degrees of freedom. That is, the shoulder joint 28R can control the angle of the upper arm 30R around each of three orthogonal axes. A certain axis (yaw axis) of the shoulder joint 28R is an axis parallel to the longitudinal direction (or axis) of the upper arm 30R, and the other two axes (pitch axis and roll axis) are orthogonal to the axis from different directions. It is an axis to do. Similarly, the shoulder joint 28L can control the angle of the upper arm 30L around each of three orthogonal axes. A certain axis (yaw axis) of the shoulder joint 28L is an axis parallel to the longitudinal direction (or axis) of the upper arm 30L, and the other two axes (pitch axis and roll axis) are orthogonal to the axes from different directions. It is an axis to do.

  In addition, an elbow joint 32R and an elbow joint 32L are provided at the respective distal ends of the upper arm 30R and the upper arm 30L. Although illustration is omitted, each of the elbow joint 32R and the elbow joint 32L has one degree of freedom, and the angles of the forearm 34R and the forearm 34L can be controlled around the axis (pitch axis).

  A hand 36 </ b> R and a hand 36 </ b> L corresponding to human hands are respectively provided at the tips of the forearm 34 </ b> R and the forearm 34 </ b> L. Although the detailed illustration is omitted, these hands 36R and 36L are configured to be openable and closable so that the robot 10 can grip or hold an object using the hands 36R and 36L. However, the shape of the hands 36R and 36L is not limited to the shape of the embodiment, and may have a shape and function very similar to a human hand.

  Although not shown, the front surface of the carriage 12, the portion corresponding to the shoulder including the shoulder joint 28R and the shoulder joint 28L, the upper arm 30R, the upper arm 30L, the forearm 34R, the forearm 34L, the hand 36R, and the hand 36L, A contact sensor 38 (shown generically in FIG. 2) is provided. The contact sensor 38 on the front surface of the carriage 12 detects contact of a person or another obstacle with the carriage 12. Therefore, when the robot 10 is in contact with an obstacle during its movement, the robot 10 can detect this and immediately stop driving the wheel 14 to suddenly stop the movement of the robot 10. Further, the other contact sensors 38 detect whether or not the respective parts are touched.

  A neck joint 40 is provided at the upper center of the body 22 (a position corresponding to a person's neck), and a head 42 is further provided thereon. Although illustration is omitted, the neck joint 40 has a degree of freedom of three axes, and the angle can be controlled around each of the three axes. A certain axis (yaw axis) is an axis directed directly above (vertically upward) of the robot 10, and the other two axes (pitch axis and roll axis) are axes orthogonal to each other in different directions.

  The head 42 is provided with a speaker 44 at a position corresponding to a human mouth. The speaker 44 is used for the robot 10 to communicate with the surrounding people by voice. A microphone 46R and a microphone 46L are provided at a position corresponding to a human ear. Hereinafter, the right microphone 46R and the left microphone 46L may be collectively referred to as a microphone 46. The microphone 46 captures ambient sounds, in particular, the voices of humans who are subjects of communication.

  Further, a right eyeball part 48R and a left eyeball part 48L are provided at positions corresponding to human eyes. The right eyeball portion 48R and the left eyeball portion 48L include a right eye camera 50R and a left eye camera 50L, respectively. Hereinafter, the right eyeball portion 48R and the left eyeball portion 48L may be collectively referred to as the eyeball portion 48. Further, the right eye camera 50R and the left eye camera 50L may be collectively referred to as an eye camera 50.

  The eye camera 50 captures a human face approaching the robot 10 and other parts or objects, and captures a corresponding video signal. In this embodiment, the robot 10 detects the line-of-sight directions (vectors) of the left and right eyes of the person based on the video signal from the eye camera 50.

  The eye camera 50 can be the same camera as the three-dimensional distance image meter 26 described above. For example, the eye camera 50 is fixed in the eyeball unit 48, and the eyeball unit 48 is attached to a predetermined position in the head 42 via an eyeball support unit (not shown). Although illustration is omitted, the eyeball support portion has two degrees of freedom, and the angle can be controlled around each of these axes. For example, one of the two axes is an axis (yaw axis) in a direction toward the top of the head 42, and the other is orthogonal to the one axis and goes straight in a direction in which the front side (face) of the head 42 faces. It is an axis (pitch axis) in the direction to be performed. By rotating the eyeball support portion around each of these two axes, the tip (front) side of the eyeball portion 48 or the eye camera 50 is displaced, and the camera axis, that is, the line-of-sight direction is moved. Note that the installation positions of the speaker 44, the microphone 46, and the eye camera 50 described above are not limited to the portions, and may be provided at appropriate positions.

  As described above, the robot 10 of this embodiment includes independent two-axis driving of the wheel 14, three degrees of freedom of the shoulder joint 28 (6 degrees of freedom on the left and right), one degree of freedom of the elbow joint 32 (two degrees of freedom on the left and right), It has a total of 17 degrees of freedom: 3 degrees of freedom of the neck joint 40 and 2 degrees of freedom of the eyeball support (4 degrees of freedom on the left and right).

  FIG. 2 is a block diagram showing an electrical configuration of the robot 10. With reference to FIG. 2, the robot 10 includes one or more computers 60. The computer 60 is connected to the memory 64, the motor control board 66, the sensor input / output board 68, and the audio input / output board 70 via the bus 62.

  Although illustration is omitted, the memory 64 includes a ROM, an HDD, and a RAM. As will be described later, the ROM and HDD store various programs and data in advance, and are also used as a buffer area or a working area for the computer 60.

  The motor control board 66 is configured by, for example, a DSP, and controls driving of motors of the axes such as the arms, the neck joint 40, and the eyeball unit 48. That is, the motor control board 66 receives control data from the computer 60, and controls two motors for controlling the angles of the two axes of the right eyeball portion 48R (in FIG. 2, they are collectively referred to as “right eyeball motor 72”). Control the rotation angle. Similarly, the motor control board 66 receives control data from the computer 60 and controls two motors for controlling the respective angles of the two axes of the left eyeball portion 48L (in FIG. 2, collectively referred to as “left eyeball motor 74”). Control the angle of rotation.

  Further, the motor control board 66 receives control data from the computer 60, and includes a total of three motors for controlling the angles of the three orthogonal axes of the shoulder joint 28R and one motor for controlling the angle of the elbow joint 32R. The rotational angles of four motors (collectively indicated as “right arm motor 76” in FIG. 2) are controlled. Similarly, the motor control board 66 receives control data from the computer 60, three motors for controlling the angles of the three orthogonal axes of the shoulder joint 28L, and one motor for controlling the angle of the elbow joint 32L. The rotation angles of a total of four motors (collectively indicated as “left arm motor 78” in FIG. 2) are controlled.

  Further, the motor control board 66 receives the control data from the computer 60 and controls three motors for controlling the angles of the three orthogonal axes of the neck joint 40 (in FIG. 2, collectively referred to as “head motor 80”). ) To control the rotation angle. The motor control board 66 receives control data from the computer 60 and controls the rotation angles of two motors (collectively referred to as “wheel motors 82” in FIG. 2) that drive the wheels 14.

  Similar to the motor control board 66, the sensor input / output board 68 is configured by a DSP, and takes in signals from each sensor and gives them to the computer 60. That is, data relating to the reflection time from each of the distance sensors 20 is input to the computer 60 through the sensor input / output board 68.

  The distance image signal from the three-dimensional distance imager 26 is subjected to predetermined processing by the sensor input / output board 68 as necessary, and then input to the computer 60 as distance image data.

  A video signal from the eye camera 50 is also input to the computer 60 in the same manner. Further, signals from the plurality of contact sensors 38 described above (collectively referred to as “contact sensors 38” in FIG. 2) are provided to the computer 60 via the sensor input / output board 68.

  Similarly, the voice input / output board 70 is also configured by a DSP, and voice or voice in accordance with voice synthesis data provided from the computer 60 is output from the speaker 44. Also, audio input from the microphone 46 is given to the computer 60 via the audio input / output board 70.

  The computer 60 is connected to the communication LAN board 84 via the bus 62. The communication LAN board 84 is composed of, for example, a DSP, and provides transmission data given from the computer 60 to the wireless communication module 86. The wireless communication module 86 sends the transmission data to a remote control device (not shown) via a network. Send to. The communication LAN board 84 receives data via the wireless communication module 86 and gives the received data to the computer 60.

  Such a robot 10 moves along a movement path 88b between the entrances and exits 88a of the moving space 88 shown in FIG. 3, for example, and provides guidance services to persons existing in the moving space 88 such as a commercial facility. Provide such services.

  Although not shown, a plurality of three-dimensional omnidirectional laser rangefinders similar to those of the robot 10 are arranged in the moving space 88 as environmental sensors.

  The robot 10 moving in the moving space 88 can basically move autonomously, but in preparation for a situation where the robot 10 cannot move autonomously, for example, a remote control device 90 as shown in FIG. It is possible to control by.

  Referring to FIG. 4, the remote control device 90 of this embodiment can communicate with the robot 10 via a network 92 in order to control the operation and movement of the robot 10 as a moving body.

  The remote operation device 90 includes a computer 94. When an operator operates an operation key or a joystick (not shown) provided on an operation console 96 connected to the computer 94, the computer 94 responds to the operation input. Can control the wheel 14 of the robot 10, that is, the movement of the robot 10.

  A memory 98 is connected to the computer 94 and a display 10 is connected. The memory 98 stores necessary data such as a program for robot control and 2D map data and 3D map data of the moving space 88 (FIG. 3). The display device 100 displays the moving state of the robot 10 and images taken by an omnidirectional camera (not shown) for safe movement of the robot 10 to show to the operator.

  A wireless communication module 102 is attached to the computer 94, and the computer 94 can perform wireless communication with the computer 60 through the network 92 and through the wireless communication module 86 (FIG. 2) of the robot 10.

  Here, the memory 64 of the computer 60 of the robot 10 of FIG. 1 includes a program storage area 104 and a data storage area 106 as shown in FIG. 5, for example.

  In the program storage area 104, in addition to a necessary program such as an OS, a movement control program 108 for controlling movement of the robot 10 and an estimation program 110 described in detail later are set in advance. The estimation program 110 estimates the characteristics of a person moving in the moving space 88, and includes a deep neural network (for example, a convolutional neural network) for that purpose.

  The data storage area 106 includes a three-dimensional distance image data storage area 112 for temporarily storing three-dimensional distance image data (data set) obtained from the laser distance meter 26 (FIGS. 1 and 2), and a remote control device. A three-dimensional map data storage area 114 that stores data of a three-dimensional map (three-dimensional environment map) of the moving space 88 acquired from 90 memories 98 is included.

  The data storage area 106 further includes a three-dimensional point cloud data storage area 116 for storing data of a three-dimensional point cloud of a moving object, which will be described later, obtained in the process according to the estimation program 110, and two-dimensional distance image data of the moving object. Is stored in the two-dimensional distance image data storage area 118.

  Such a robot 10 acquires scan data by the laser distance meter 28 while moving in a place as shown in FIG. The place illustrated in FIG. 6 is a part of the second floor of a commercial facility used by the inventors for the experiment of the estimation system. In the experiment, the robot 10 moves about 530 m over about 25 minutes in this place of the commercial facility in one run, and outputs position data (data set) of about 182 million point clouds.

  FIG. 7 shows representative three-dimensional scan data 120 acquired by the laser distance meter 28, and the arc-shaped line in FIG. 7 schematically shows a state in which the laser distance meter 28 is scanning. As shown in FIG. 7, the scan data includes not only static objects such as walls and floors but also data from dynamic objects such as humans. On the other hand, although not understood in FIG. 7, the actual scan data is colored in the height direction, and the low position is expressed in red and the high position is expressed in blue, respectively, and 3 in the memory 84 of the robot 10 as a three-dimensional distance image. It is stored in the dimensional distance image data storage area 112 (FIG. 5).

  In step S1 shown in FIG. 8, the self-position is first identified using such 3D distance image data and 3D environment map data stored in advance in the 3D map data storage area 114 of the memory 64. . Specifically, the distance between the position of the robot 10 and the position of the wall (indicated by “124” in the scan data 122 in FIG. 7) recorded as a fixed object (stationary object) in the three-dimensional environment map (3 By calculating (by the principle of angle measurement), the current position (coordinate position) of the robot 10 can be specified.

  However, the calculation for specifying the current position of the robot 10 in step S1 uses a plurality of three-dimensional omnidirectional laser distance meters (not shown) distributed in the moving space 88 (FIG. 3). Can also be calculated.

  Then, after the current position of the robot 10 is specified, it is not recorded on the map by comparing the 3D range image data obtained from the 3D omnidirectional laser rangefinder 26 of the robot 10 with the 3D environment map data. Dimensional distance image data can be extracted. The object extracted by this is a movable object (a person or something that can move) in the movement space. Because what is on the 3D environment map is a fixed object such as a wall, what is not on the 3D environment map is a temporary moving object.

  The three-dimensional distance image data indicating such a moving object is clustered by the method shown in Japanese Patent No. 595484 cited in the background art (step S3), so that the three-dimensional points of the moving object as shown in FIG. Can be divided into groups.

  One way of thinking is to use such a 3D point cloud as an input for an estimation algorithm such as a neural network, and the output is the characteristics of the object corresponding to each point cloud (whether it is a person, gender, body orientation at that moment, Etc.). If the point cloud and feature data are sufficient, the neural network can be learned. For example, if you collect a lot of scans of people who have sex, you can use it as sex learning data. However, learning data can also be created with a simulator.

  However, in this embodiment, the three-dimensional point group is not directly input to the neural network.

  First, in step S5, a two-dimensional distance image as shown in FIG. 10 is created using projection.

  Specifically, as shown in FIG. 11, a rectangular plane perpendicular to the line extending from the three-dimensional distance meter 26 is used as the projection plane. However, the width and height of the projection surface are arbitrary (for example, 1 m × 2 m). Then, the center of the X-axis of the projection plane is aligned with the center of the point group (the object image is set to the center). The distance from the three-dimensional distance meter 26 on the projection plane is also made the same as the average distance of the point group. When viewed from the three-dimensional distance meter 26, approximately half of the points are in front of the surface and half are behind the surface. However, the position of the Y axis on the projection plane need not be considered very strictly, but preferably the same Y axis position is used for all point groups so that they can be compared. For example, the highest point of the projection plane is 2 m so that a human head can enter. It is then divided into a grid of pixels on the projection plane (pixel width and height are constant).

  In this way, all the points of the three-dimensional point group are projected onto the projection plane. Then, the value of the pixel is determined by calculating the average distance from the plane of all points that entered the pixel. For example, the average distance 0 m is 0.5, 0.5 m is “0” in the direction of the laser distance meter 26, and 0.5 m is “1” in the reverse direction. However, the value of a pixel that does not contain one point is 0 (zero).

  An example of a two-dimensional distance image obtained by projecting a three-dimensional point group by such a method is shown in FIG.

  Thereafter, in step S7, a convolutional neural network is used. The input to the convolutional neural network at this time is the two-dimensional distance image created in step S5, which may be a convolutional neural network used for normal camera input, and avoids the complicated configuration of the convolutional neural network.

  The convolutional neural network into which the two-dimensional distance image is input can distinguish between a person and an object, and can extract a person's characteristics such as a body orientation and sex.

  As described above, in this embodiment, it is possible to improve the accuracy and robustness of this process by using the latest machine learning technique such as a deep neural network (DNN). At the same time as distinguishing between people and objects, false detection can be reduced and not only the position of the person but also various other features can be estimated. Furthermore, if this feature estimation result is used as feedback, human detection accuracy can be improved. In addition, if the tracking result of the person is used, the robustness of the robot's self-position estimation can be improved.

  Further, in the conventional human detection system of a three-dimensional range sensor (three-dimensional omnidirectional laser rangefinder), the concept that all moving objects are people is used, but this is not necessarily correct. Especially in the case of a moving robot, there is a high possibility that a part of the environment is a moving object due to an error in self-position estimation. On the other hand, as in the embodiment, after specifying the current position of the robot 10, a part of the environment (fixed object) is extracted by extracting a three-dimensional distance image of an object not on the three-dimensional environment map. Misrecognition as a moving object is suppressed as much as possible. In addition, a recognition program for detecting whether a person or an object from the shape of a three-dimensional moving object can be realized by a technique such as deep learning, so that only the person can be detected and tracked while ignoring the object.

  Furthermore, in the conventional method, it has been difficult to detect human features from the 3D range image data of the 3D range sensor, but as described above, 3D measurement is performed using machine learning techniques such as deep learning. The direction of the body can be estimated from the three-dimensional distance image data of the person shown in the area sensor. In addition, when the distance between the person and the sensor is sufficiently close, the orientation of the face can be estimated. Furthermore, it is possible to estimate not only the orientation but also other characteristics of the person. For example, gender identification or height, baggage presence, hairstyle, clothing type (eg skirt or trousers) can be identified. However, the estimation accuracy depends on the resolution of the three-dimensional range sensor, the distance from a person, and the like.

  In the human tracking system, it is necessary to determine whether or not the same person is detected when the person is detected again because the surrounding person or object may be obstructed and may be invisible while the person is being tracked. In the conventional human detection algorithm using three-dimensional data, human features could not be used, and this re-recognition stage was difficult. However, by using the result of the above feature identification, it is determined whether or not they are the same person. Becomes more accurate. As a result, the robustness of human detection and tracking is improved, and the accuracy is improved.

  In the conventional method, a general approach is that a robot detects features in the environment and performs self-position estimation. This can be used stably in places where there are few or few people, but if there are many people around the robot, it will interfere with the sensor and reduce the number of environmental features that can be detected, making self-position estimation difficult and unstable. May be. This is a phenomenon often seen especially in the case of service robots. On the other hand, if the human detection and re-recognition results as in the above-described embodiment are used for self-position estimation in addition to the environmental features, the stability of the self-position estimation of the robot 10 increases.

  In the above-described embodiment, the convolutional neural network is used as the deep neural network (DNN) because it is suitable for the problem to be solved. However, other deep neural networks may be used.

  Further, in the above-described embodiment, it has been described that all the steps in FIG. 8 are executed by the computer 60 (FIG. 2) of the robot 10. However, the computer 94 of the remote control device 90 shown in FIG. 4 may execute all or some of the steps. In either case, the person estimation system is configured.

DESCRIPTION OF SYMBOLS 10 ... Robot 26 ... Three-dimensional omnidirectional laser distance meter 60, 94 ... Computer 64, 98 ... Memory

Claims (5)

Moving robot,
A three-dimensional laser rangefinder mounted on the robot;
An extraction unit that compares the three-dimensional distance image data from the three-dimensional laser rangefinder with the three-dimensional environment map data to extract the three-dimensional distance image data of the moving object;
A clustering unit that clusters the three-dimensional distance image data of the moving object to generate a three-dimensional point group of the moving object;
A projection unit that generates two-dimensional distance image data of the moving object by projecting the three-dimensional point group, and a deep neural network that receives the two-dimensional distance image data,
A human estimation system in which the deep neural network identifies a person and an object.   The human estimation system according to claim 1, wherein the deep neural network is a convolutional neural network.   The projection unit sets a projection plane perpendicular to a line from the three-dimensional laser rangefinder, and determines a pixel value Dell of each pixel based on an average distance from the projection plane of all points entering each pixel. The human estimation system according to claim 1 or 2. An estimation program executed by a computer of a human estimation system including a movable robot equipped with a three-dimensional laser rangefinder, the estimation program comprising: An extraction unit that compares the three-dimensional environmental map data and extracts the three-dimensional distance image data of the moving object;
A clustering unit that clusters the three-dimensional distance image data of the moving object to generate a three-dimensional point group of the moving object;
Projecting the three-dimensional point cloud to generate two-dimensional distance image data of the moving object, and functioning as a deep neural network that receives the two-dimensional distance image data,
An estimation program in which the deep neural network identifies a person and an object, and identifies a person's characteristics. An estimation method executed by a computer of a human estimation system including a movable robot equipped with a three-dimensional laser rangefinder,
An extraction step of extracting the 3D distance image data of the moving object by comparing the 3D distance image data from the 3D laser rangefinder with the 3D environment map data;
A clustering step of clustering the three-dimensional distance image data of the moving object to generate a three-dimensional point group of the moving object;
An estimation method comprising: a projecting step of generating two-dimensional distance image data of the moving object by projecting the three-dimensional point group; and inputting the two-dimensional distance image data into a deep neural network.