JP2021076948A - Person detection device, person detection system, and person detection method - Google Patents

Abstract

To provide a person detection device that detects a person with high accuracy even when a plurality of persons overlap and a part of the person behind cannot be seen or when a plurality of persons are close to each other.SOLUTION: In a person detection system 1, a person detection device 10 includes: a first person detection unit 107 that detects a person included in an RGB image acquired from an RGB camera 101; a point cloud clustering unit 108 that acquires a point cloud group by spatially grouping point clouds included in point cloud data generated from a point cloud image acquired from a ToF sensor 102; a person point cloud cluster association unit 109 that associates a person detection area in which the first person detection unit 107 detected the person in the RGB image with the point cloud group grouped by the point cloud clustering unit 108; and a second person detection unit 112 that determines whether or not a non-corresponding point cloud group is a person according to a first feature quantity of the non-corresponding point cloud group of the point cloud group that is not associated with the person detection area by the person point cloud cluster association unit 109.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for detecting a person included in an image.

In the surveillance technology using an image, a person included in the image may be detected. Patent Document 1, Patent Document 2, and Patent Document 3 are prior arts in the present technical field.

The following Patent Document 1 describes, "A distance measuring unit that measures a distance with respect to a predetermined range, a human processing unit that detects a person based on the distribution of the distance measured by the distance measuring unit, and the human processing unit. It is a value in the height direction included in the measured distance in the tracking unit that assigns an identifier to each person detected by and the first period set by urging the person to start and end raising his / her hand. A situation estimation unit that estimates a person's reaction based on a person data height is provided, and the situation estimation unit sets the person data height for each person assigned an identifier by the tracking unit for the first period. Discloses a technique called "motion estimation device" that measures a plurality of times, extracts a plurality of identifiers whose data heights are within a predetermined range, and estimates the reaction of a person corresponding to the extracted identifiers. See claim 1).

The following Patent Document 2 describes, "A distance data receiving means for receiving distance data indicating a distance from a passerby's foot from a laser sensor and a plurality of the distance data acquired by the plurality of laser sensors are used at the same time. Sensing image data generation means that generates sensing image data indicating the detection position of the laser sensor on the spatial plane, clustering means that clusters a plurality of the detection positions in the sensing image data, and one passerby from the clustering result. From the passerby identification means for specifying the cluster corresponding to the foot and the specified passerby cluster, parameters including the position of the point of the passerby's foot, walking direction, stride length, walking cycle, and walking phase are determined. A technique called a "passerby behavior analysis device" comprising a parameter determining means and a locus analysis processing means for analyzing the locus of a passerby using the parameters and a particle filter algorithm is disclosed (claim). 1).

The following Patent Document 3 describes, "A acquisition means for acquiring a movement vector of a crowd for each range of an image, a generation means for generating a human flow cluster based on the movement vector acquired by the acquisition means, and a generation means generated by the generation means. A classification means for classifying a normal human flow cluster into a normal human flow cluster and an abnormal human flow cluster, and displaying the normal human flow cluster and the abnormal human flow cluster superimposed on the image in a visually distinguishable form. It discloses a technique called "an image processing device having a display means" (see claim 1).

Japanese Unexamined Patent Publication No. 2016-218610 Japanese Unexamined Patent Publication No. 2009-110185 JP-A-2019-067208

By using the prior art, it is possible to detect a person in the area to be photographed. However, as the number of people increases, there are cases where multiple people overlap and some of the people behind are not visible, so the people behind are not detected, or when multiple people are close to each other, multiple people are detected as one person. There are more cases where it ends up.

The present invention has been made in view of the above problems, and can detect a person with high accuracy even when a plurality of people overlap and a part of the person behind cannot be seen or when a plurality of people are close to each other. It is an object of the present invention to provide a person detection device capable of performing.

The person detection device according to the present invention detects a human area included in an RGB image, then calculates a feature amount of an uncorresponding point group group that does not correspond to the human area in the point group cluster, and calculates the feature amount. Therefore, it is determined whether or not the uncorresponding point group is a person.

According to the person detection device according to the present invention, a person can be detected with high accuracy even when a plurality of people overlap and a part of the person behind cannot be seen or when a plurality of people are close to each other. Issues, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

It is a block diagram of the person detection system 1 which concerns on Embodiment 1. It shows the situation where the same set of markers is photographed by the RGB camera 101 and the ToF sensor 102. It is a flowchart explaining the process which the person detection system 1 detects a person in an image. This is an example of a shooting environment. It is a processing flow diagram which shows the procedure of detecting the subject 405A and 406A in FIG. 4A according to the flowchart of FIG. It is a block diagram of the person detection system 1 which concerns on Embodiment 2. It is a conceptual diagram explaining the details of the processing performed by the person cluster shape feature amount extraction unit 502, the point cloud cluster fitting unit 504, and the second person detection unit 112. It is a flowchart explaining the process which the person detection system 1 in Embodiment 2 detects a person in an image. It is a flowchart explaining the detail of step S703. It is a block diagram of the person detection system 1 which concerns on Embodiment 3. It is a figure explaining the detail of the person detection method by the point cloud density maximum value detection unit 802 and the second person detection unit 112. It is a flowchart explaining the process which the person detection system 1 in Embodiment 3 detects a person in an image.

In all the drawings for explaining the embodiment, the same members are, in principle, given the same reference numerals, and the repeated description thereof will be omitted. Further, in the following embodiments, the components (including element steps and the like) are not necessarily essential unless otherwise specified or clearly considered to be essential in principle. Needless to say. In addition, when saying "consisting of A", "consisting of A", "having A", and "including A", other elements are excluded unless it is clearly stated that it is only that element. It goes without saying that it is not something to do. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of a component or the like, the shape is substantially the same unless otherwise specified or when it is considered that it is not apparent in principle. Etc., etc. shall be included.

The notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and do not necessarily limit the number, order, or contents thereof. is not it. In addition, numbers for identifying components are used for each context, and numbers used in one context do not always indicate the same composition in other contexts. Further, it does not prevent the component identified by a certain number from having the function of the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

<Embodiment 1>
FIG. 1 is a configuration diagram of a person detection system 1 according to the first embodiment of the present invention. The person detection system 1 is a system for detecting a person included in an image. The human detection system 1 includes a photographing processing unit 11, a first information processing unit 12, and a second information processing unit 13. The first information processing unit 12 and the second information processing unit 13 can configure the person detection device 10 (the same applies to the second and subsequent embodiments).

The photographing processing unit 11 has an RGB camera 101 and a ToF (Time of Flight) sensor 102. The RGB camera 101 acquires an image (hereinafter referred to as an RGB image) in which each pixel has general gradation information (RGB or the like) and outputs the image to the RGB image acquisition unit 103. The ToF sensor 102 acquires a distance image in which each pixel has distance information and outputs the distance image to the distance image acquisition unit 104. Each pixel in the distance image can be converted into three-dimensional coordinates by coordinate conversion, and the group of points indicated by the three-dimensional coordinates after the conversion is called a point cloud.

The first information processing unit 12 includes an RGB image acquisition unit 103, a distance image acquisition unit 104, a pixel synchronization unit 105, a synchronization information holding unit 106, a first person detection unit 107, a point cloud clustering unit 108, and a point cloud cluster association. It has a unit 109 and a person cluster removing unit 110.

The RGB image acquisition unit 103 acquires an RGB image from the RGB camera 101 and outputs it to the pixel synchronization unit 105 and the first person detection unit 107. The distance image acquisition unit 104 acquires a distance image from the ToF sensor 102 and outputs it to the pixel synchronization unit 105 and the point cloud clustering unit 108.

The pixel synchronization unit 105 synchronizes the positions of each pixel of the RGB image acquired from the RGB image acquisition unit 103 and the distance image acquired from the distance image acquisition unit 104, and the correspondence relationship of each pixel in both images (hereinafter, pixel synchronization information). Is output to the synchronization information holding unit 106. This synchronization processing may be performed each time the RGB image and the distance image are acquired, or may be performed only when the RGB image and the distance image are first acquired, and the processing frequency of the pixel synchronization unit 105 is not limited. The details of the synchronization process will be described later.

The synchronization information holding unit 106 holds the pixel synchronization information acquired from the pixel synchronization unit 105, and outputs the pixel synchronization information to the person point cloud cluster association unit 109.

The first person detection unit 107 performs human detection on the RGB image acquired from the RGB image acquisition unit 103, and transmits the number of people detected and the person detection area information to the person point cloud cluster association unit 109 and the second person detection unit. Output to 112. For example, an image recognition algorithm using CNN (Convolutional Neural Network) is used to detect the whole body and face of a person, and the number of people detected and the person detection area information indicating the number and position of the detected people are associated with a point cloud cluster. Output to unit 109 and second person detection unit 112. When the number of detected persons is 0, the number of detected persons is set to 0 and the person detection area information is not output.

The point cloud clustering unit 108 converts the distance image acquired from the distance image acquisition unit 104 into a point cloud, and the point cloud is densely packed with a value of a certain density or more as a number of agglomerates (hereinafter, referred to as a point cloud cluster). Group the groups spatially. As another method for grouping the point cloud, for example, the distribution of the point cloud can be grouped according to the likelihood of the Gauss distribution model with respect to the coordinates of each point cloud when the distribution of the point cloud is reproduced by GMM (Gaussian Mixture Model). Other suitable grouping methods may be used. The point cloud clustering unit 108 outputs the grouped point cloud clusters to the human point cloud cluster association unit 109. This grouping process is also called clustering.

The point cloud cluster association unit 109 acquires the person detection area information acquired from the first person detection unit 107 from the point cloud clustering unit 108 based on the pixel synchronization information read from the synchronization information holding unit 106. (This point cloud cluster is called a region-corresponding point cloud cluster) and is output to the human cluster removal unit 110. At this time, the area-corresponding point cloud cluster includes a person-corresponding point cloud cluster to which the person detection area information is associated and a person-unsupported point cloud cluster to which the person detection area is not associated. When the number of detected persons acquired from the first person detection unit 107 is 0, the association processing is not performed on the point cloud cluster. At this time, the point cloud cluster output to the human cluster removal unit 110 is equal to the human unsupported point cloud cluster.

The human cluster removal unit 110 removes the human-corresponding point cloud from the area-corresponding point cloud cluster acquired from the human-point group cluster mapping unit 109, and outputs the non-human-corresponding point cloud to the moving object detection unit 111.

The second information processing unit 13 includes a moving object detection unit 111 and a second person detection unit 112.

The moving object detection unit 111 acquires a parameter representing the movement operation of the human uncorresponding point cloud cluster acquired from the human cluster removal unit 110. For example, the movement amount (for example, the movement distance from the first removed position) and the speed of the unsupported point cloud cluster are detected as the parameters, and the information is output to the second person detection unit 112.

The second person detection unit 112 determines whether or not the information such as the movement amount and the speed of the human uncorresponding point cloud cluster acquired from the moving object detection unit 111 is a value equivalent to a person, and determines the determination result according to the threshold value. By integrating the number of people detected and the person detection area information acquired from the one person detection unit 107, the number of people detected and the person detection area information are generated again. As the judgment criteria, for example, (a) if the moving distance exceeds 50 cm, it is judged as a person, and (b) if the speed is about a general walking speed (4 km / h), it is judged as a person. , Etc. can be considered. Once the non-human point cloud cluster is detected as a human, it follows even if it is stopped, as long as it does not deviate from the visual field region of the RGB camera 101 and the ToF sensor 102.

FIG. 2 shows a situation in which the same set of markers is photographed by the RGB camera 101 and the ToF sensor 102. The details of the synchronization processing performed by the pixel synchronization unit 105 will be described with reference to FIG. It is assumed that the scales of the images captured by the RGB camera 101 and the ToF sensor 102 are the same and are not distorted from each other.

The processing flow 21 shows a process of detecting two pixel positions of the marker photographed by the RGB camera 101. The processing flow 22 shows a process of detecting the pixel positions of two vertices of the marker photographed by the ToF sensor 102. The processing flow 23 indicates a process of detecting information necessary for synchronization.

The RGB image 201 is an RGB image acquired by the RGB image acquisition unit 103 from the RGB camera 101. The distance image 202 is a distance image acquired by the distance image acquisition unit 104 from the ToF sensor 102. The markers 203 and 204 are markers on the RGB image 201. The markers 205 and 206 are markers on the distance image 202.

The marker 203 visible at the two pixel coordinates (x11, y11) on the RGB image 201 and the marker 204 visible at the pixel coordinates (x12, y12) are the two pixel coordinates (x21, y21) and (x22, x22, on the distance image 202). It is also visible in y22) (markers 205 and 206). At this time, the positional relationship (pixel synchronization information) between the pixels for referencing the same portion of the marker between the RGB image 201 and the distance image 202 may be obtained. For example, the offset (gx, gy) and the rotation angle φ are set so that the markers 203 and 204 overlap the markers 205 and 206, and the numbers 1 and 2 are derived. By solving the equations 1 and 2 simultaneously, the offset (gx, gy) can be obtained by the equations 3 and 4, and the angle of rotation φ can be obtained by the inverse trigonometric function in the equation 5. By applying the calculated offset (gx, gy) and rotation angle φ to Equation 6, any pixel coordinates (xRGB, yRGB) on the RGB image 201 are associated with the pixel coordinates (xToF, yToF) on the distance image 202. be able to.

The synchronization information holding unit 106 holds the coordinates (gx, gy) and the rotation angle φ. The point cloud cluster association unit 109 refers to the coordinates (gx, gy) and the angle of rotation φ when associating the position of each pixel in the person detection region with the point cloud cluster according to Equation 6.

The synchronization process may be performed manually only once when the person detection system 1 is installed, or a template matching method for an arbitrary shape in the acquired image each time the RGB image 201 and the distance image 202 are acquired. Even if the relationship between the pixels is automatically calculated using the above and the obtained coordinates (gx, gy) and the rotation angle φ are stored in the synchronization information holding unit 106 at an arbitrary timing (for example, once an hour). Good.

When the scales of the two images captured by the RGB camera 101 and the ToF sensor 102 are different from each other, the synchronization process may be performed after performing the same magnification conversion so that the size of the marker detection area becomes the same by the affine transformation. When the image captured by the RGB camera 101 is distorted, the distortion correction process may be performed using a plurality of chess boards captured from various viewpoints in advance, and then the synchronization process may be performed. If either of the above cases occurs, each countermeasure may be combined.

FIG. 3 is a flowchart illustrating a process in which the human detection system 1 detects a person in an image. Each step of FIG. 3 will be described below.

(FIG. 3: Steps S301 to S303)
The RGB image acquisition unit 103 acquires an RGB image from the RGB camera 101 (S301). The distance image acquisition unit 104 acquires a distance image from the ToF sensor 102 and generates a point cloud from the acquired distance image (S302). The pixel synchronization unit 105 calculates the coordinates (gx, gy) and the angle of rotation φ by synchronizing the pixel positions of the RGB image and the pixel positions of the distance image by the method illustrated in FIG. 2 (S303).

(FIG. 3: Steps S304 to S305)
The first person detection unit 107 generates the person detection area information and the number of people detected by performing the person detection process on the RGB image (S304). The point cloud clustering unit 108 generates a point cloud cluster by performing a clustering process on the point cloud (S305).

(FIG. 3: Step S306)
If the number of detected persons generated in step S304 is larger than 0, the process proceeds to step S307, and if the number of detected persons generated in step S304 is 0, the process skips to step S310.

(FIG. 3: Step S307)
The person point cloud cluster association unit 109 classifies the person corresponding point cloud cluster and the person unsupported point cloud cluster by associating the person detection area information detected in step S304 with the point cloud cluster clustered in step S305.

(Fig. 3: Step S308)
In step S307, the person cluster removing unit 110 removes the person corresponding point cloud cluster associated with the person detection area information. This step is for convenience to clearly identify the unsupported point cloud cluster, and does not necessarily have to be performed as long as the unsupported point cloud cluster can be identified.

(FIG. 3: Steps S309 to S310)
The moving object detection unit 111 detects the human unsupported point cloud cluster remaining after step S308 (S309). The moving object detection unit 111 detects the movement operation parameters (for example, movement amount, speed, etc.) of the human unsupported point cloud cluster detected in step S309 (S310).

(Fig. 3: Step S311)
The second person detection unit 112 determines whether or not the movement operation parameter detected in step S310 is equivalent to a person. The second person detection unit 112 regenerates the number of people detected and the person detection area information by integrating the determination result into the number of people detected and the person detection area information acquired from the first person detection unit 107.

(Fig. 3: Step S312)
The human detection system 1 confirms whether or not a termination command has occurred. Examples of the exit command include an exit code generated by exception handling from the inside, and an exit command manually input from the outside. If there is an end command, this flowchart is ended, otherwise the process returns to step S301 and the same process is repeated.

FIG. 4A is an example of a shooting environment. In FIG. 4A, two people overlap and a part of the person behind is hidden by the person in front. Therefore, when only the person detection process for the RGB image is used, the person behind cannot be detected, and when only the distance image is used, two people are clustered as a lump. In such a situation, it will be described that two people can be detected by the flowchart of FIG.

The RGB image 401A is an RGB image acquired by the RGB image acquisition unit 103 from the RGB camera. The distance image 402A is a distance image acquired by the distance image acquisition unit 104 from the ToF sensor 102. The field of view area 403A is the field of view area of the RGB camera 101. The visual field region 404A is the visual field region of the ToF sensor 102. The subject 405A and the subject 406A are persons who exist within the range of the visual field area 403A and the visual field area 404A. The subject 407A and the subject 408A are the subjects 405A and 406A on the RGB image 401A captured and acquired by the RGB camera 101, respectively. The subjects 409A and 410A are the subjects 405A and 406A on the distance image 402A captured and acquired by the ToF sensor 102, respectively.

FIG. 4B is a processing flow diagram showing a procedure for detecting the subjects 405A and 406A in FIG. 4A according to the flowchart of FIG. Hereinafter, a process of distinguishing two subjects 405A and 406A will be described with reference to FIG. 4B.

The processing flow 41B shows the processing for the image acquired from the RGB camera 101 and the ToF sensor 102 in FIG. 4A. From the viewpoint of the RGB camera 101 and the ToF sensor 102, a part of the rear subjects 408A and 410A is hidden by the front subjects 407A and 409A, and the RGB images and the distance images acquired in steps S301 and S302 reflect this. doing. On the RGB image, only the subject 407A in front is detected and only the person detection area information 401B is output (step S304). On the distance image, the subjects 409A and 410A are clustered as a single point cloud cluster 402B (step S305). When the person detection area information 401B is associated with the point cloud cluster 402B (step S307), it is divided into a person corresponding point group cluster 403B and a person unsupported point group cluster 404B. The human-corresponding point cloud cluster 403B is removed (step S308), the unhumanized point cloud cluster 404B is detected (step S309), and the detection point group region 405B is output.

The processing flow 42B shows processing for an image acquired from the RGB camera 101 and the ToF sensor 102 after a time step of one frame from the processing flow 41B. The person detection area information 401B is the detection information of the subject 407A detected in step S304 in the processing flow 41B. In the point cloud cluster 402B, the subjects 409A and 410A are clustered together by step S305 in the processing flow 41B. The detection point group area 405B is information representing the detection result of the human unsupported point cloud cluster 404B in the processing flow 41B. The unhumanized point cloud cluster 406B is a movement of the unhumanized point cloud cluster 404B at the time of the processing flow 41B in the processing flow 42B. The detection point group region 407B is information representing the detection result of the human unsupported point cloud cluster 406B in the processing flow 42B.

In the processing flow 42B, after undergoing the same processing as in the processing flow 41B, the unsupported point cloud cluster 404B moves to become the unsupported human point cloud cluster 406B after undergoing a time step for one frame. By detecting this, the detection point group area 407B is output. In step S310, the movement operation parameter 408B (movement amount or velocity) between the detection point group regions 405B and 407B is detected.

<Embodiment 1: Summary>
The person detection system 1 according to the first embodiment identifies the unsupported point cloud cluster by associating the detected human detection area in the RGB image with the point cloud cluster, and sets it as a movement operation parameter of the unsupported point cloud cluster. Therefore, it is determined whether or not the day and the unpaired point cloud cluster are humans. As a result, a plurality of people are in close contact with each other, and the person behind the RGB image cannot be detected only by the person detection process, and the point cloud clustering process for the point cloud clusters the plurality of people as a group. However, it can accurately detect multiple people.

<Embodiment 2>
In the second embodiment of the present invention, a configuration example in which a person is detected according to a feature amount related to the shape of the point cloud cluster will be described instead of the movement operation parameter of the point cloud cluster that does not correspond to a person. Since other configurations are the same as those in the first embodiment, the differences will be mainly described below.

FIG. 5 is a configuration diagram of the person detection system 1 according to the second embodiment. In the second embodiment, the second information processing unit 13 includes a point cloud cluster fitting unit 504 instead of the moving object detection unit 111. The second information processing unit 13 further includes a human cluster shape feature amount extraction unit 502, a feature amount holding unit 503, and a point cloud cluster fitting unit 504.

In addition to the processing described in the first embodiment, the person point cloud cluster association unit 109 outputs the person correspondence point group cluster to the person cluster shape feature amount extraction unit 502. The person cluster shape feature amount extraction unit 502 extracts the feature amount related to the shape of the person correspondence point group cluster acquired from the person point cloud cluster association unit 109 and outputs the feature amount to the feature amount holding unit 503.

The feature amount holding unit 503 acquires the feature amount from the human cluster shape feature amount extracting unit 502, holds the acquired feature amount according to the acquired feature amount and the previously held feature amount, and points the retained feature amount. Output to the group cluster fitting unit 504. The feature amount acquired from the human cluster shape feature amount extraction unit 502 may be retained in a cycle of a certain period, or may be retained every time the feature amount is acquired from the human cluster shape feature amount extraction unit 502. Alternatively, when the feature amount acquired from the human cluster shape feature amount extraction unit 502 is significantly different from any of the feature amounts held in the feature amount holding unit 503, the feature amount acquired from the human cluster shape feature amount extraction unit 502 is sequentially added. It may be retained, or the feature amount acquired from the human cluster shape feature amount extraction unit 502 may be output as it is. One or more retained features may be output at random. Output rules are not limited to random.

The point cloud cluster fitting unit 504 calculates the feature amount for the unsupported point cloud cluster acquired from the person cluster removal unit 110 according to the feature amount acquired from the feature amount holding unit 503, and performs the fitting process. The fitting result is output to the second person detection unit.

The second person detection unit 112 determines whether or not the fitting is sufficient based on the fitting result obtained from the point cloud cluster fitting unit 504. If the fitting is sufficient, the number of people detected and the person detection area information are generated again by integrating the number of people detected and the person detection area information acquired from the first person detection unit 107 with the fitting result.

FIG. 6 is a conceptual diagram illustrating details of the processing performed by the human cluster shape feature amount extraction unit 502, the point cloud cluster fitting unit 504, and the second person detection unit 112. Here, a process of associating a human-corresponding point cloud cluster with a human-uncorresponding point group cluster will be described using an example in which a human face portion is schematically represented by spheres (reference numerals 601 and 606 in FIG. 6).

The person-corresponding point cloud cluster 601 is a person-corresponding point cloud cluster acquired from the person-point cloud cluster association unit 109. Each of the representative points 602 is a certain point in the person-corresponding point cloud cluster 601. For example, the points remaining after thinning out the point clouds of the human-corresponding point cloud cluster 601 at equal intervals may be adopted as the representative points 602. The reference sphere 603 is a sphere having a certain radius centered on the representative point 602. The normal vector 604 is a vector having the smallest eigenvalue among the eigenvectors of the covariance matrix generated based on the vector from the representative point 602 to each point in the reference sphere 603.

The human feature amount 605 is a frequency distribution for the internal angle between the vector from the representative point 602 to each point in the reference sphere 603 and the normal vector 604. This frequency distribution represents the surface shape of the human-corresponding point cloud cluster 601 (that is, the human face). If a human face also exists in the human uncorresponding point cloud cluster, it is considered that the human uncorresponding point cloud cluster also has a frequency distribution having a similar shape.

As a method for obtaining the human feature amount 605, for example, an extraction method such as an FPFH (Fast Point Histogram) feature amount or a SHOT (Signature of Histograms of Origin States) feature amount can be used.

The human unsupported point cloud cluster 606 is a human unsupported point cloud cluster acquired from the human cluster removal unit 110. The representative points 607, 610, and 613 are one point in the unsupported point cloud cluster 606. The normal vectors 609, 612, and 615 are normal vectors of the reference spheres 608, 611, and 614, respectively. Each normal vector can be obtained in the same manner as the method for obtaining the normal vector 604 in the reference sphere 603. The human feature amounts 616, 617, and 618 are obtained for the representative points 607, 610, and 613 in the same manner as the human feature amount 605.

The human cluster shape feature amount extraction unit 502 obtains the human feature amount 605 by setting the reference sphere 603 and calculating the normal vector 604 at the representative point 602 of the human correspondence point group cluster 601.

The point cloud cluster fitting unit 504 calculates the unhumanized feature amount for all the representative points of the unhumanized point cloud cluster 606 with respect to the unhumanized point cloud cluster 606 in the same manner as described above. Specifically, for the representative points 607, 610, and 613, the reference spheres 608, 611, and 614 are set and the normal vectors 609, 612, and 615 are calculated to obtain the human-unsupported feature amounts 616 to 618.

The point cloud cluster fitting unit 504 searches for the one most similar to the human feature amount 605 among the unsupported feature amounts 616 to 618. Let the unsupported feature amount obtained by the search be the corresponding feature amount. As a search method, for example, the average square error between each of the human feature quantities 616 to 620 and the human feature quantity 605 may be obtained, and the human uncorresponding feature quantity that minimizes this may be searched. In addition to this, for example, a fitting process by RANSAC (RANdom Sample Consensus) may be used. In FIG. 6, the corresponding feature amount of the human feature amount 605 is the non-human feature amount 616.

From the above procedure, it can be seen that the representative point 607 corresponds to the representative point 602 in the unsupported point cloud cluster 606. If the unhumanized point cloud cluster 606 is a human face, the average square error between the human feature amount 605 and the unhumanized feature amount 616 is assumed to be sufficiently small, and if it is not a human face, it is assumed to be large.

The second person detection unit 112 has sufficient fitting if the average square error is less than the threshold value with respect to the fitting result acquired from the point group cluster fitting unit 504 (that is, the person unsupported point group cluster 606 is a person). By integrating the fitting result with the number of people detected and the person detection area information acquired from the first person detection unit 107, the number of people detected and the person detection area information are generated again.

In FIG. 6, the representative point 602 is mentioned as the representative point of the human correspondence point cloud cluster 601 but the representative point is not limited to one point. When there are a plurality of representative points in the person correspondence point cloud cluster 601, the corresponding representative points in the person uncorresponding point group cluster 606 are searched for each representative point. In addition, the representative points 607, 610, and 613 are mentioned as the representative points of the human unsupported point cloud cluster 606, but the representative points are not limited to three points. When there are a plurality of representative points in the human-corresponding point cloud cluster, the minimum average squared error between the two clusters may be acquired for each representative point, and the average value thereof may be used as the fitting result. The same applies to FIGS. 7A to 7B described later.

FIG. 7A is a flowchart illustrating a process in which the person detection system 1 in the second embodiment detects a person in an image. Compared with FIG. 3, steps S701 to S705 were newly established, and steps S309 to S311 were removed. Hereinafter, steps different from those in FIG. 3 will be mainly described.

(FIG. 7A: steps S701 to S702)
The human cluster shape feature amount extraction unit 501 extracts the human correspondence feature amount of the human correspondence point group cluster (S701). The feature amount holding unit 503 holds the feature amount corresponding to the person extracted in step S701 (S702).

(FIG. 7A: step S703)
The point cloud cluster fitting unit 504 acquires the minimum average squared error between both clusters as a fitting result by associating the point cloud clusters corresponding to humans with the point cloud clusters not corresponding to humans according to the procedure described with reference to FIG. ..

(FIG. 7A: step S704)
The second person detection unit 112 determines whether or not the fitting is sufficient based on the fitting result obtained from the point cloud cluster fitting unit 504. If the fitting is sufficient, the number of people detected and the person detection area information are generated again by integrating the fitting result into the number of people detected and the person detection area information acquired from the first person detection unit 107.

(FIG. 7A: step S705)
The point cloud cluster fitting unit 504 reads out the human-compatible feature amount held by the feature amount holding unit 503, and executes step S703.

FIG. 7B is a flowchart illustrating the details of step S703. Steps S70301 to S70305 are processes for searching for a representative point in the person-corresponding point cloud cluster corresponding to one representative point in the person-corresponding point cloud cluster. Step S70306 is a loop process for carrying out the same process for each representative point in the human correspondence point cloud cluster. Each step of FIG. 7B will be described below.

(FIG. 7B: Steps S70301-S70303)
The point cloud cluster fitting unit 504 selects any one of the human-compatible feature quantities acquired from the feature quantity holding unit 503 (S7031). The point cloud cluster fitting unit 504 selects one of the representative points in the unsupported point cloud cluster, and calculates the unsupported feature amount for humans based on the representative points (S70302). The point cloud cluster fitting unit 504 calculates the average squared error between the human feature amount and the unhumanized feature amount (S70303).

(FIG. 7B: step S70304)
The point cloud cluster fitting unit 504 determines whether or not step S70302 and step S70303 have been performed for all the unhumanized feature quantities (that is, all the representative points) of the unhumanized point cloud cluster. If the implementation is completed for all the unsupported features, the process proceeds to step S70305. If not, the process returns to step S70302 for the unselected features that have not been selected, and the same process is repeated.

(FIG. 7B: step S70305)
The point cloud cluster fitting unit 504 searches for the minimum value among the average squared errors calculated in S70303. This step corresponds to searching for a representative point corresponding to the representative point selected in S7031 among the representative points of the human uncorresponding point cloud cluster.

(FIG. 7B: step S70306)
The point cloud cluster fitting unit 504 determines whether or not steps S70302 to S70305 have been performed for all human features (that is, all representative points) of the human-corresponding point cloud cluster. If the implementation is completed for all human features, the process proceeds to step S70307, and if not completed, the human features that have not yet been selected (that is, the representative points that have not yet been selected in the human correspondence point cloud cluster) are The process returns to step S70301 and the same process is repeated.

(FIG. 7B: step S70307)
The point cloud cluster fitting unit 504 acquires the minimum average squared error for all the human features in the point cloud cluster corresponding to humans, and calculates the average value thereof. The second person detection unit 112 can use this average value as the fitting result.

<Embodiment 2: Summary>
In the person detection system 1 according to the second embodiment, the human-unsupported point cloud cluster is a person by searching the part corresponding to the shape feature amount of the human-compatible point cloud cluster in the human-unsupported point cloud cluster. Judge whether or not. As a result, as in the first embodiment, even when a plurality of people are in close contact with each other, the plurality of people can be accurately detected. Moreover, since the movement operation parameter is not used, there is an advantage that a person can be detected even when the person is not moving.

<Embodiment 3>
In the second embodiment of the present invention, a configuration example in which a person is detected according to the height of the point cloud cluster will be described instead of the movement operation parameter of the point cloud cluster that does not correspond to a person. Since other configurations are the same as those in the first embodiment, the differences will be mainly described below.

FIG. 8 is a configuration diagram of the person detection system 1 according to the third embodiment. In the third embodiment, the second information processing unit 13 includes a height filtering unit 801 and a point cloud density maximum value detecting unit 802 instead of the moving object detecting unit 111.

The height filtering unit 801 removes a region less than a certain height from the human uncorresponding point cloud cluster acquired from the human cluster removing unit 110, and outputs the removed region to the point cloud density maximum value detecting unit 802. The point cloud density maximum value detection unit 802 searches for the height at which the density of the point cloud becomes the maximum value in the unsupported point cloud cluster acquired from the height filtering unit 801 and detects the height and density of the second person. Output to unit 112.

The second person detection unit 112 determines whether or not the height and density acquired from the point cloud density maximum value detection unit 802 are values equivalent to a person, and the determination result is obtained from the first person detection unit 107. By integrating the number and the person detection area information, the number of people detected and the person detection area information are generated again.

FIG. 9 is a diagram illustrating details of a person detection method by the point cloud density maximum value detection unit 802 and the second person detection unit 112. The density distribution 91 is a frequency distribution for the density of the point cloud of the unsupported point cloud cluster at a certain height. The threshold value 901 is a threshold value when the height filtering unit 801 filters the point cloud. The detection point 902 is a point showing a maximum value in the density distribution 91, and has height and density values.

The point cloud density maximum value detection unit 802 acquires the unsupported point cloud clusters removed by the height filtering unit 801 according to the threshold value 901, and calculates the density distribution 91. The point cloud density maximum value detection unit 802 detects the detection point 902 by searching for the maximum value in the density distribution 91. The point cloud density maximum value detection unit 802 outputs the detection point 902 to the second person detection unit 112. The second person detection unit 112 detects a person if the detection points 902 acquired from the point cloud density maximum value detection unit 802 are within a predetermined height range and equal to or higher than a predetermined density. In general, the density with respect to the height of the point cloud changes depending on the installation position of the ToF sensor 102, so the density using the height as a function may be used as a criterion.

FIG. 10 is a flowchart illustrating a process in which the person detection system 1 in the third embodiment detects a person in an image. Compared with FIG. 3, steps S1001 to S1003 were newly established, and steps S309 to S311 were removed. Hereinafter, steps different from those in FIG. 3 will be mainly described.

(FIG. 10: Step S1001)
The height filtering unit 801 generates a human uncorresponding point cloud cluster by removing the region less than the threshold value 901 from the human uncorresponding point cloud cluster associated with the human detection region in step S307.

(FIG. 10: Step S1002)
The point cloud density maximum value detection unit 802 detects the height at which the density of the point cloud becomes the maximum value and the density at that height in the unsupported point cloud cluster generated in step S1001.

(FIG. 10: Step S1003)
The second person detection unit 112 determines whether or not the height and density detected in step S1002 are values equivalent to a person. When it is determined that the person is a person, the number of people detected and the person detection area information are generated again by integrating the determination result into the number of people detected and the person detection area information acquired from the first person detection unit 107.

(FIG. 10: Steps S1002 to S1003: Supplement)
When there are a plurality of maximum values in the density distribution 91, each maximum value may be a person. Therefore, the point cloud density maximum value detection unit 802 and the second person detection unit 112 may determine whether or not the person is a person for each maximum value. The determination procedure is the same as that described with reference to FIG.

<Embodiment 3: Summary>
The person detection system 1 according to the third embodiment filters the height of the unsupported point cloud cluster, and then determines whether or not the maximum value of the height distribution is within a predetermined range. Correspondence point group Determine if the cluster is a person. As a result, as in the first embodiment, even when a plurality of people are in close contact with each other, the plurality of people can be accurately detected. Moreover, since the movement operation parameter is not used, there is an advantage that a person can be detected even when the person is not moving. Furthermore, since the density distribution for height can be calculated by scanning the point cloud only once, it is possible to accurately detect a plurality of people while suppressing the calculation cost.

<About a modified example of the present invention>
The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

In the above embodiment, the first person detection unit 107 can detect the human face region by performing the detection on the face RGB image. The second person detection unit 112 can also consider the area of the point cloud cluster in which the human body is assumed to exist below each face area as a person. For example, a cylindrical region of a predetermined size below the face region may be regarded as a person. As a result, if a plurality of people overlap in an area other than the face, the person can be detected by a simple process.

In the above embodiment, the synchronization information holding unit 106 can be configured by a storage device that stores the coordinates (gx, gy) and the rotation angle φ. The first information processing unit 12 and the second information processing unit 13 can be configured by using hardware such as a circuit device that implements these functions, and software that implements these functions is provided by a CPU (Central Processing Unit). ) And other arithmetic units can be executed. The same applies to each functional unit included in the first information processing unit 12 and the second information processing unit 13.

1: Person detection system 10: Person detection device 11: Imaging processing unit 101: RGB camera 102 ... ToF sensor 12: First information processing unit 103: RGB image acquisition unit 104: Distance image acquisition unit 105: Pixel synchronization unit 106: Synchronization Information holding unit 107: First person detection unit 108: Point cloud clustering unit 109: Point cloud cluster association unit 110: Person cluster removal unit 13: Second information processing unit 111: Moving object detection unit 112: Second person detection unit 502: Human cluster shape feature amount extraction unit 503: Feature amount holding unit 504: Point cloud cluster fitting unit 801: Height filtering unit 802: Point cloud density maximum value detection unit

Claims (12)

A person detection device that detects people included in an image.
First person detection unit that detects people included in the RGB image acquired from the RGB camera,
A point cloud grouping unit that acquires a point cloud group by spatially grouping the point clouds included in the point cloud data generated from the point cloud image acquired from the ToF sensor.
A mapping processing unit that associates a person detection region in which a person is detected in the RGB image by the first person detection unit with the point cloud group grouped by the point cloud grouping unit.
Among the point group groups, the association processing unit determines whether or not the uncorresponding point group group is a person according to the first feature amount of the uncorresponding point group group that is not associated with the person detection area. , Second person detector,
A person detection device comprising. The person detection device further includes a synchronization processing unit that calculates synchronization data representing a difference between the pixel position of the RGB image and the pixel position of the point cloud image.
The point cloud grouping unit is characterized in that the RGB image and the point cloud image are aligned according to the synchronization data, and then the point cloud group is acquired from the point cloud image. Item 1. The person detection device according to item 1. The synchronization processing unit periodically updates the difference between the pixel position of the RGB image and the pixel position of the point cloud image by periodically calculating the synchronization data. Item 2. The person detection device according to item 2. The second person detection unit calculates a parameter describing the movement operation of the uncorresponding point group group as the first feature amount, and calculates the parameter.
The second person detection unit is characterized in that it determines whether or not the uncorresponding point group group is a person by comparing a parameter representing the movement movement with a threshold value representing the movement movement of a person. The person detection device according to claim 1. The person detection device further includes a first feature amount calculation unit that calculates a feature amount representing the shape of the uncorresponding point group group as the first feature amount.
The person detection device further includes a second feature amount calculation unit in which the association processing unit calculates a parameter representing the shape of the point cloud group associated with the person detection region as a second feature amount.
The second person detection unit according to claim 1, wherein the second person detection unit determines whether or not the uncorresponding point group is a person by comparing the first feature amount with the second feature amount. Person detection device. The first feature amount calculation unit calculates a first histogram of the angle formed by the normal vector of the surface of the uncorresponding point group group with the reference angle as the first feature amount.
The second feature amount calculation unit obtains a second histogram of the angle formed by the normal vector of the surface of the point cloud group associated with the person detection region by the association processing unit with respect to the reference angle. Created as the second feature amount,
The second person detection unit determines whether or not the uncorresponding point cloud group is a person according to whether or not the difference between the first histogram and the second histogram is less than the threshold value. 5. The person detection device according to claim 5. The person detection device further includes height filtering for filtering point groups whose height is less than a threshold value among the point groups included in the uncorresponding point group group.
The person detection device further includes a height histogram creating unit that creates a height histogram representing the frequency distribution of the heights of the point groups included in the uncorresponding point group after the filtering.
The second person detection unit according to claim 1, wherein the second person detection unit determines whether or not the uncorresponding point cloud group is a person according to whether or not the height histogram satisfies the person detection condition. Human detector. In the person detection condition, if the maximum value of the frequency value of the height histogram is within the predetermined range and the density of the point group is equal to or more than the predetermined value, the uncorresponding point group group is regarded as a person, otherwise. The person detection device according to claim 7, wherein the non-corresponding point group group is configured to be regarded as not a person. The second person detection unit determines whether or not the person detection condition is satisfied for each of the maximum values, thereby determining whether or not the uncorresponding point group group is a person for each of the maximum values. 8. The person detection device according to claim 8. The first person detection unit detects a person's face region included in the RGB image by performing face detection on the RGB image.
The second person detection unit is characterized in that, among the uncorresponding point group groups, a point cloud existing in a predetermined range below the face region is regarded as a person corresponding to the face region. 1. The person detection device according to 1. It is a person detection system that detects people included in images.
RGB camera,
ToF sensor,
Human detector,
With
The person detection device is
A first person detection unit that detects a person included in an RGB image acquired from the RGB camera,
A point cloud grouping unit that acquires a point cloud group by spatially grouping the point clouds included in the point cloud data generated from the point cloud image acquired from the ToF sensor.
A mapping processing unit that associates a person detection region in which a person is detected in the RGB image by the first person detection unit with the point cloud group grouped by the point cloud grouping unit.
Among the point group groups, the association processing unit determines whether or not the uncorresponding point group group is a person according to the first feature amount of the uncorresponding point group group that is not associated with the person detection area. , Second person detector,
A person detection system characterized by being equipped with. It is a person detection method that detects people included in the image.
Steps to detect people included in an RGB image acquired from an RGB camera,
A step of acquiring a point cloud group by spatially grouping the point clouds included in the point cloud data generated from the point cloud image acquired from the ToF sensor.
A step of associating a person detection region in which a person is detected in the RGB image with the grouped point cloud group.
A step of determining whether or not the uncorresponding point group is a person according to the first feature amount of the uncorresponding point group that is not associated with the person detection region in the associating step of the point group. ,
A person detection method characterized by having.

Images