JP5950296B2 - Person tracking attribute estimation device, person tracking attribute estimation method, program - Google Patents

Description

  The present invention relates to a method and apparatus for detecting and tracking a person's position, and by adding person recognition by a camera, it is possible to link person attribute information (gender, age, etc.) to person tracking. is there.

  A system for detecting and tracking a person in a three-dimensional real space with a radar range sensor has been put into practical use. A conventional person position detection / tracking apparatus detects an object position by laser irradiation and determines a person from the size of the detected object.

In addition, as a technique in which a laser range sensor and a camera are integrated, a technique described in Document 1 below has been proposed. This is the weighted sum of the probability density distribution that complementarily integrates the position of the human head in the three-dimensional real space from the distance observation by the laser range sensor and the head shape from the observation by the camera. By using it, the accuracy and robustness of person tracking is improved.
Takanori Kobayashi, et al. “Person Tracking in Areas by Integration of Distributed Camera and Laser Range Sensor” Image Recognition and Understanding Symposium (MIRU2007) July 2007

  The human position detection tracking of the conventional laser range sensor is based on the delay time (light flight time) of the reflected light with respect to the projection on the circumference due to the rotation of the internal mirror of the laser beam by the laser range sensor. By measuring the distance and direction to the object and the size of the object, the position of the person estimated from the size of the object is detected and tracked, and the person attribute cannot be estimated. In addition, the human tracking that integrates the conventional laser range sensor and camera is a method that complementarily integrates different sensor modalities of distance observation by the laser range sensor and appearance observation by the camera. The purpose was to improve the accuracy and robustness of person tracking, and person attribute information could not be linked to person tracking.

  The present invention relates to a person tracking attribute estimation apparatus that associates attribute estimation with person position detection tracking, reduces the amount of calculation, and improves detection accuracy.

The human tracking attribute estimation apparatus according to the present invention includes a human candidate clustering unit that extracts point candidates by clustering point groups that are within a predetermined threshold from measurement data of a laser range sensor, and a human candidate between scan frames. comparison, the distance difference grant candidate human image ID person candidate is within a predetermined range as the same person candidate, and the character candidate tracking section for outputting the position coordinates of a candidate human ID same person candidate and the position coordinates A viewport conversion unit that converts the image into two-dimensional image coordinates, and an upper-body search that sets a window size and coordinate data as an upper-body search region corresponding to the person candidate given the person candidate ID based on the two-dimensional image coordinates an area setting unit, a candidate human ID from the upper body search area setting unit, based on the window size and the coordinate data, a plurality of the same person within the set range The upper body search area of the auxiliary ID integrated into one as the detection window of the person ID, have rows face detection of a person in the detection window in the frame image data from the camera, People attribute of performing a further face detection estimating the a person attribute estimation unit configured to output together with the person ID corresponding to the person attribute, and the person candidate ID and position coordinates output from the candidate human image tracking unit, and the personal ID and the person attribute information from the person attribute estimation unit based on, and having a a person tracking attribute determining unit for performing the association between the person ID and position coordinates and the person attribute information matching the candidate human ID.

In addition, the person tracking attribute determination unit of the person tracking attribute estimation apparatus of the present invention determines the person attribute of the person ID based on the person attribute information to which the person ID of the same person is assigned in a plurality of frame image data. May be.

In the person tracking attribute estimation apparatus, the person tracking attribute determination unit may determine the person attribute of the person ID based on one of an average value, a median value, and a majority vote of a plurality of person attribute information.

In the person tracking attribute estimation unit, person tracking attribute determining unit further receives the thumbnail image from the person attribute estimation unit, based on the position coordinates corresponding to the person ID, plotting said thumbnail image and the person attribute information Then, it may be output so as to be displayed on the display device.

In the person tracking attribute estimation apparatus, the person tracking attribute determination unit may calculate a motion vector from the position coordinates corresponding to the person ID, and determine a motion with respect to the set side.

In the person tracking attribute estimation apparatus, the person tracking attribute determination unit may total person attribute information of a plurality of person IDs.

In the person tracking attribute estimation apparatus, the person tracking attribute determination unit may total the person attribute information of a plurality of person IDs and movements with respect to the set side.

The human tracking attribute estimation method according to the present invention includes a human candidate clustering step of extracting a human candidate by clustering a point cloud having a distance within a predetermined threshold from measurement data of a laser range sensor, and the human candidate between scan frames. And a candidate candidate tracking step of assigning a candidate person ID to a candidate person whose distance difference is within a predetermined range as the same candidate person, and outputting the position candidate coordinates and the position coordinates of the same candidate person. viewport transformation step of converting the position coordinates into the two-dimensional image coordinates, on the basis of the two-dimensional image coordinates, set the window size and the coordinate data as the upper body search region corresponding to the person candidate imparted with the person candidate ID The upper body search area setting step, the person candidate ID set in the upper body search area setting step, the window size, and Based on the target data, the upper body search areas of a plurality of the same person candidate IDs within the set range are integrated into a person ID detection window to detect a person's face in the detection window in the frame image data from the camera. The person attribute estimation step for outputting the person attribute together with the corresponding person ID, the person candidate ID and position coordinates output in the person candidate tracking step, and the person attribute A person tracking attribute determination step for associating a person ID that matches the person candidate ID with the position coordinates and the person attribute information based on the person ID and the person attribute information output in the estimation step. To do.

The human tracking attribute estimation program according to the present invention is a person candidate clustering step for clustering a point cloud that is a distance within a predetermined threshold from measurement data of a laser range sensor and extracting a human candidate on a computer. A person candidate that compares between scanning frames, assigns a person candidate ID as a person candidate whose distance difference is within a predetermined range, and outputs the person candidate ID and the position coordinates of the same person candidate tracking step, viewport transformation step of converting the position coordinates into the two-dimensional image coordinates, on the basis of the two-dimensional image coordinates, window size and the coordinate data as the upper body search region corresponding to the person candidate imparted with the person candidate ID Upper body search area setting step for setting the human candidate ID set in the upper body search area setting step, Based on the window size and coordinate data, the upper body search areas of the same person candidate ID within the set range are integrated into a person ID detection window, and the person in the detection window in the frame image data from the camera The person attribute of the person who performed the face detection is estimated, the person attribute of the person who has detected the face is estimated, and the person attribute is output together with the corresponding person ID. And a person tracking attribute determining step for associating the person ID that matches the person candidate ID with the position coordinates and the person attribute information based on the person ID and the person attribute information output in the person attribute estimating step. It is characterized by that.

  According to the present invention, the person tracking / attribute integration unit 30 is provided, so that a three-dimensional real space position tracking function of an object by a laser range sensor and a face detection attribute estimation function by a camera are respectively supported for a plurality of tracking persons. And can be tied. In addition, the upper body identification unit 41 can identify the upper body by using the data from the upper body search area setting unit 32, thereby reducing the number of scans. Furthermore, it is possible to realize 3D real space person tracking with person attribute information.

  In addition, it is possible to make a comprehensive judgment of person attributes in a sequence image using a person ID from a laser range sensor, and in a state where a change in lighting conditions, a change in face orientation, a face hiding, etc. have occurred In addition, stable and highly accurate person attribute determination can be performed.

  Furthermore, it is possible to generate a person tracking / attribute integrated bird's-eye view that integrates the person tracking position information from the laser range sensor and the person attribute information from the camera image, and the action tracking with attribute information and a person photograph can be used as a bird's eye view. It can be displayed very clearly.

  In addition, it is possible to collect behaviors linked to person attributes. For example, it is possible to collect store customers' buying behavior, and by knowing the flow in the store of the store customers that cannot be seen from the POS data, Placement and floor layout can be optimized. It is also possible to strategically examine store products and restaurant menus in office store complex buildings by grasping the flow of people coming out of elevators in the building by time zone. .

It is an example of the installation arrangement | positioning of the person tracking attribute system comprised from the camera concerning this invention, a laser range sensor, and a person tracking attribute estimation apparatus. It is a block diagram of a person tracking attribute presumption system in one embodiment of the present invention. It is a figure showing the relation of world coordinates, camera coordinates, and image coordinates. It is an example of the change step of the size of the upper body detection window when the upper body image is a square. It is an example of an image when a plurality of upper body identification windows are integrated into one upper body detection window. Data flow between a human candidate detection tracking unit 20 to which a signal from the laser range sensor 11 is input, a human tracking / attribute integration unit 30, and a human attribute estimation unit 40 to which an image from the camera 10 is input It is an example. It is an example of the determination processing flow in the person tracking attribute determination part 33. It is a specific flow of the determination method in the person tracking attribute determination part 33. FIG. It is a flow of the determination method in the next processed image (processed image number = 2). Furthermore, it is a flow of the determination method in the next processed image (processed image number = 3). It is the flow of the determination method in process image number = 4. It is a flow of the determination method in process image number = 5. It is an example of the screen output from the person tracking attribute determination part 33. It is an example of arrangement | positioning of the camera 10 and the laser range sensor 11. FIG. It is a figure which shows the person's entrance / exit determination method. It is a figure showing an example of the result of totaling the number of people who went to the shop & restaurant direction and the number of people who went to the entrance / exit direction after exiting the elevator by men's age group for each time zone from 7 am to 10 pm is there. It is a figure showing an example of the result of totaling the number of people who went to the shop & restaurant direction and the number of people who went to the entrance / exit direction by going out the elevators by women's age group for each time period from 7 am to 10 pm is there. It is a figure of an example which totaled the number of persons and the age ratio for every person who came out of the elevator. It is an example of the figure which calculated | required the number of persons and the ratio of the other side after leaving the elevator for every time slot | zone in the man of 40 years old or older and less than 60 years old. It is an example of the figure which calculated | required the number of persons and the ratio of the other side after leaving the elevator restrict | squeezed to the woman 20 years old or older and younger than 40 years old.

  An embodiment in which the present invention is applied to visitor tracking and attribute estimation in an arbitrary area in a facility will be described below. In this embodiment, the laser range sensor is installed so that the laser emits light in a horizontal circumferential direction, and the camera is installed at a position where a person's face in the area is captured in an appropriate size. Real-time 3D coordinate detection and tracking of human candidates in the area is performed using a laser range sensor. Also, with the camera, the person detection and person attribute estimation corresponding to each person candidate from the laser range sensor is performed from the image position obtained by projecting the real space 3D coordinates of the person candidate from the laser range sensor to the camera image coordinates. In addition, the result is comprehensively determined by the average value, median value, majority vote, etc. using the tracking information from the laser range sensor, so that highly accurate and stable person attribute information can be linked to person tracking.

FIG. 1 is an example of an installation arrangement of a person tracking attribute system including a camera, a laser range sensor, and a person tracking attribute estimation device according to the present invention. The camera 10 and the laser range sensor 11 are respectively installed so as to be fixed. The camera 10 is a video camera, and is arranged and arranged so that a person's face in the person detection / tracking area is captured with an appropriate size. For example, the laser range sensor 11 is disposed so as to project a laser beam horizontally at a height of 80 cm from the floor surface assuming the waist height of a person. The camera 10 and the laser range sensor 11 are connected to the person tracking attribute estimation device 12 through interfaces such as USB, LAN, and RS-232C.

  The laser range sensor 11 irradiates a laser, and measures the distance to the object from the time it takes for the object to reflect and return. By reflecting the laser emitted from the transmitter with a rotating mirror and scanning the front in a fan shape, the distance from the floor surface to an object existing on a flat surface can be measured. Since this system is equipped with the laser range sensor 11 as shown in FIG. 1, the plane is scanned with a height of 80 cm from the floor parallel to the floor.

  The rotating mirror of the laser range sensor 11 uses a step angle, and the scanning direction scans counterclockwise. For example, 1 [step] is about 1.05 [deg], the maximum number of steps is 228 [step], and a fan shape of about 240 degrees is the detection area. The laser range sensor 11 outputs measurement data composed of polar coordinate data on the r-θ plane at an angle θ and a distance r for each point. Measurement data of these detected point groups is input to the person tracking attribute estimation device 12 at the timing of the scanning frequency. Instead of the laser range sensor 11, for example, an acceleration sensor and an azimuth angle sensor may be combined with GPS to acquire angle and coordinate data. In addition, data may be acquired by performing distance measurement using a stereo camera.

  FIG. 2 is a block diagram of a person tracking attribute estimation system in one embodiment of the present invention. The person tracking attribute estimation system includes a camera 10, a laser range sensor 11, and a person tracking attribute estimation device 12. The person tracking attribute estimation device 12 includes a person candidate detection tracking unit 20, a person tracking / attribute integration unit 30, and a person attribute estimation unit 40. In one aspect, the person tracking attribute estimation device 12 is realized by cooperation of hardware resources and software, and is, for example, a computer. Specifically, the various functions of the person tracking attribute estimation device 12 are such that a person tracking attribute estimation program recorded in a recording medium such as a ROM or a hard disk is read into a main memory and a control device (for example, a processor such as a CPU). It is realized by being executed by. However, the person tracking attribute estimation device 12 may be realized only by hardware. Further, the person tracking attribute estimation device 12 may be physically realized by one device or may be realized by a plurality of devices. In addition, the person tracking attribute estimation device 12 includes an input interface (not shown) for inputting image frames taken by a camera and data output from a laser range sensor.

  The person candidate detection tracking unit 20 of the person tracking attribute estimation device 12 includes a polar coordinate / planar coordinate conversion unit 21, a background registration unit 22, a person candidate clustering unit 23, and a person candidate tracking unit 24.

  The polar coordinate / planar coordinate conversion unit 21 converts the measurement data having polar coordinate data of the r-θ plane of the angle θ and the distance r input from the laser range sensor 11 at the timing for each scanning frequency into the plane coordinates of the xy plane. Convert to data. Thereby, it can display as a bird's-eye view sketch.

  The background registration unit 22 registers in advance the measurement result of the laser range sensor as a background when there is no person in the person detection tracking area. Thereby, it is possible to detect an object other than the background by extracting only the point group having a large difference from the background measurement.

  The person candidate clustering unit 23 takes out a difference from the background portion registered in the background registration unit 22 that is the point group output from the laser range sensor 11, and performs clustering of points having a close distance difference. By clustering, the point cloud can be classified for each object. For example, if a point group with a width of 25 cm or more is a person candidate, the person candidate clustering unit 23 outputs the xy coordinates of the center of the point group as the person candidate coordinates.

  The person candidate tracking unit 24 assigns the same person candidate ID by determining that the person candidates having a close distance difference between the laser scanning frames are the same person, and performs person tracking. As described above, the person candidate detection tracking unit 20 outputs the person candidate ID and the xy position coordinates (x, y) for each detected person candidate to the person tracking / attribute estimation integration unit 30 for each laser scanning frame. To do.

  The person tracking / attribute integration unit 30 includes a viewport conversion unit 31, an upper body search area setting unit 32, and a person tracking attribute determination unit 33. When the xy position coordinates for each detected person candidate ID output from the person candidate detection / tracking unit 20 are input, the viewport conversion unit 31 determines the camera internal parameters and the camera externals obtained in advance by camera calibration. By projective conversion using parameters, the three-dimensional real space coordinates of the head corresponding to the human candidate ID detected by the laser range sensor 11 are converted into image coordinates.

  The upper body search region setting unit 32 takes into consideration the difference from the average height and detection error in the height from the chest to the top of the head, centering on the image coordinates of the human candidate head detected by the laser range sensor 11. Set the upper body search area. Further, the xy position coordinates of the human candidate detected from the laser range sensor 11 with the height of the upper end (top) and the height of the lower end (chest) of the upper body area from the average height as Zw of the three-dimensional space coordinates, respectively. By performing viewport conversion together with (Xw, Yw), the height of the upper body region in the image coordinates can be obtained. Further, from this information, the upper body detection window size range (the upper body size range to be searched) is set in consideration of the difference from the average height and detection error. These person candidate IDs, the corresponding upper body search scale range (window size), and search area (coordinate data) are input to the person attribute estimation unit 40. However, the output timing to the person attribute estimation unit 40 is not the timing of the laser scanning frequency, but the timing at which the person attribute estimation unit 40 ends the person estimation process and the attribute information is input to the person tracking attribute determination unit 33. . That is, the information of the laser scanning frame acquired during the time required for the person estimation process is not used. Regarding the output timing of the information of the person candidate ID, the upper body search area, and the scale range from the person tracking / attribute integrating unit 30 to the person attribute estimating unit 40, the person attribute estimating unit 40 finishes the person attribute estimating process for the person candidate ID. In response, the information regarding the next same person candidate ID is output.

  The person attribute estimation unit 40 includes an upper body identification unit 41, an upper body determination unit 42, and a face detection attribute estimation unit 43. The frame image data acquired by the camera 10 and the person candidate ID input from the upper body search area setting unit 32, Face detection is performed using the upper body search scale range and the search area, and person attributes are estimated.

The upper body identifying unit 41 uses a known image recognition technique to perform multi-frame detection on the frame image from the camera, using a well-known image recognition technique within the upper body search scale range and search region corresponding to the person candidate ID input from the upper body search region setting unit 32. The human upper body is detected on the scale, and the presence / absence of the upper body, the size of the detected upper body, and the image coordinate position are determined. It should be noted that the upper body identification unit 41 receives the frame image of the camera photographed at the same time when the human candidate ID, the upper body search region, and the scale range are input from the upper body search region setting unit 32 to the person attribute estimation unit 40. To detect the upper body of the person. Accordingly, not all camera frame images input to the person attribute estimation unit 40 are necessarily used.
Any known method can be applied as the process for identifying the upper body of the person. The method of the present invention will be described in the case where human upper body detection is performed by a support vector machine classifier using a luminance gradient direction histogram as a feature amount.

  The human upper body detection method is divided into a feature amount learning process and an identification process. In the learning process, a large number of upper body images and non-upper body images that are detection targets are prepared in advance, luminance gradient direction histogram feature amounts are extracted from each image, and upper body shape features are learned using a support vector machine. As a result, a discriminator that determines whether the luminance gradient direction histogram feature value extracted from the input image is the upper body of the person or not is obtained.

  In the identification process, an upper body detection window is set for the input image, and this detection window is sequentially scanned in the input image. While scanning, we extract the luminance gradient direction histogram feature amount of the region cut out by the upper body detection window for each scan, and evaluate the upper body quality of the detection window by the support vector machine classifier obtained by learning in advance Identify the upper body.

  Further, in order to detect the upper body of a person of various sizes in the input image, the size of the upper body detection window is changed, and sequential scanning in the input image is repeated to identify the upper body of each size.

  Based on the upper body detection window size range input from the upper body search area setting unit 32, the upper body identification unit 41 selects, for example, two or three levels of window sizes from the size of the upper body detection window having 16 levels. Scan. Therefore, the number of scans can be reduced by using the window size range input from the upper body search area setting unit 32. Further, by using the image coordinate data of the upper body search area input from the upper body search area setting unit 32, the detection window / scan range in the input image can be changed from the whole image scan to the upper body search area. . By using the three-dimensional spatial coordinates of the detected person candidate from the laser range sensor in this way, the number of scans of the upper body detection window in the input image can be dramatically reduced, and the area is limited to the person candidate area. Since upper body detection is performed, erroneous detection due to a complicated background or the like can be reduced. By these processes, the upper body identification unit 41 outputs the upper body identification result corresponding to each person candidate ID and the image coordinates and size of the identification window when the upper body is identified to the upper body determination unit 42.

  The upper body determination unit 42 uses the image coordinate position and size (window size) of a plurality of upper body identification windows, which are upper body identification results for each person candidate ID input from the upper body identification unit 41, to determine the position of the identification window. If the size is within a preset range, the same person upper body is assumed and a plurality of upper body identification windows corresponding to the same person candidate ID are integrated into one detection window. When the upper body of a person exists in the input image, a plurality of upper body identification windows are usually identified by being superimposed. Among the plurality of upper body identification windows, these are integrated into one identification window whose upper body is within a predetermined range.

  Note that the integration process may be performed by determining that the upper body has been detected only when a predetermined number or more of upper body identification windows are integrated. By doing in this way, false detection can also be reduced. In addition, as an integrated determination result of the upper body identification window, an upper body determination result corresponding to each person candidate ID, and if it is determined to be an upper body, the person candidate ID is set as a person ID, and the image coordinates and size of the upper body detection window are set. It outputs to the face detection attribute estimation part 43.

When the person ID and the image coordinates and size of the upper body detection window are input from the upper body determination unit 42, the face detection attribute estimation unit 43 determines the position of the face search scale range and the search area from the position and size of the upper body detection window. The multi-scale face detection is performed using a well-known image recognition method. Then , when a face is detected with a predetermined size or more, attribute estimation from the face image is performed for each person ID using a well-known image recognition method, and the person ID and person attribute information as a result of the estimation are obtained. The result is output to the person tracking attribute determination unit 33 together with the upper body determination result for each candidate person ID from the determination unit 42.

  Note that the human attribute estimation unit 40 first detects the upper body and performs face detection attribute estimation on the upper body detection area. However, in the person attribute estimation unit 40, an image from the upper body search area setting unit 32 is used. Based on the upper body search scale range and the search area information in the coordinates, only the face detection and the attribute estimation may be performed without performing the upper body detection.

In person tracking attribute determination unit 33, finally integrating the tracking result with the attribute information by the tracking result and camera 10 by the laser SOKUIKI sensor 11. The person tracking attribute determination unit 33 holds the xy position coordinates for each detected person candidate ID output from the person candidate detection tracking unit 20. Moreover, the person tracking attribute determining section 33, the person attribute information corresponding to the person ID from the face detecting attribute estimating unit 43 is input, stores them temporarily. Also, the upper body search area setting unit 32 is notified that the person attribute information has been input. The person tracking attribute determination unit 33 compares the person ID input from the face detection attribute estimation unit 43 with the person candidate ID tracked by the laser range sensor 11 and input from the person candidate detection tracking unit 20, and the same. Information having person IDs is associated with each other. In addition, with respect to the person attribute information corresponding to the same person ID in different image frame data, the attribute for the same person tracked by taking the average value, median value, majority vote, etc. of the determination result and its parameters for each attribute eliminating the fluctuation of the information, to increase the accuracy, and outputs in association with the person attribute information to the person tracking the three-dimensional real space position information. Details of the processing in the person tracking attribute determination unit 33 will be described later.

  FIG. 3 is a diagram showing the relationship between world coordinates, camera coordinates, and image coordinates. The three-dimensional real space coordinates of the human candidate detected by the laser range sensor 11 become world coordinates with the position of the laser range sensor 11 as the origin. Further, using a pinhole camera by perspective projection as a model, the optical center of the pinhole camera is taken as the coordinate origin Oc, and the X axis and Y axis of the three-dimensional coordinate are respectively the u axis and v on the image plane on which the image sensor is placed. The coordinates defined so as to be parallel to the axis and the Z axis parallel to the optical axis are called camera coordinates. Furthermore, there are image coordinates defined in units of pixels that are determined by where the origin of the image data is taken or how the fineness and aspect ratio of the pixels are taken regardless of the physical size and position.

ここに、ｒ１，１・・・ｒ３，３は回転を表すパラメーターであり、ｔｘ、ｔｙ、ｔｚは平行移動を表すパラメーターである。 A camera external parameter that indicates the positional relationship of the camera coordinates determined from the position and orientation of the camera with respect to the world coordinates by rotation and translation is the camera external parameter, and the camera coordinates (Xc, Yc, Zc) from the world coordinates (Xw, Yw, Zw). Conversion to) is as follows.

Here, r1, 1... R3, 3 are parameters representing rotation, and tx, ty, tz are parameters representing translation.

ここに、ｆはカメラの焦点距離を表し、δｕとδｖはそれぞれ横方向と縦方向の画素の物理的な間隔を表し、ｃｕ、ｃｖは画像座標系における光軸と画像面との交点位置を表し、Ｓは定数を表す。 The conversion from camera coordinates (Xc, Yc, Zc) to image coordinates (u, v) in units of pixels using the camera internal parameters is as follows.

Here, f represents the focal length of the camera, δu and δv represent the physical spacing between the horizontal and vertical pixels, respectively, and cu and cv represent the intersection position of the optical axis and the image plane in the image coordinate system. And S represents a constant.

  As a specific example, for an xy position coordinate (Xw, Yw) of a human candidate detected by the laser range sensor 11 at a height of 80 cm from the floor assuming the waist height of the person, for example, the average human head height is 145 cm. In this case, the viewport conversion unit 31 projects and converts the three-dimensional real space coordinates (Xw, Yw, 145) into the two-dimensional image coordinates (u, v).

  FIG. 4 is an example of a step of changing the size of the upper body detection window when the upper body image is a square. In this example, the size of the upper body detection window is sequentially reduced in 16 steps from the size of 480 pixels × 480 pixels to the size of 48 pixels × 48 pixels, and the input for each detection window size. The upper body of various sizes is detected by repeatedly identifying the upper body while scanning the detection window in the image.

FIG. 5 is an example of an image when a plurality of upper body identification windows are integrated into one upper body detection window. It is an example of an image when the upper body determination unit 42 determines a person's upper body using a plurality of window sizes. When a person's upper body exists in the input image, a plurality of upper body identification windows are identified as shown in the upper image example of FIG. Of these upper body identification windows, the upper body identification window whose upper body is within a predetermined range is integrated into one upper body detection window. The integrated example is the lower image example of FIG.

  FIG. 6 shows a human candidate detection tracking unit 20 to which a signal from the laser range sensor 11 is input, a human tracking / attribute integration unit 30, and a human attribute estimation unit 40 to which a video from the camera 10 is input. It is an example of the data flow between. When there is a human candidate object in the detection area of the laser range sensor 11, the human candidate detection tracking unit 20 to which the laser measurement result from the laser range sensor 11 is input detects the object and determines whether it is a human candidate. . When the person candidate is determined, the person candidate tracking unit 24 of the person candidate detection and tracking unit 20 inputs the person ID and world coordinates (x, y) of each person candidate to the person tracking / attribute integrating unit 30.

  The person tracking / attribute integrating unit 30 holds the information of the person candidate ID and world coordinates (x, y) of the person candidate input from the person candidate detecting / tracking unit 20. Then, the viewport conversion unit 31 performs projection conversion to two-dimensional image coordinates based on the world coordinates (x, y). The upper body search area setting unit 32 sets an upper body search scale range and an upper body search area based on the projection-transformed coordinates. In this way, the person candidate ID, the upper body search scale range, and the upper body search area are output from the person tracking / attribute integration unit 30 to the person attribute estimation unit 40.

  In the person attribute estimation unit 40, based on the frame image from the camera 10, the upper body identification unit 41 detects the upper body of the person, determines the upper body position and range by the window detected by the upper body determination unit 42, and the face detection attribute estimation unit At 43, face detection and attribute estimation are performed, and the person ID and person attribute information are output to the person tracking / attribute integrating unit 30.

  The person tracking attribute determination unit 33 outputs the world coordinates (x, y) and the person attribute information in association with each other by matching the held person candidate ID with the person ID.

  Next, processing in the person tracking attribute determination unit 33 will be described in detail. The person attribute determination in the sequence image is performed using the person ID from the laser range sensor.

  FIG. 7 is an example of a determination process flow in the person tracking attribute determination unit 33. The person tracking attribute determination unit 33 receives the person ID and the attribute information of the person from the face detection attribute estimation unit 43. For example, a gender identification value and an age estimation value corresponding to each person ID are input to the person tracking attribute determination unit 33 as person attribute information.

  When a plurality of persons are shown in the camera image, a plurality of person IDs, gender identification values and age estimation values corresponding to the respective person IDs are input to the person tracking attribute determination unit 33. The person tracking attribute determination unit 33 performs processing for each person ID in each frame image. First, it is determined whether gender identification and age estimation corresponding to the person ID is performed in the current processed image (step 71). This is because there are cases where gender identification and age estimation are not performed, such as when a person is detected by a laser range sensor, but the face is hidden in the image. When gender identification and age estimation are performed, the gender identification value and age estimation value corresponding to the person ID are input. When gender identification and age estimation are not performed, “NA” is input. Has been.

人物追跡属性判定部３３は、人物属性パラメーター・テーブルとしてそれぞれの人物ＩＤごとに、判定回数、平均性別識別値、平均年齢推定値を保存している。 If gender identification and age estimation are performed in the current processed image, an average gender identification value update process is performed (step 72). The calculation of the average gender identification value update process is as follows.

The person tracking attribute determination unit 33 stores the number of determinations, the average gender identification value, and the average age estimated value for each person ID as a person attribute parameter table.

  The gender identification value is a value obtained as a result of gender identification by the support vector machine, and represents a distance from the gender identification curved surface. For example, when the gender identification value is a positive value, it is identified as a male, and when it is negative, it is identified as a female. As the absolute value of the gender identification value is larger, it indicates that the processed face image is farther from the identification curved surface for determining gender, and represents the probability of identification. Although the identification value of the support vector machine is used here, the likelihood and probability in another statistical identification method can be used instead.

そして、人物属性パラメーター・テーブルの判定回数を１増加する（ステップ７４）。その後、人物追跡属性判定結果を出力する（ステップ７５）。 Next, an average age estimated value update process is performed (step 73). Like the average gender identification value updating process updates the average age estimate by calculation below.

Then, increase by 1 the number of determinations of the person attribute parameter table (step 74). Thereafter, the person tracking attribute determination result is output (step 75).

  FIG. 8 is a specific flow of the determination method in the person tracking attribute determination unit 33. As shown in FIG. 8, when the attribute estimation of the person ID has not been performed yet, the parameter corresponding to the person ID in the person attribute parameter table has the determination number = 0, the average gender identification value = NA. Average age estimated value = NA.

  When the person ID, the gender identification value, and the age estimation value for each person of the current processed image (processed image number = 1) are input from the face detection attribute estimation unit 43, the average gender is calculated by the average gender identification value update process. An identification update value is calculated. In the case of the person ID = 1, the face detection attribute estimation unit 43 sets the person ID of the currently processed image in the state where the determination count corresponding to the person ID = 1 in the person attribute parameter table = 0 and the average gender identification value = 0. 1 is input, the average gender identification update value is 1.26, and the person attribute parameter table is updated to the average gender identification value of person ID = 1 = 1.26. Is done.

  In this example, the value of the result of gender identification by the support vector machine is shown. When the gender identification value is positive, it is identified as male, and when it is negative, it is identified as female. Is shown.

  Next, an average age estimated value update process is performed, and an average age estimated value update process is performed in the same manner as the average gender identification value update process to update the average age estimated value. In the case of the person ID = 1, the average age estimated update value is 24.84 by inputting the age estimated value 24.84 corresponding to the person ID = 1 of the currently processed image from the face detection attribute estimating unit 43. The person attribute parameter table is updated to the average age estimate of person ID = 1 = 24.84.

  Then, the number of determinations in the person attribute parameter table is increased by 1, and the number of determinations in the person attribute parameter table is set to 1. Thereafter, the person tracking attribute determination result is output. As an output of the determination result, if the person ID = 1 is taken as an example, since the average gender identification value corresponding to the person ID = 1 in the person attribute parameter table at this time = + 1.26, it is output as male and the average age is output. Since the estimated value is 24.84, 25 years old is output.

の場合、終了時の人物属性パラメーター・テーブルは、

となり、判定結果として

を出力する。 The person attribute determination process flow shown in FIG. 7 is performed for each person ID, and the processes performed so far for person ID = 1 in FIG. 8 are similarly performed for all person IDs in the current process image. Is called. That is, the same processing is performed for the person ID = 2 and the person ID = 3, and the person tracking attribute determination in the current processed image is finished. Person ID attribute information of person ID = 2 and person ID = 3 of the currently processed image from the face detection attribute estimation unit 43 is obtained.

, The person attribute parameter table at the end is

And as a judgment result

Is output.

の場合、終了時の人物属性パラメーター・テーブルは、

となり、判定結果として

を出力する。 FIG. 9 is a flow of the determination method in the next processed image (processed image number = 2). The person tracking attribute determination process is performed as shown in FIG. For example, the person ID attribute information from the face detection attribute estimation unit 43 is

, The person attribute parameter table at the end is

And as a judgment result

Is output.

の場合、終了時の人物属性パラメーター・テーブルは、

となり、判定結果として

を出力する。 FIG. 10 is a flowchart of the determination method for the next processed image (processed image number = 3). The person tracking attribute determination process is performed in the same manner as in FIGS. The person ID attribute information from the face detection attribute estimation unit 43 is

, The person attribute parameter table at the end is

And as a judgment result

Is output.

  In the person tracking attribute determination process with the processed image number = 3, the face detection attribute estimation unit 43 did not detect the face for the person ID = 1 and did not perform attribute estimation, so gender identification value = NA, age estimation value = An example in which NA is input to the person tracking attribute determination unit 33 is shown. In the determination process of step 71 of FIG. 7, since the person attribute is not estimated in the face detection attribute estimation unit 43, the process of steps 72 to 75 is not performed. As a result, person ID = 1 in the person attribute parameter table The number of determinations, the average gender identification value, and the average age estimate corresponding to are not updated. However, the previous value is retained, and as a result, both gender determination and age determination can be output. In this way, even when a scene with a hidden face occurs in the camera video sequence, the correct determination can be made by using the person attribute determination result of the person ID tracked in the camera video sequence so far. The result can be output.

  Furthermore, in the person tracking attribute determination process of process image number = 3, the correct gender identification is performed in the face detection attribute estimation unit 43 due to the influence of the variation in the illumination condition of the face image, partial hiding, etc. with respect to the person ID = 2. An example of the case where there was no such case is shown. In this processed image (processed image number = 3), the face detection attribute estimation unit 43 has the gender identification value = + 0.18 for the person ID = 2, and is identified as being slightly male. However, the gender identification value of the person ID = 2 in the camera video sequence is comprehensively determined including the gender identification value of the same person ID in the past processed image by performing the average gender identification value update process. As a result, the average gender identification value in this processed image = −1.18, and it can be determined correctly as a woman.

  FIG. 11 is a flowchart of the determination method when the processing image number = 4. FIG. 11 shows an example in which the face detection attribute estimation unit 43 does not perform the attribute estimation of the person ID = 3. In this case as well, as a result of the determination at step 71 in FIG. 7, the processing at steps 72 to 75 is not performed, and the determination count corresponding to the person ID = 3 in the person attribute parameter table, average gender identification value, average age estimation The value has not been updated. However, since the previous value is retained, sex determination and age determination are correctly performed.

となる。図１０に示される平均性別識別値更新処理５２と平均年齢推定値更新処理５３の演算により、人物属性パラメーター・テーブルが更新されるが、人物ＩＤ＝１は顔検出属性推定部４３からの人物ＩＤ属性情報が無いため更新は行なわれず、以前の値が保持される。また、人物ＩＤ＝４はこの処理画像で初めて人物ＩＤ属性情報が入力されたので、新しく人物属性パラメーター・テーブルに加わり、

となり、判定結果として

を出力する。 FIG. 12 is a flowchart of the determination method when the processing image number = 5. In the laser detection sensor person detection, person ID = 1 disappears from the detection area, and a new person ID = 4 is added to the detection area. Corresponding to the person detection of the laser range sensor, the person ID attribute information from the face detection attribute estimation unit 43 is also for three persons of person ID = 2, person ID = 3, and person ID = 4.

It becomes. The person attribute parameter table is updated by the calculation of the average gender identification value update process 52 and the average age estimated value update process 53 shown in FIG. 10, but person ID = 1 is the person ID from the face detection attribute estimation unit 43. Since there is no attribute information, no update is performed and the previous value is retained. In addition, since the person ID attribute information is input for the first time in this processed image, the person ID = 4 is newly added to the person attribute parameter table,

And as a judgment result

Is output.

  For the person ID = 3, the gender estimated value from the face detection attribute estimation unit 43 is −0 due to the influence of the illumination condition of the face image, partial hiding, etc., as in the case of the person ID = 2 of the processing image number = 3. Although it is slightly misidentified as female, such as 04, the gender identification value = + 0.61 is obtained by the person tracking attribute determination method of this method, and it is correctly determined as male. Although an example in which the average value is used for the determination has been described here, a median or a majority vote or the like can be used for the gender determination instead of the average value.

  Next, an example of output from the person tracking attribute determination unit 33 will be described. As described above, the person tracking attribute determination unit 33 outputs person attribute information such as a person ID, person tracking world coordinates (x, y), and age / sex. By outputting this to a display (not shown), it is possible to plot it together with the person ID on the two-dimensional bird's-eye view of the x-axis and y-axis. In addition, the display on the bird's eye view is updated corresponding to the person ID and world coordinates of the person candidate input from the person candidate tracking unit 24 sequentially.

  FIG. 13 is an example of a screen output from the person tracking attribute determination unit 33. The attribute information (gender identification value, age estimation value) associated with the person ID is input to the person tracking / attribute integration unit 30 for each processed image from the person attribute estimation unit 40 to which the camera 10 is connected. The number of determinations, the average gender identification value, and the average age estimated value corresponding to each person ID are sequentially calculated and stored as the aforementioned person attribute parameter table. For the person ID based on the data from the laser range sensor 11 and the person tracking world coordinates (x, y), gender determination and age determination from the average gender identification value and the average age estimated value corresponding to each person ID, By performing the additional display, it is possible to generate a bird's eye view in which person tracking information and attribute information are displayed in an integrated manner.

  Further, as shown in FIG. 13, the upper body image determined by the upper body determination unit 42 may be displayed as a thumbnail image corresponding to the person ID in accordance with the plotted position. In this case, the person tracking attribute determination unit 33 receives the upper body detection image from the face detection attribute estimation unit 43 in a form associated with the person ID. This makes it possible to create and display a person tracking / attribute integrated bird's-eye view that is easier to understand as shown in FIG. For the generation of thumbnail images, for example, an image having the largest face detection identification value is selected from the upper body detection images of the same person ID in the sequence image from the camera video output from the face detection attribute estimation unit 43. By updating and saving, the upper body image showing the most front face is used as the thumbnail image. In this way, it is possible to increase the ease of understanding the bird's-eye view display. The person tracking position information from the laser range sensor and the person attribute information from the camera image can be integrated to generate a person tracking / attribute integrated bird's-eye view, and the action tracking with attribute information and a person photograph is very useful as a bird's eye view. It can be displayed clearly.

  Next, an example of collecting behaviors of people who have come out of an elevator in an office store complex will be described below as a method of collecting behaviors of customers using the person tracking attribute estimation device.

  FIG. 14 is an example of the arrangement of the camera 10 and the laser range sensor 11. The camera 10 and the laser range sensor 11 are installed so that the face of the person who has left the elevator can be photographed and the area that has left the elevator becomes a person attribute tracking area. By doing in this way, attribute determination information can be linked | related with the position tracking of the person who came out of the elevator.

  As shown in FIG. 14, one side of the shop / restaurant area in the building in the person attribute tracking area is A side, and one side of the building entrance / exit direction is B side, and the setting side for tracking the person attribute is determined. By tracking the position of the person attribute information, the number of people who got out of the elevator from the side A of the person attribute tracking area and the number of people who came out of the B side for each time zone To sum up.

FIG. 15 is a diagram illustrating a person entering / exiting determination method. How the person crosses the A side or the B side is determined by tracking the human candidate by the laser range sensor 11. The human candidate tracking unit 24 outputs the world coordinates (x, y) together with the person ID at the timing (t = n) for each scanning frequency of the laser range sensor. The person tracking attribute determination unit 33 uses the real space coordinates (x _n−1 , y _n−1 ) of the same person ID and the previous timing (t = _n−1 ) and (x _n , y _n ) of the current timing. , The motion vector of the human candidate on the real space coordinates is calculated. Then, by determining in which direction the motion vector crosses the A side or B side, it is determined whether the set A side or B side has exited the elevator or entered the elevator. Can do.

  Due to the specifications of the laser range sensor 11, when the scanning frequency is very fast relative to the movement of the person candidate, the person ID and the real space coordinates output at the timing of the scanning frequency are thinned out to, for example, 1/5. By performing the same processing, it is possible to stably perform the crossing judgment on the set A side or B side.

  One laser range sensor can track a human candidate on real space coordinates. As shown in FIG. 14, a plurality of sides are set for setting a candidate for entering / exiting a human candidate with one laser range sensor. be able to.

  Figure 16 shows an example of the results of counting the number of men who left the elevators by age group and headed for shops and restaurants and the number of people headed for entrances and exits for each time period from 7 am to 10 pm FIG. Figure 17 shows an example of the results of counting the number of women who left the elevators by age and headed for shops and restaurants and the number of people headed for entrances and exits for each time period from 7 am to 10 pm FIG. In these examples, men and women are classified into four ages of under 20 years old, 20 years old and under 40 years old, 40 years old and under 60 years old, and 60 years old and over. As described above, the attribute information for each determined person ID is accumulated, and the person tracking attribute determination unit 33 aggregates the information so that the gender ratio and the age ratio of each time zone of the entire person who has exited the elevator Can be analyzed. FIG. 18 is a diagram of an example in which the number of people and the age ratio are tabulated for each person who comes out of the elevator.

  Furthermore, after exiting the elevator, it is possible to investigate a trend for each person attribute as a behavioral survey as to whether it headed for the shop & restaurant or headed for the entrance / exit. FIG. 19 is an example of a diagram in which the number of persons and the ratio of the person after exiting the elevator for each time zone in a male who is 40 years old or older and younger than 60 years old are obtained. FIG. 20 is an example of a diagram in which the number and ratio of the destinations after exiting an elevator focused on women aged between 20 and 40 years old. By doing this analysis, men between the ages of 40 and under 60 turn their feet to the shops and restaurants only after lunch breaks and regular hours, whereas women between the ages of 20 and under 40 years of time from lunch breaks to after regular hours. As a result, it can be seen that, for example, the time from lunch break to after regular hours can be expected to increase sales by carrying out a sales strategy targeting women. .

  In this way, it is possible to collect behaviors associated with person attributes. For example, it is possible to collect store customers' buying behaviors, and by knowing the flow in the store of the store customers that cannot be seen from the POS data. Product layout and floor layout can be optimized. It is also possible to strategically examine store products and restaurant menus in office store complex buildings by grasping the flow of people coming out of elevators in the building by time zone. .

DESCRIPTION OF SYMBOLS 10 Camera 11 Laser range sensor 12 Person tracking attribute estimation apparatus 20 Person candidate detection tracking part 21 Polar coordinate / plane coordinate conversion part 22 Background registration part 23 Person candidate clustering part 24 Person candidate tracking part 30 Person tracking and attribute integration part 31 Viewport Conversion unit 32 Upper body search area setting unit 33 Person tracking attribute determination unit 40 Person attribute estimation unit 41 Upper body identification unit 42 Upper body determination unit 43 Face detection attribute estimation unit

Claims (9)

A human candidate clustering unit that extracts human candidates by clustering point clouds that are distances within a predetermined threshold from the measurement data of the laser range sensor;
Comparing the candidate human image between scan frames, the distance difference grant candidate human image ID person candidate is within a predetermined range as the same person candidate, the position coordinates of the same person candidate and the candidate human ID A candidate candidate tracking unit to output;
A viewport converter for converting the position coordinates into two-dimensional image coordinates;
An upper body search area setting unit that sets a window size and coordinate data as an upper body search area corresponding to the person candidate assigned with the person candidate ID based on the two-dimensional image coordinates;
Based on the person candidate ID, the window size, and the coordinate data from the upper body search area setting unit, a plurality of the same body candidate ID upper body search areas within the setting range are integrated into one to obtain a person ID. and a detection window, have rows face detection of a person in the detection window in the frame image data from the camera, further estimates the People attribute of performing the face detection, and outputs together with the person ID corresponding the person attribute A person attribute estimation unit;
And the person candidate ID and position coordinates output from the candidate human image tracking unit, on the basis of said personal ID and person attribute information from the person attribute estimation unit, the position and the person ID that matches the person candidate ID person tracking attribute estimation apparatus comprising: the person tracking attribute determination unit which performs mapping between the coordinate and the person attribute information. The person tracking attribute determination unit determines a person attribute of the person ID based on person attribute information to which the person ID of the same person is assigned in a plurality of frame image data. The described person tracking attribute estimation device. The person tracking attribute determining unit, an average value of a plurality of the person attribute information, the median, the person tracking attribute of claim 2, wherein determining the person attribute of the person ID based on any of the majority Estimating device. The person tracking attribute determining unit further receives the thumbnail image from the person attribute estimation unit, based on the position coordinates corresponding to the person ID, and displayed on the display device by plotting said thumbnail image and the person attribute information The person tracking attribute estimation apparatus according to any one of claims 1 to 3 , wherein the person tracking attribute estimation apparatus outputs the following information. The person tracking attribute determining unit calculates a motion vector from the position coordinates corresponding to the person ID, a person of any one of claims 1 to 4, characterized in that to determine the movement of the setting side Tracking attribute estimation device. The person tracking attribute determining unit, person tracking attribute estimating apparatus of any one of claims 1 to 4, characterized in that aggregating the person attribute information of the plurality of the person ID. The person tracking attribute determining unit, person tracking attribute estimating apparatus according to claim 5, wherein the aggregate combined motion for the plurality of the person ID person attribute information and the setting side of the. A human candidate clustering step of extracting human candidates by clustering point clouds that are distances within a predetermined threshold from the measurement data of the laser range sensor,
The person candidates are compared between scanning frames, a person candidate whose distance difference is within a predetermined range is assigned as a person candidate ID, and the person candidate ID and the position coordinates of the same person candidate are obtained. Human candidate tracking step to output,
A viewport conversion step of converting the position coordinates into two-dimensional image coordinates;
An upper body search area setting step for setting a window size and coordinate data as an upper body search area corresponding to the person candidate assigned with the person candidate ID based on the two-dimensional image coordinates;
Based on the person candidate ID, the window size, and the coordinate data set in the upper body search area setting step, a plurality of the same person candidate ID upper body search areas within the setting range are integrated into one person. As the ID detection window, the face of the person in the detection window in the frame image data from the camera is detected, the person attribute of the person who has detected the face is estimated, and the person attribute is output together with the corresponding person ID Person attribute estimation step,
Based on the person candidate ID and position coordinates output in the person candidate tracking step and the person ID and person attribute information output in the person attribute estimation step, the person ID that matches the person candidate ID and A person tracking attribute estimation method comprising: a person tracking attribute determination step for associating the position coordinates with the person attribute information. On the computer,
A human candidate clustering step of extracting human candidates by clustering point clouds that are distances within a predetermined threshold from the measurement data of the laser range sensor,
The person candidates are compared between scanning frames, a person candidate whose distance difference is within a predetermined range is assigned as a person candidate ID, and the person candidate ID and the position coordinates of the same person candidate are obtained. Human candidate tracking step to output,
A viewport conversion step of converting the position coordinates into two-dimensional image coordinates;
An upper body search area setting step for setting a window size and coordinate data as an upper body search area corresponding to the person candidate assigned with the person candidate ID based on the two-dimensional image coordinates;
Based on the person candidate ID, the window size, and the coordinate data set in the upper body search area setting step, a plurality of the same person candidate ID upper body search areas within the setting range are integrated into one person. As the ID detection window, the face of the person in the detection window in the frame image data from the camera is detected, the person attribute of the person who has detected the face is estimated, and the person attribute is output together with the corresponding person ID Person attribute estimation step,
Based on the person candidate ID and position coordinates output in the person candidate tracking step and the person ID and person attribute information output in the person attribute estimation step, the person ID that matches the person candidate ID and A person tracking attribute determination step for associating the position coordinates with the person attribute information;
A person tracking attribute estimation program characterized in that is executed.

Images