JP2013156718A - Person tracking attribute estimation device, person tracking attribute estimation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a person tracking attribute estimation device capable of enhancing detection accuracy by linking an attribute estimation with a position detection track with a fewer calculation amount.SOLUTION: The person tracking attribute estimation device has a person candidate clustering section that performs clustering dot groups which are positioned within a distance of a predetermined threshold value to extract a person candidate from measurement data obtained by a laser measurement range sensor. A person candidate tracking section compares the measurement data between scanning frames and gives a person ID to a dot group the distance difference of which is within a predetermined range as the same person candidate and outputs a person ID and coordinates of the position. A viewport conversion section converts the coordinates of the position to two-dimensional image coordinates. An upper-body image search area setting section sets a window size and coordinate data as an upper-body image search area corresponding to the person ID on the basis of the two dimensional image coordinates. A face detection attribute estimation section detects a face from an image data of a camera on the basis of the upper-body image search area to estimate the person attribute. A person tracking attribute determination section links the person ID to the position coordinates and the person attribute information.

Description

  The present invention relates to a method and apparatus for detecting and tracking a person's position, and by adding person recognition by a camera, it is possible to link person attribute information (gender, age, etc.) to person tracking. is there.

  A system for detecting and tracking a person in a three-dimensional real space with a radar range sensor has been put into practical use. A conventional person position detection / tracking apparatus detects an object position by laser irradiation and determines a person from the size of the detected object.

In addition, as a technique in which a laser range sensor and a camera are integrated, a technique described in Document 1 below has been proposed. This is the weighted sum of the probability density distribution that complementarily integrates the position of the human head in the three-dimensional real space from the distance observation by the laser range sensor and the head shape from the observation by the camera. By using it, the accuracy and robustness of person tracking is improved.
Takanori Kobayashi, et al. “Person Tracking in Areas by Integration of Distributed Camera and Laser Range Sensor” Image Recognition and Understanding Symposium (MIRU2007) July 2007

  The human position detection tracking of the conventional laser range sensor is based on the delay time (light flight time) of the reflected light with respect to the projection on the circumference due to the rotation of the internal mirror of the laser beam by the laser range sensor. By measuring the distance and direction to the object and the size of the object, the position of the person estimated from the size of the object is detected and tracked, and the person attribute cannot be estimated. In addition, the human tracking that integrates the conventional laser range sensor and camera is a method that complementarily integrates different sensor modalities of distance observation by the laser range sensor and appearance observation by the camera. The purpose was to improve the accuracy and robustness of person tracking, and person attribute information could not be linked to person tracking.

  The present invention relates to a person tracking attribute estimation apparatus that associates attribute estimation with person position detection tracking, reduces the amount of calculation, and improves detection accuracy.

  The present invention compares a human candidate clustering unit that extracts point candidates by clustering a point group that is a distance within a predetermined threshold from measurement data of a laser range sensor, and compares the measurement data between scan frames, and the distance difference is A person candidate tracking unit that assigns a person ID to a point group within a predetermined range as the same person candidate, and outputs the person ID and the position coordinates; a viewport conversion unit that converts the position coordinates into two-dimensional image coordinates; Based on the two-dimensional image coordinates, an upper body search area setting unit that sets a window size and coordinate data as an upper body search area corresponding to the person ID, and from frame image data from the camera, based on the upper body search area From a face detection attribute estimation unit that performs face detection and estimates a person attribute, a person ID and position coordinates output from the person candidate tracking unit, and a face detection attribute estimation unit Based on the personal ID and the person attribute information, to provide the person ID and position coordinates, and the person tracking attribute determination unit that performs mapping of the person attribute information, the person tracking attribute estimation apparatus having a.

  In addition, the person attribute tracking determination unit of the person tracking attribute estimation apparatus of the present invention may determine the person attribute of the person ID based on the person attribute information to which the same person ID is assigned in a plurality of frame image data. Good.

  In the person tracking attribute estimation apparatus, the person attribute tracking determination unit may determine the person attribute of the person ID based on any of an average value, a median value, and a majority vote of a plurality of person attribute information.

  In the person tracking attribute estimation device, the person attribute tracking determination unit further receives a thumbnail image from the face detection attribute estimation unit, and determines the thumbnail image and the person attribute information based on the position coordinates corresponding to the person ID. You may make it output so that it may plot and display on a display apparatus.

  In the person tracking attribute estimation apparatus, the person attribute tracking determination unit may calculate a motion vector from the position coordinates corresponding to the person ID, and determine a movement with respect to the set side.

  In the person tracking attribute estimation device, the person attribute tracking determination unit may total person attribute information of a plurality of person IDs.

  In the person tracking attribute estimation apparatus, the person attribute tracking determination unit may total the person attribute information of a plurality of person IDs and movements with respect to the set side.

  The present invention also provides a human candidate clustering step for clustering a point group that is a distance within a predetermined threshold from measurement data of a laser range sensor to extract a human candidate, comparing measurement data between scan frames, Is a person candidate tracking step of assigning a person ID to a point group having a predetermined range as the same person candidate, and outputting the person ID and the position coordinates, a viewport conversion step of converting the position coordinates into two-dimensional image coordinates, Upper body search area setting step for setting window size and coordinate data as an upper body search area corresponding to the person ID based on the two-dimensional image coordinates, face detection based on the upper body search area from frame image data from the camera And the person ID and position output in the face detection attribute estimation step and the person candidate tracking step for estimating the person attribute. A person tracking attribute estimation method comprising: a person tracking attribute determination step for associating a person ID with position coordinates and person attribute information based on the coordinates and the person ID and person attribute information output in the face detection attribute estimation step I will provide a.

  Further, the present invention provides a human candidate clustering step of clustering a point group having a distance within a predetermined threshold from measurement data of a laser range sensor to extract a human candidate on a computer, and comparing the measurement data between scanning frames. , A person candidate tracking step of assigning a person ID to a point group having a distance difference within a predetermined range as the same person candidate, and outputting the person ID and position coordinates, and a viewport for converting the position coordinates into two-dimensional image coordinates Conversion step: Upper body search area setting step for setting window size and coordinate data as an upper body search area corresponding to the person ID based on the two-dimensional image coordinates, from frame image data from the camera, based on the upper body search area Output in face detection attribute estimation step and person candidate tracking step to perform face detection and estimate person attributes Based on the object ID, the position coordinates, and the person ID and person attribute information output in the face detection attribute estimation step, a person tracking attribute determination step for associating the person ID with the position coordinates and the person attribute information is executed. A person tracking attribute estimation program is provided.

  According to the present invention, the person tracking / attribute integration unit 30 is provided, so that a three-dimensional real space position tracking function of an object by a laser range sensor and a face detection attribute estimation function by a camera are respectively supported for a plurality of tracking persons. And can be tied. In addition, the upper body identification unit 41 can identify the upper body by using the data from the upper body search area setting unit 32, thereby reducing the number of scans. Furthermore, it is possible to realize 3D real space person tracking with person attribute information.

  In addition, it is possible to make a comprehensive judgment of person attributes in a sequence image using a person ID from a laser range sensor, and in a state where a change in lighting conditions, a change in face orientation, a face hiding, etc. have occurred In addition, stable and highly accurate person attribute determination can be performed.

  Furthermore, it is possible to generate a person tracking / attribute integrated bird's-eye view that integrates the person tracking position information from the laser range sensor and the person attribute information from the camera image, and the action tracking with attribute information and a person photograph can be used as a bird's eye view. It can be displayed very clearly.

  In addition, it is possible to collect behaviors linked to person attributes. For example, it is possible to collect store customers' buying behavior, and by knowing the flow in the store of the store customers that cannot be seen from the POS data, Placement and floor layout can be optimized. It is also possible to strategically examine store products and restaurant menus in office store complex buildings by grasping the flow of people coming out of elevators in the building by time zone. .

It is an example of the installation arrangement | positioning of the person tracking attribute system comprised from the camera concerning this invention, a laser range sensor, and a person tracking attribute estimation apparatus. It is a block diagram of a person tracking attribute presumption system in one embodiment of the present invention. It is a figure showing the relation of world coordinates, camera coordinates, and image coordinates. It is an example of the change step of the size of the upper body detection window when the upper body image is a square. It is an example of an image when a plurality of upper body identification windows are integrated into one upper body detection window. Data flow between a human candidate detection tracking unit 20 to which a signal from the laser range sensor 11 is input, a human tracking / attribute integration unit 30, and a human attribute estimation unit 40 to which an image from the camera 10 is input It is an example. It is an example of the determination processing flow in the person tracking attribute determination part 33. It is a specific flow of the determination method in the person tracking attribute determination part 33. FIG. It is a flow of the determination method in the next processed image (processed image number = 2). Furthermore, it is a flow of the determination method in the next processed image (processed image number = 3). It is the flow of the determination method in process image number = 4. It is a flow of the determination method in process image number = 5. It is an example of the screen output from the person tracking attribute determination part 33. It is an example of arrangement | positioning of the camera 10 and the laser range sensor 11. FIG. It is a figure which shows the person's entrance / exit determination method. It is a figure showing an example of the result of totaling the number of people who went to the shop & restaurant direction and the number of people who went to the entrance / exit direction after exiting the elevator by men's age group for each time zone from 7 am to 10 pm is there. It is a figure showing an example of the result of totaling the number of people who went to the shop & restaurant direction and the number of people who went to the entrance / exit direction by going out the elevators by women's age group for each time period from 7 am to 10 pm is there. It is a figure of an example which totaled the number of persons and the age ratio for every person who came out of the elevator. It is an example of the figure which calculated | required the number of persons and the ratio of the other side after leaving the elevator for every time slot | zone in the man of 40 years old or older and less than 60 years old. It is an example of the figure which calculated | required the number of persons and the ratio of the other side after leaving the elevator restrict | squeezed to the woman 20 years old or older and less than 40 years old.

  An embodiment in which the present invention is applied to visitor tracking and attribute estimation in an arbitrary area in a facility will be described below. In this embodiment, the laser range sensor is installed so that the laser emits light in a horizontal circumferential direction, and the camera is installed at a position where a person's face in the area is captured in an appropriate size. Real-time 3D coordinate detection and tracking of human candidates in the area is performed using a laser range sensor. Also, with the camera, the person detection and person attribute estimation corresponding to each person candidate from the laser range sensor is performed from the image position obtained by projecting the real space 3D coordinates of the person candidate from the laser range sensor to the camera image coordinates. In addition, the result is comprehensively determined by the average value, median value, majority vote, etc. using the tracking information from the laser range sensor, so that highly accurate and stable person attribute information can be linked to person tracking.

  FIG. 1 is an example of an installation arrangement of a person tracking attribute system including a camera, a laser range sensor, and a person tracking attribute estimation device according to the present invention. The camera 10 and the laser range sensor 11 are respectively installed so as to be fixed. The camera 10 is a video camera, and is arranged and arranged so that a person's face in the person detection / tracking area is captured with an appropriate size. For example, the laser range sensor is arranged so as to project a laser beam horizontally at a height of 80 cm from the floor surface assuming the waist height of a person. The camera and the laser range sensor are connected to the person tracking attribute estimation device 12 through interfaces such as USB, LAN, and RS-232C.

  The laser range sensor 11 irradiates a laser, and measures the distance to the object from the time it takes for the object to reflect and return. By reflecting the laser emitted from the transmitter with a rotating mirror and scanning the front in a fan shape, the distance from the floor surface to an object existing on a flat surface can be measured. Since this system is equipped with the laser range sensor 11 as shown in FIG. 1, the plane is scanned with a height of 80 cm from the floor parallel to the floor.

  The rotating mirror of the laser range sensor 11 uses a step angle, and the scanning direction scans counterclockwise. For example, 1 [step] is about 1.05 [deg], the maximum number of steps is 228 [step], and a fan shape of about 240 degrees is the detection area. The laser range sensor 11 outputs measurement data composed of polar coordinate data on the r-θ plane at an angle θ and a distance r for each point. Measurement data of these detected point groups is input to the person tracking attribute estimation device 12 at the timing of the scanning frequency. Instead of the laser range sensor 11, for example, an acceleration sensor and an azimuth angle sensor may be combined with GPS to acquire angle and coordinate data. In addition, data may be acquired by performing distance measurement using a stereo camera.

  FIG. 2 is a block diagram of a person tracking attribute estimation system in one embodiment of the present invention. The person tracking attribute estimation system includes a camera 10, a laser range sensor 11, and a person tracking attribute estimation device 12. The person tracking attribute estimation device 12 includes a person candidate detection tracking unit 20, a person tracking / attribute integration unit 30, and a person attribute estimation unit 40. In one aspect, the person tracking attribute estimation device 12 is realized by cooperation of hardware resources and software, and is, for example, a computer. Specifically, the various functions of the person tracking attribute estimation device 12 are such that a person tracking attribute estimation program recorded in a recording medium such as a ROM or a hard disk is read into a main memory and a control device (for example, a processor such as a CPU). It is realized by being executed by. However, the person tracking attribute estimation device 12 may be realized only by hardware. Further, the person tracking attribute estimation device 12 may be physically realized by one device or may be realized by a plurality of devices. In addition, the person tracking attribute estimation device 12 includes an input interface (not shown) for inputting image frames taken by a camera and data output from a laser range sensor.

  The person candidate detection tracking unit 20 of the person tracking attribute estimation device 12 includes a polar coordinate / planar coordinate conversion unit 21, a background registration unit 22, a person candidate clustering unit 23, and a person candidate tracking unit 24.

  The polar coordinate / planar coordinate conversion unit 21 converts the measurement data having polar coordinate data of the r-θ plane of the angle θ and the distance r input from the laser range sensor 11 at the timing for each scanning frequency into the plane coordinates of the xy plane. Convert to data. Thereby, it can display as a bird's-eye view sketch.

  The background registration unit 22 registers in advance the measurement result of the laser range sensor as a background when there is no person in the person detection tracking area. Thereby, it is possible to detect an object other than the background by extracting only the point group having a large difference from the background measurement.

  The person candidate clustering unit 23 takes out a difference from the background portion registered in the background registration unit 22 that is the point group output from the laser range sensor 11, and performs clustering of points having a close distance difference. By clustering, the point cloud can be classified for each object. For example, if a point group with a width of 25 cm or more is a person candidate, the person candidate clustering unit 23 outputs the xy coordinates of the center of the point group as the person candidate coordinates.

  The person candidate tracking unit 24 assigns the same person candidate ID by determining that the person candidates having a close distance difference between the laser scanning frames are the same person, and performs person tracking. As described above, the person candidate detection tracking unit 20 outputs the person candidate ID and the xy position coordinates (x, y) for each detected person candidate to the person tracking / attribute estimation integration unit 30 for each laser scanning frame. To do.

  The person tracking / attribute integration unit 30 includes a viewport conversion unit 31, an upper body search area setting unit 32, and a person tracking attribute determination unit 33. When the xy position coordinates for each detected person candidate ID output from the person candidate detection / tracking unit 20 are input, the viewport conversion unit 31 determines the camera internal parameters and the camera externals obtained in advance by camera calibration. By projective conversion using parameters, the three-dimensional real space coordinates of the head corresponding to the human candidate ID detected by the laser range sensor 11 are converted into image coordinates.

  The upper body search region setting unit 32 takes into consideration the difference from the average height and detection error in the height from the chest to the top of the head, centering on the image coordinates of the human candidate head detected by the laser range sensor 11. Set the upper body search area. Further, the xy position coordinates of the human candidate detected from the laser range sensor 11 with the height of the upper end (top) and the height of the lower end (chest) of the upper body area from the average height as Zw of the three-dimensional space coordinates, respectively. By performing viewport conversion together with (Xw, Yw), the height of the upper body region in the image coordinates can be obtained. Further, from this information, the upper body detection window size range (the upper body size range to be searched) is set in consideration of the difference from the average height and detection error. These person candidate IDs, the corresponding upper body search scale range (window size), and search area (coordinate data) are input to the person attribute estimation unit 40. However, the output timing to the person attribute estimation unit 40 is not the timing of the laser scanning frequency, but the timing at which the person attribute estimation unit 40 ends the person estimation process and the attribute information is input to the person tracking attribute determination unit 33. . That is, the information of the laser scanning frame acquired during the time required for the person estimation process is not used. Regarding the output timing of the information of the person candidate ID, the upper body search area, and the scale range from the person tracking / attribute integrating unit 30 to the person attribute estimating unit 40, the person attribute estimating unit 40 finishes the person attribute estimating process for the person candidate ID. In response, the information regarding the next same person candidate ID is output.

  The person attribute estimation unit 40 includes an upper body identification unit 41, an upper body determination unit 42, and a face detection attribute estimation unit 43. The frame image data acquired by the camera 10 and the person candidate ID input from the upper body search area setting unit 32, Face detection is performed using the upper body search scale range and the search area, and person attributes are estimated.

The upper body identifying unit 41 uses a known image recognition technique to perform multi-frame detection on the frame image from the camera, using a well-known image recognition technique within the upper body search scale range and search region corresponding to the person candidate ID input from the upper body search region setting unit 32. The human upper body is detected on the scale, and the presence / absence of the upper body, the size of the detected upper body, and the image coordinate position are determined. It should be noted that the upper body identification unit 41 receives the frame image of the camera photographed at the same time when the human candidate ID, the upper body search region, and the scale range are input from the upper body search region setting unit 32 to the person attribute estimation unit 40. To detect the upper body of the person. Accordingly, not all camera frame images input to the person attribute estimation unit 40 are necessarily used.
Any known method can be applied as the process for identifying the upper body of the person. The method of the present invention will be described in the case where human upper body detection is performed by a support vector machine classifier using a luminance gradient direction histogram as a feature amount.

  The human upper body detection method is divided into a feature amount learning process and an identification process. In the learning process, a large number of upper body images and non-upper body images that are detection targets are prepared in advance, luminance gradient direction histogram feature amounts are extracted from each image, and upper body shape features are learned using a support vector machine. As a result, a discriminator that determines whether the luminance gradient direction histogram feature value extracted from the input image is the upper body of the person or not is obtained.

  In the identification process, an upper body detection window is set for the input image, and this detection window is sequentially scanned in the input image. While scanning, we extract the luminance gradient direction histogram feature amount of the region cut out by the upper body detection window for each scan, and evaluate the upper body quality of the detection window by the support vector machine classifier obtained by learning in advance Identify the upper body.

  Further, in order to detect the upper body of a person of various sizes in the input image, the size of the upper body detection window is changed, and sequential scanning in the input image is repeated to identify the upper body of each size.

  Based on the upper body detection window size range input from the upper body search area setting unit 32, the upper body identification unit 41 selects, for example, two or three levels of window sizes from the size of the upper body detection window having 16 levels. Scan. Therefore, the number of scans can be reduced by using the window size range input from the upper body search area setting unit 32. Further, by using the image coordinate data of the upper body search area input from the upper body search area setting unit 32, the detection window / scan range in the input image can be changed from the whole image scan to the upper body search area. . By using the three-dimensional spatial coordinates of the detected person candidate from the laser range sensor in this way, the number of scans of the upper body detection window in the input image can be dramatically reduced, and the area is limited to the person candidate area. Since upper body detection is performed, erroneous detection due to a complicated background or the like can be reduced. By these processes, the upper body identification unit 41 outputs the upper body identification result corresponding to each person candidate ID and the image coordinates and size of the identification window when the upper body is identified to the upper body determination unit 42.

  The upper body determination unit 42 uses the image coordinate position and size (window size) of a plurality of upper body identification windows, which are upper body identification results for each person candidate ID input from the upper body identification unit 41, to determine the position of the identification window. If the size is within a preset range, the same person upper body is assumed and a plurality of upper body identification windows corresponding to the same person candidate ID are integrated into one detection window. When the upper body of a person exists in the input image, a plurality of upper body identification windows are usually identified by being superimposed. Among the plurality of upper body identification windows, these are integrated into one identification window whose upper body is within a predetermined range.

  Note that the integration process may be performed by determining that the upper body has been detected only when a predetermined number or more of upper body identification windows are integrated. By doing in this way, false detection can also be reduced. In addition, as an integrated determination result of the upper body identification window, an upper body determination result corresponding to each person candidate ID, and if it is determined to be an upper body, the person candidate ID is set as a person ID, and the image coordinates and size of the upper body detection window are set. It outputs to the face detection attribute estimation part 43.

  When the person ID input from the upper body determination unit 42 and the image coordinates and size of the upper body detection window are input, the face detection attribute estimation unit 43 calculates the face search scale range and the search from the position and size of the upper body detection window. Within a region, face detection in multi-scale is performed using a well-known image recognition technique. When a face is detected with a predetermined size or more, attribute estimation from the face image is performed for each person ID using a well-known image recognition method, and the resulting person ID and person attribute information are determined as upper body The result is output to the person attribute tracking determination unit 33 together with the upper body determination result for each candidate person ID from the unit 42.

  Note that the human attribute estimation unit 40 first detects the upper body and performs face detection attribute estimation on the upper body detection area. However, in the person attribute estimation unit 40, an image from the upper body search area setting unit 32 is used. Based on the upper body search scale range and the search area information in the coordinates, only the face detection and the attribute estimation may be performed without performing the upper body detection.

  The person attribute tracking determination unit 33 finally integrates the tracking result by the laser range sensor 11 and the tracking result with attribute information by the camera 10. The person attribute tracking determination unit 33 holds the xy position coordinates for each detected person candidate ID output from the person candidate detection tracking unit 20. When the person attribute information corresponding to the person ID is input from the face detection attribute estimation unit 43, they are temporarily stored. Also, the upper body search area setting unit 32 is notified that the person attribute information has been input. The person ID input from the face detection attribute estimation unit 43 is compared with the person candidate ID tracked by the laser range sensor 11 and input from the person candidate detection tracking unit 20, and information having the same person ID is matched. Put it on. In addition, with respect to the person attribute information corresponding to the same person ID in different image frame data, the attribute for the same person tracked by taking the average value, median value, majority vote, etc. of the determination result and its parameters for each attribute It eliminates the variation of information, improves accuracy, and outputs the person tracking information linked to the person tracking three-dimensional real space position information. Details of the processing in the person attribute tracking determination unit 33 will be described later.

  FIG. 3 is a diagram showing the relationship between world coordinates, camera coordinates, and image coordinates. The three-dimensional real space coordinates of the human candidate detected by the laser range sensor 11 become world coordinates with the position of the laser range sensor 11 as the origin. Further, using a pinhole camera by perspective projection as a model, the optical center of the pinhole camera is taken as the coordinate origin Oc, and the X axis and Y axis of the three-dimensional coordinate are respectively the u axis and v on the image plane on which the image sensor is placed. The coordinates defined so as to be parallel to the axis and the Z axis parallel to the optical axis are called camera coordinates. Furthermore, there are image coordinates defined in units of pixels that are determined by where the origin of the image data is taken or how the fineness and aspect ratio of the pixels are taken regardless of the physical size and position.

ここに、ｒ１，１・・・ｒ３，３は回転を表すパラメーターであり、ｔｘ、ｔｙ、ｔｚは平行移動を表すパラメーターである。 A camera external parameter that indicates the positional relationship of the camera coordinates determined from the position and orientation of the camera with respect to the world coordinates by rotation and translation is the camera external parameter. Conversion to) is as follows.

Here, r1, 1... R3, 3 are parameters representing rotation, and tx, ty, tz are parameters representing translation.

ここに、ｆはカメラの焦点距離を表し、δｕとδｖはそれぞれ横方向と縦方向の画素の物理的な間隔を表し、ｃｕ、ｃｖは画像座標系における光軸と画像面との交点位置を表し、Ｓは定数を表す。 The conversion from camera coordinates (Xc, Yc, Zc) to image coordinates (u, v) in units of pixels using the camera internal parameters is as follows.

Here, f represents the focal length of the camera, δu and δv represent the physical spacing between the horizontal and vertical pixels, respectively, and cu and cv represent the intersection position of the optical axis and the image plane in the image coordinate system. And S represents a constant.

  As a specific example, for an xy position coordinate (Xw, Yw) of a human candidate detected by the laser range sensor 11 at a height of 80 cm from the floor assuming the waist height of the person, for example, the average human head height is 145 cm. In this case, the viewport conversion unit 31 projects and converts the three-dimensional real space coordinates (Xw, Yw, 145) into the two-dimensional image coordinates (u, v).

  FIG. 4 is an example of a step of changing the size of the upper body detection window when the upper body image is a square. In this example, the size of the upper body detection window is sequentially reduced in 16 steps from the size of 480 pixels × 480 pixels to the size of 48 pixels × 48 pixels, and the input for each detection window size. The upper body of various sizes is detected by repeatedly identifying the upper body while scanning the detection window in the image.

FIG. 5 is an example of an image when a plurality of upper body identification windows are integrated into one upper body detection window. It is an example of an image when the upper body determination unit 42 determines a person's upper body using a plurality of window sizes. When a person's upper body exists in the input image, a plurality of upper body identification windows are identified as shown in the left example of FIG. Of these multiple upper body identification windows, they are integrated into one detection window whose upper body is within a predetermined range. An example of integration is the right example of FIG.

  FIG. 6 shows a human candidate detection tracking unit 20 to which a signal from the laser range sensor 11 is input, a human tracking / attribute integration unit 30, and a human attribute estimation unit 40 to which a video from the camera 10 is input. It is an example of the data flow between. When there is a human candidate object in the detection area of the laser range sensor 11, the human candidate detection tracking unit 20 to which the laser measurement result from the laser range sensor 11 is input detects the object and determines whether it is a human candidate. . When the person candidate is determined, the person candidate tracking unit 24 of the person candidate detection and tracking unit 20 inputs the person ID and world coordinates (x, y) of each person candidate to the person tracking / attribute integrating unit 30.

  The person tracking / attribute integrating unit 30 holds the information of the person candidate ID and world coordinates (x, y) of the person candidate input from the person candidate detecting / tracking unit 20. Then, the viewport conversion unit 31 performs projection conversion to two-dimensional image coordinates based on the world coordinates (x, y). The upper body search area setting unit 32 sets an upper body search scale range and an upper body search area based on the projection-transformed coordinates. In this way, the person candidate ID, the upper body search scale range, and the upper body search area are output from the person tracking / attribute integration unit 30 to the person attribute estimation unit 40.

  In the person attribute estimation unit 40, based on the frame image from the camera 10, the upper body identification unit 41 detects the upper body of the person, determines the upper body position and range by the window detected by the upper body determination unit 42, and the face detection attribute estimation unit At 43, face detection and attribute estimation are performed, and the person ID and person attribute information are output to the person tracking / attribute integrating unit 30.

  The person tracking attribute determination unit 33 outputs the world coordinates (x, y) and the person attribute information in association with each other by matching the held person candidate ID with the person ID.

  Next, processing in the person tracking attribute determination unit 33 will be described in detail. The person attribute determination in the sequence image is performed using the person ID from the laser range sensor.

  FIG. 7 is an example of a determination process flow in the person tracking attribute determination unit 33. The person tracking attribute determination unit 33 receives the person ID and the attribute information of the person from the face detection attribute estimation unit 43. For example, a gender identification value and an age estimation value corresponding to each person ID are input to the person tracking attribute determination unit 33 as person attribute information.

  When a plurality of persons are shown in the camera image, a plurality of person IDs, gender identification values and age estimation values corresponding to the respective person IDs are input to the person tracking attribute determination unit 33. The person tracking attribute determination unit 33 performs processing for each person ID in each frame image. First, it is determined whether gender identification and age estimation corresponding to the person ID is performed in the current processed image (step 71). This is because there are cases where gender identification and age estimation are not performed, such as when a person is detected by a laser range sensor, but the face is hidden in the image. When gender identification and age estimation are performed, the gender identification value and age estimation value corresponding to the person ID are input. When gender identification and age estimation are not performed, “NA” is input. Has been.

人物追跡属性判定部３３は、人物属性パラメーター・テーブルとしてそれぞれの人物ＩＤごとに、判定回数、平均性別識別値、平均年齢推定値を保存している。 If gender identification and age estimation are performed in the current processed image, an average gender identification value update process is performed (step 72). The calculation of the average gender identification value update process is as follows.

The person tracking attribute determination unit 33 stores the number of determinations, the average gender identification value, and the average age estimated value for each person ID as a person attribute parameter table.

  The gender identification value is a value obtained as a result of gender identification by the support vector machine, and represents a distance from the gender identification curved surface. For example, when the gender identification value is a positive value, it is identified as a male, and when it is negative, it is identified as a female. As the absolute value of the gender identification value is larger, it indicates that the processed face image is farther from the identification curved surface for determining gender, and represents the probability of identification. Although the identification value of the support vector machine is used here, the likelihood and probability in another statistical identification method can be used instead.

そして、人物属性パラメーター・テーブルの判定回数を１増加し、人物属性パラメーター・テーブルを判定回数＝１とする（ステップ７４）。その後、人物追跡属性判定結果を出力する（ステップ７５）。 Next, an average age estimated value update process is performed (step 73). Similar to the average gender identification value update process, the average age estimated value is updated by the following calculation.

Then, the number of determinations in the person attribute parameter table is increased by 1, and the number of determinations in the person attribute parameter table is set to 1 (step 74). Thereafter, the person tracking attribute determination result is output (step 75).

  FIG. 8 is a specific flow of the determination method in the person tracking attribute determination unit 33. As shown in FIG. 8, when the attribute estimation of the person ID has not been performed yet, the parameter corresponding to the person ID in the person attribute parameter table has the determination number = 0, the average gender identification value = NA. Average age estimated value = NA.

  When the person ID, the gender identification value, and the age estimation value for each person of the current processed image (processed image number = 1) are input from the face detection attribute estimation unit 43, the average gender is calculated by the average gender identification value update process. An identification update value is calculated. In the case of the person ID = 1, the face detection attribute estimation unit 43 sets the person ID of the currently processed image in the state where the determination count corresponding to the person ID = 1 in the person attribute parameter table = 0 and the average gender identification value = 0. 1 is input, the average gender identification update value is 1.26, and the person attribute parameter table is updated to the average gender identification value of person ID = 1 = 1.26. Is done.

  In this example, the value of the result of gender identification by the support vector machine is shown. When the gender identification value is positive, it is identified as male, and when it is negative, it is identified as female. Is shown.

  Next, an average age estimated value update process is performed, and an average age estimated value update process is performed in the same manner as the average gender identification value update process to update the average age estimated value. In the case of the person ID = 1, the average age estimated update value is 24.84 by inputting the age estimated value 24.84 corresponding to the person ID = 1 of the currently processed image from the face detection attribute estimating unit 43. The person attribute parameter table is updated to the average age estimate of person ID = 1 = 24.84.

  Then, the number of determinations in the person attribute parameter table is increased by 1, and the number of determinations in the person attribute parameter table is set to 1. Thereafter, the person tracking attribute determination result is output. As an output of the determination result, if the person ID = 1 is taken as an example, since the average gender identification value corresponding to the person ID = 1 in the person attribute parameter table at this time = + 1.26, it is output as male and the average age is output. Since the estimated value is 24.84, 25 years old is output.

の場合、終了時の人物属性パラメーター・テーブルは、

となり、判定結果として

を出力する。 The person attribute determination process flow shown in FIG. 7 is performed for each person ID, and the processes performed so far for person ID = 1 in FIG. 8 are similarly performed for all person IDs in the current process image. Is called. That is, the same processing is performed for the person ID = 2 and the person ID = 3, and the person tracking attribute determination in the current processed image is finished. Person ID attribute information of person ID = 2 and person ID = 3 of the currently processed image from the face detection attribute estimation unit 43 is obtained.

, The person attribute parameter table at the end is

And as a judgment result

Is output.

の場合、終了時の人物属性パラメーター・テーブルは、

となり、判定結果として

を出力する。 FIG. 9 is a flow of the determination method in the next processed image (processed image number = 2). The person tracking attribute determination process is performed as shown in FIG. For example, the person ID attribute information from the face detection attribute estimation unit 43 is

, The person attribute parameter table at the end is

And as a judgment result

Is output.

の場合、終了時の人物属性パラメーター・テーブルは、

となり、判定結果として

を出力する。 FIG. 10 is a flowchart of the determination method for the next processed image (processed image number = 3). The person tracking attribute determination process is performed in the same manner as in FIGS. The person ID attribute information from the face detection attribute estimation unit 43 is

, The person attribute parameter table at the end is

And as a judgment result

Is output.

  In the person tracking attribute determination process with the processed image number = 3, the face detection attribute estimation unit 43 did not detect the face for the person ID = 1 and did not perform attribute estimation, so gender identification value = NA, age estimation value = An example in which NA is input to the person tracking attribute determination unit 33 is shown. In the determination process of step 71 of FIG. 7, since the person attribute is not estimated in the face detection attribute estimation unit 43, the process of steps 72 to 75 is not performed. As a result, person ID = 1 in the person attribute parameter table The number of determinations, the average gender identification value, and the average age estimate corresponding to are not updated. However, the previous value is retained, and as a result, both gender determination and age determination can be output. In this way, even when a scene with a hidden face occurs in the camera video sequence, the correct determination can be made by using the person attribute determination result of the person ID tracked in the camera video sequence so far. The result can be output.

  Furthermore, in the person tracking attribute determination process of process image number = 3, the correct gender identification is performed in the face detection attribute estimation unit 43 due to the influence of the variation in the illumination condition of the face image, partial hiding, etc. with respect to the person ID = 2. An example of the case where there was no such case is shown. In this processed image (processed image number = 3), the face detection attribute estimation unit 43 has the gender identification value = + 0.18 for the person ID = 2, and is identified as being slightly male. However, the gender identification value of the person ID = 2 in the camera video sequence is comprehensively determined including the gender identification value of the same person ID in the past processed image by performing the average gender identification value update process. As a result, the average gender identification value in this processed image = −1.18, and it can be determined correctly as a woman.

  FIG. 11 is a flowchart of the determination method when the processing image number = 4. FIG. 11 shows an example in which the face detection attribute estimation unit 43 does not perform the attribute estimation of the person ID = 3. In this case as well, as a result of the determination at step 71 in FIG. 7, the processing at steps 72 to 75 is not performed, and the determination count corresponding to the person ID = 3 in the person attribute parameter table, average gender identification value, average age estimation The value has not been updated. However, since the previous value is retained, sex determination and age determination are correctly performed.

となる。図１０に示される平均性別識別値更新処理５２と平均年齢推定値更新処理５３の演算により、人物属性パラメーター・テーブルが更新されるが、人物ＩＤ＝１は顔検出属性推定部４３からの人物ＩＤ属性情報が無いため更新は行なわれず、以前の値が保持される。また、人物ＩＤ＝４はこの処理画像で初めて人物ＩＤ属性情報が入力されたので、新しく人物属性パラメーター・テーブルに加わり、

となり、判定結果として

を出力する。 FIG. 12 is a flowchart of the determination method when the processing image number = 5. In the laser detection sensor person detection, person ID = 1 disappears from the detection area, and a new person ID = 4 is added to the detection area. Corresponding to the person detection of the laser range sensor, the person ID attribute information from the face detection attribute estimation unit 43 is also for three persons of person ID = 2, person ID = 3, and person ID = 4.

It becomes. The person attribute parameter table is updated by the calculation of the average gender identification value update process 52 and the average age estimated value update process 53 shown in FIG. 10, but person ID = 1 is the person ID from the face detection attribute estimation unit 43. Since there is no attribute information, no update is performed and the previous value is retained. In addition, since the person ID attribute information is input for the first time in this processed image, the person ID = 4 is newly added to the person attribute parameter table,

And as a judgment result

Is output.

  For the person ID = 3, the gender estimated value from the face detection attribute estimation unit 43 is −0 due to the influence of the illumination condition of the face image, partial hiding, etc., as in the case of the person ID = 2 of the processing image number = 3. Although it is slightly misidentified as female, such as 04, the gender identification value = + 0.61 is obtained by the person tracking attribute determination method of this method, and it is correctly determined as male. Although an example in which the average value is used for the determination has been described here, a median or a majority vote or the like can be used for the gender determination instead of the average value.

  Next, an example of output from the person tracking attribute determination unit 33 will be described. As described above, the person tracking attribute determination unit 33 outputs person attribute information such as a person ID, person tracking world coordinates (x, y), and age / sex. By outputting this to a display (not shown), it is possible to plot it together with the person ID on the two-dimensional bird's-eye view of the x axis and the y axis. In addition, the display on the bird's eye view is updated corresponding to the person ID and world coordinates of the person candidate input from the person candidate tracking unit 24 sequentially.

  FIG. 13 is an example of a screen output from the person tracking attribute determination unit 33. The attribute information (gender identification value, age estimation value) associated with the person ID is input to the person tracking / attribute integration unit 30 for each processed image from the person attribute estimation unit 40 to which the camera 10 is connected. The number of determinations, the average gender identification value, and the average age estimated value corresponding to each person ID are sequentially calculated and stored as the aforementioned person attribute parameter table. For the person ID based on the data from the laser range sensor 11 and the person tracking world coordinates (x, y), gender determination and age determination from the average gender identification value and the average age estimated value corresponding to each person ID, By performing the additional display, it is possible to generate a bird's eye view in which the person tracking information and the attribute information are displayed in an integrated manner.

  Further, as shown in FIG. 13, the upper body image determined by the upper body determination unit 42 may be displayed as a thumbnail image corresponding to the person ID in accordance with the plotted position. In this case, the person tracking attribute determination unit 33 receives the upper body detection image from the face detection attribute estimation unit 43 in a form associated with the person ID. This makes it possible to create and display a person tracking / attribute integrated bird's-eye view that is easier to understand as shown in FIG. For the generation of thumbnail images, for example, an image having the largest face detection identification value is selected from the upper body detection images of the same person ID in the sequence image from the camera video output from the face detection attribute estimation unit 43. By updating and saving, the upper body image showing the most front face is used as the thumbnail image. In this way, it is possible to increase the ease of understanding the bird's-eye view display. The person tracking position information from the laser range sensor and the person attribute information from the camera image can be integrated to generate a person tracking / attribute integrated bird's-eye view, and the action tracking with attribute information and a person photograph is very useful as a bird's eye view. It can be displayed clearly.

  Next, an example of collecting behaviors of people who have come out of an elevator in an office store complex will be described below as a method of collecting behaviors of customers using the person tracking attribute estimation device.

  FIG. 14 is an example of the arrangement of the camera 10 and the laser range sensor 11. The camera 10 and the laser range sensor 11 are installed so that the face of the person who has left the elevator can be photographed and the area that has left the elevator becomes a person attribute tracking area. By doing in this way, attribute determination information can be linked | related with the position tracking of the person who came out of the elevator.

  As shown in FIG. 14, one side of the shop / restaurant area in the building in the person attribute tracking area is A side, and one side of the building entrance / exit direction is B side, and the setting side for tracking the person attribute is determined. By tracking the position of the person attribute information, the number of people who got out of the elevator from the side A of the person attribute tracking area and the number of people who came out of the B side for each time zone To sum up.

FIG. 15 is a diagram illustrating a person entering / exiting determination method. How the person crosses the A side or the B side is determined by tracking the human candidate by the laser range sensor 11. The human candidate tracking unit 24 outputs the world coordinates (x, y) together with the person ID at the timing (t = n) for each scanning frequency of the laser range sensor. The person tracking attribute determination unit 33 uses the real space coordinates (x _n−1 , y _n−1 ) of the same person ID and the previous timing (t = _n−1 ) and (x _n , y _n ) of the current timing. , The motion vector of the human candidate on the real space coordinates is calculated. Then, by determining in which direction the motion vector has crossed the A side or B side, it is determined whether the set A side or B side has exited the elevator or entered the elevator. Can do.

  Due to the specifications of the laser range sensor 11, when the scanning frequency is very fast relative to the movement of the person candidate, the person ID and the real space coordinates output at the timing of the scanning frequency are thinned out to, for example, 1/5. By performing the same processing, it is possible to stably perform the crossing judgment on the set A side or B side.

  One laser range sensor can track a human candidate on real space coordinates. As shown in FIG. 14, a plurality of sides are set for setting a candidate for entering / exiting a human candidate with one laser range sensor. be able to.

  Figure 16 shows an example of the results of counting the number of men who left the elevators by age group and headed for shops and restaurants and the number of people headed for entrances and exits for each time period from 7 am to 10 pm FIG. Figure 17 shows an example of the results of counting the number of women who left the elevators by age and headed for shops and restaurants and the number of people headed for entrances and exits for each time period from 7 am to 10 pm FIG. In these examples, men and women are classified into four ages of under 20 years old, 20 years old and under 40 years old, 40 years old and under 60 years old, and 60 years old and over. As described above, the attribute information for each determined person ID is accumulated, and the person tracking attribute determination unit 33 aggregates the information so that the gender ratio and the age ratio of each time zone of the entire person who has exited the elevator Can be analyzed. FIG. 18 is a diagram of an example in which the number of people and the age ratio are tabulated for each person who comes out of the elevator.

  Furthermore, after exiting the elevator, it is possible to investigate a trend for each person attribute as a behavioral survey as to whether it headed for the shop & restaurant or headed for the entrance / exit. FIG. 19 is an example of a diagram in which the number of persons and the ratio of the person after exiting the elevator for each time zone in a male who is 40 years old or older and younger than 60 years old are obtained. FIG. 20 is an example of a diagram in which the number and ratio of the destinations after exiting an elevator focused on women aged between 20 and 40 years old. By doing this analysis, men between the ages of 40 and under 60 turn their feet to the shops and restaurants only after lunch breaks and regular hours, whereas women between the ages of 20 and under 40 years of time from lunch breaks to after regular hours. As a result, it can be seen that, for example, the time from lunch break to after regular hours can be expected to increase sales by carrying out a sales strategy targeting women. .

  In this way, it is possible to collect behaviors associated with person attributes. For example, it is possible to collect store customers' buying behaviors, and by knowing the flow in the store of the store customers that cannot be seen from the POS data. Product layout and floor layout can be optimized. It is also possible to strategically examine store products and restaurant menus in office store complex buildings by grasping the flow of people coming out of elevators in the building by time zone. .

Claims (9)

A human candidate clustering unit that extracts human candidates by clustering point clouds that are distances within a predetermined threshold from the measurement data of the laser range sensor;
A human candidate tracking unit that compares the measurement data between scanning frames, assigns a person ID to a point group having a distance difference within a predetermined range as the same human candidate, and outputs the human ID and position coordinates;
A viewport converter for converting the position coordinates into two-dimensional image coordinates;
An upper body search area setting unit that sets a window size and coordinate data as an upper body search area corresponding to the person ID based on the two-dimensional image coordinates;
A face detection attribute estimation unit that performs face detection based on the upper body search area from the frame image data from the camera and estimates a person attribute;
A person who associates a person ID with position coordinates and person attribute information based on the person ID and position coordinates output from the person candidate tracking unit and the person ID and person attribute information from the face detection attribute estimation unit. And a tracking attribute determination unit.   2. The person according to claim 1, wherein the person attribute tracking determination unit determines a person attribute of the person ID based on person attribute information to which the same person ID is assigned in a plurality of frame image data. Tracking attribute estimation device.   3. The person tracking attribute estimation according to claim 2, wherein the person attribute tracking determination unit determines a person attribute of the person ID based on one of an average value, a median value, and a majority vote of a plurality of person attribute information. apparatus.   The person attribute tracking determination unit further receives a thumbnail image from the face detection attribute estimation unit, plots the thumbnail image and the person attribute information based on the position coordinates corresponding to the person ID, and displays them on the display device The person tracking attribute estimation apparatus according to claim 1, wherein the person tracking attribute estimation apparatus outputs the information.   The person tracking attribute estimation apparatus according to claim 1, wherein the person attribute tracking determination unit calculates a motion vector from the position coordinates corresponding to the person ID, and determines a movement with respect to a set side.   The person tracking attribute estimation device according to claim 1, wherein the person attribute tracking determination unit aggregates person attribute information of a plurality of person IDs.   6. The person tracking attribute estimation apparatus according to claim 5, wherein the person attribute tracking determination unit totals together the person attribute information of a plurality of person IDs and the movement with respect to the set side. A human candidate clustering step of extracting human candidates by clustering point clouds that are distances within a predetermined threshold from the measurement data of the laser range sensor,
A human candidate tracking step of comparing the measurement data between scanning frames, assigning a person ID to a group of points whose distance difference is within a predetermined range as the same human candidate, and outputting the human ID and position coordinates;
A viewport conversion step of converting the position coordinates into two-dimensional image coordinates;
An upper body search area setting step for setting a window size and coordinate data as an upper body search area corresponding to the person ID based on the two-dimensional image coordinates;
A face detection attribute estimation step for performing face detection based on the upper body search area from frame image data from a camera and estimating a person attribute;
Based on the person ID and position coordinates output in the person candidate tracking step and the person ID and person attribute information output in the face detection attribute estimation step, the person ID is associated with the position coordinates and the person attribute information. A person tracking attribute estimation method comprising: a person tracking attribute determination step. On the computer,
A human candidate clustering step of extracting human candidates by clustering point clouds that are distances within a predetermined threshold from the measurement data of the laser range sensor,
A human candidate tracking step of comparing the measurement data between scanning frames, assigning a person ID to a group of points whose distance difference is within a predetermined range as the same human candidate, and outputting the human ID and position coordinates;
A viewport conversion step of converting the position coordinates into two-dimensional image coordinates;
An upper body search area setting step for setting a window size and coordinate data as an upper body search area corresponding to the person ID based on the two-dimensional image coordinates;
A face detection attribute estimation step for performing face detection based on the upper body search area from frame image data from a camera and estimating a person attribute;
Based on the person ID and position coordinates output in the person candidate tracking step and the person ID and person attribute information output in the face detection attribute estimation step, the person ID is associated with the position coordinates and the person attribute information. A person tracking attribute determination step to be performed;
Person tracking attribute estimation program to execute.

Images