KR20170007070A - Method for visitor access statistics analysis and apparatus for the same - Google Patents

Abstract

The present invention relates to a human detection method using a depth image analysis. A method of detecting a person using a depth image according to an aspect of the present invention includes: obtaining one or more depth images from one or more depth cameras; generating one analysis target height map using the one or more depth images; determining one or more candidate regions in the one analysis target height map; extracting feature information from each of the one or more candidate regions; and determining if a human object is detected in each of the one or more candidate regions based on the feature information. The feature information may include information on an area according to a height level and an area variation according to the height level in one candidate region. Further, an apparatus and a method for statistically analyzing an access of visitors can be provided based on the person detection method.

Description

[0001] METHOD FOR VISITOR ACCESS STATISTICS ANALYSIS AND APPARATUS FOR THE SAME [0002]

The present invention relates to a human detection scheme, and more particularly, to a visitor access statistical analysis method, apparatus, software, and recording medium storing such software using depth image analysis.

There is a method of using an apparatus for issuing an wait number table as one of the conventional visitor counting methods. This is mainly for the purpose of efficient customer service according to the order, but it can be used for the purpose of counting the visiting customers and analyzing the statistics by recording the number of times and the time on a separate server each time a waiting ticket is issued. However, since a person has to receive a number card directly, the issuance record can be manipulated, and the average staying time of the visitors can not be known because the visitor can not know when the visitor leaves.

As another conventional visitor counting method, there is a method using an infrared sensor installed at a door. This is a relatively simple and inexpensive method, but there is a problem in that the accuracy of the entrance detection and the coefficient is greatly deteriorated in a congested situation where several people pass the infrared ray sensor's sensing line or the sensor is covered by another object. Further, there is a problem that it can not be used for the purpose of acquiring additional information or monitoring other than the simple counting function.

In order to solve such a problem, a method of detecting a person by analyzing an image photographed through a camera is required in fields such as customer analysis (counting of entrance, counting of a moving line, etc.). The conventional human detection method is performed by analyzing an image captured by a camera in a specific space area such as near the entrance door. In general, a two-dimensional camera (for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor Camera), the accuracy of detection may be greatly reduced due to the complexity of the illumination environment or the background area. In addition, when a large number of people are moving, there is a problem that it is difficult to accurately count the number of people to enter because of the occlusion of objects to be detected. Therefore, in recent years, a method of detecting a person more accurately using a depth camera has been developed.

The depth camera can acquire three-dimensional spatial information of a photographed object from the photographed image. Therefore, compared to a two-dimensional image-based human detection method, it is less affected by occlusion or ambient lighting environment .

However, in a conventional human detection method using a depth camera, a camera system for acquiring depth information (for example, a depth camera may include structured light, TOF (time of flight ), Stereo (stereo), etc.), the effective shooting distance is limited with a fixed angle of view, so that there is a restriction in the installation environment. If the angle of view of the camera is fixed, the photographable area may vary depending on the distance from the camera to the object to be photographed. In addition, a general depth camera may have a limitation on the minimum distance and maximum distance that can be photographed.

Therefore, in consideration of the limitation of the depth camera, conventionally, a method has been proposed in which the camera is installed to be spaced apart from the central vertical plane of the entrance area, as shown in FIG. 1, so that the photographing area covers the entire entrance and exit area. However, in such a case, the angle between the direction of the camera and the bottom surface becomes low, which may cause occlusion of a part of the object located at the rear by the object at the front, .

In another conventional technique, a stereo camera is installed on the ceiling in a direction perpendicular to the floor so that problems such as obscuration of objects are not generated while using a depth camera. However, in such a case, it is very difficult to install the camera in a direction perpendicular to the floor in a high ceiling environment, and in a low ceiling environment, the shooting area becomes very narrow due to restriction of the camera angle of view. There was a problem that could not be achieved correctly.

The present invention relates to a method of detecting a person more accurately and efficiently from an image obtained from a depth camera without restriction on the installation position of a depth camera, detecting on the basis of the presence or absence of a visitor in real time, calculating and analyzing visitor access statistics And an object of the present invention is to provide an apparatus.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, unless further departing from the spirit and scope of the invention as defined by the appended claims. It will be possible.

A method of detecting a person using a depth image according to an aspect of the present invention includes: obtaining one or more depth images from one or more depth cameras; Generating one analysis target height map using the one or more depth images; Determining one or more candidate regions in the one analysis target height map; Extracting feature information from each of the one or more candidate regions; And determining if a human object is detected in each of the one or more candidate regions based on the feature information. The feature information may include an area according to a height level in one candidate region and information on a variation amount of the area according to the height level.

According to another aspect of the present invention, there is provided an apparatus for detecting a person using a depth image, comprising: an image receiving unit for obtaining one or more depth images from one or more depth cameras; An analysis object height map generation unit for generating an analysis object height map using the at least one depth image; A candidate region determining unit for determining one or more candidate regions in the one analysis target height map; A feature information extracting unit for extracting feature information from each of the one or more candidate regions; And a person judging unit for judging whether a human object is detected in each of the one or more candidate regions based on the feature information. The feature information may include an area according to a height level in one candidate region and information on a variation amount of the area according to the height level.

According to another aspect of the present invention, a computer-readable medium having stored thereon software executable by an apparatus for detecting a person using a depth image may be provided. The executable instructions causing the apparatus to obtain one or more depth images from one or more depth cameras; Generate one analysis target height map using the one or more depth images; Determine one or more candidate regions in the one analysis target height map; Extracting feature information from each of the one or more candidate regions; And to determine whether a human object is detected in each of the one or more candidate regions based on the feature information. The feature information may include an area according to a height level in one candidate region and information on a variation amount of the area according to the height level.

According to still another aspect of the present invention, a method of determining a person's access using a depth image can be provided. The method includes determining whether a human detection result exists in an nth frame of the depth image; Determining if an existing path exists if a human detection result exists in the nth frame; If the existing path exists, determining whether an addable path exists based on a time threshold and a spatial threshold; Updating the existing path if an addable path exists; And determining the entrance and exit of the person based on the start position and the end position of the updated path, wherein the difference between the last detected position of the existing path and the position of the human detection result in the nth frame is the spatial threshold And if the difference between the last detection point of the existing path and the human detection point in the nth frame is less than or equal to the time threshold, it can be determined that the additional path exists.

In various aspects of the present invention, the feature information may include a feature vector including elements representing the area occupied by the coordinates belonging to each of the plurality of coordinate groups classified according to the height level.

In various aspects of the present invention, one element of the feature vector may represent the area of the convex hull relative to the coordinates belonging to one coordinate group corresponding to one height level.

In various aspects of the present invention, the final feature vector x weighted by the area change amount between adjacent height levels is defined by the following equation,

s _i denotes an i-th element of the feature vector, and ? _k denotes a weight of a k-th element of the final feature vector.

In various aspects of the present invention, the final feature vector x, to which the weights are applied, is defined by the following equation,

s _i denotes an i-th element of the feature vector,

alpha _k denotes a weight for the k-th element of the final feature vector,

r _i is a value obtained by dividing the length of the minor axis by the length of the major axis in the area corresponding to the i-th height level, and may have a value of more than 0 and less than 1.

In various aspects of the present invention, the feature information includes information on an area according to one or more height levels corresponding to a head portion of a person, and an area according to one or more height levels corresponding to a part of the upper half of the person from a shoulder of a person can do.

In various aspects of the present invention, the one analysis target height map may be generated based on a height map generated for each of the one or more depth images.

In various aspects of the present invention, the one analysis target height map may be generated by combining a plurality of height maps.

In various aspects of the present invention, a coordinate transformation is applied to each of the one or more depth images, and a height map may be generated based on the depth image on which the coordinate transformation is performed.

The coordinate transformation may be performed by converting depth information of a depth image obtained from one depth camera into three-dimensional coordinates of a camera reference coordinate system, To three-dimensional coordinates on the real world coordinate system.

In the various aspects of the present invention, the conversion from the depth information of each pixel of the one depth image to the three-dimensional coordinate of the camera reference coordinate system is defined according to the following equation,

i represents a row index and a column index of pixels of the one depth image, d represents a value of the depth information, and x , y , and z represent values on the X axis, Y axis, and Z axis of the camera reference coordinate system Respectively, and ρ _hor , ρ _ver Represents a horizontal resolution and a vertical resolution of the one depth image, and θ _hor and θ _ver represent horizontal and vertical FOVs of the one depth camera, respectively.

In the various aspects of the present invention, the conversion from the three-dimensional coordinates of the camera reference coordinate system to the three-dimensional coordinates on the real world coordinate system may be performed on the X, Y, and Z axes of the camera reference coordinate system by φ, H, represents the height at which one camera is installed, and [phi], [theta], and [phi] represent the height in the X, Y, and Z axes of the camera reference coordinate system It is possible to indicate the angle at which one camera is installed.

In various aspects of the present invention, each of the one or more candidate regions may include one local peak.

In various aspects of the invention, at least one of the respective installation locations or installation angles of the one or more depth cameras may be adjustable.

In various aspects of the invention, one or more of the installation locations or installation angles of the plurality of depth cameras may be different.

In various aspects of the present invention, the determination of whether the human object is detected may be based on whether the feature information matches the feature information of the human object determined using a pre-trained classifier.

The features briefly summarized above for the present invention are only illustrative aspects of the detailed description of the invention which are described below and do not limit the scope of the invention.

According to the present invention, it is possible to detect a person more accurately and efficiently from an image acquired from a depth camera without restriction on the installation position of the depth camera, to detect whether or not a visitor enters or exits on the basis thereof, A method and apparatus may be provided.

The effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description will be.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included to provide a further understanding of the invention, illustrate various embodiments of the invention and, together with the description, serve to explain the principles of the invention.
1 and 2 are views for explaining examples of depth camera installation positions for human detection.
3 is a diagram for explaining the configuration and operation of the person detecting apparatus according to the present invention.
4 and 5 are views for explaining acquisition of one or more depth images according to the present invention.
6 is a diagram for explaining the configuration and operation of a human detection apparatus using one or more depth cameras according to the present invention.
7 is a diagram for explaining coordinate transformation according to the present invention.
8 is a diagram for explaining the relationship between the camera reference coordinate system and the real world coordinate system.
9 is a diagram for explaining an operation of converting coordinates of a camera reference coordinate system into coordinates of a real world reference coordinate system according to the present invention.
10 is a diagram for explaining a method of extracting feature information from each candidate region according to the present invention.
11 is a diagram showing an example of feature information of a candidate region according to the present invention.
12 is a diagram illustrating a system for analyzing visitor entrance statistics according to the present invention.
13 is a block diagram showing an example of an entrance / exit detector.
FIG. 14 is an exemplary view illustrating generation of a height map according to the present invention. FIG.
15 is a diagram exemplarily showing generation of a target height map according to the present invention.
16 is a diagram for explaining movement tracking and access determination operations according to the present invention.
17 is a view showing an exemplary entrance sensor control screen.
18 and 19 are views showing exemplary visitor entrance statistical analysis information.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification.

Depth image-based person detection method and apparatus

3 is a diagram for explaining the configuration and operation of the person detecting apparatus according to the present invention.

According to the present invention, a depth image photographed by each of the one or more depth cameras 110 can be transmitted to the image receiving unit 120. The image receiving unit 120 may acquire a depth image (i.e., one or more depth images) taken by each of the one or more depth cameras 110 and transmit the information to the analysis target height map generating unit 130 S10). The analysis target height map generation unit 130 may generate one analysis target height map used for human detection from the obtained information of one or more depth images (S20). The candidate region determination unit 140 may analyze one or more candidate regions for object detection using the local maxima method by analyzing the target height map (S30). The feature information extracting unit 150 may extract feature information such as the height of the head and the shoulder of each of the one or more candidate regions (S40). The person determining unit 160 may determine whether the detected object is a human object based on the extracted feature information (S50).

Hereinafter, a specific configuration of a depth image-based human detection method and apparatus according to the present invention will be described.

4 and 5 are views for explaining acquisition of one or more depth images according to the present invention.

In the present invention, when the depth camera photographs a specific space area, there is no restriction on the installation position, height, angle, and number of the depth camera. That is, in the present invention, one or a plurality of depth cameras are appropriately installed so as to cover all the specific spatial regions, and one or more depth images obtained therefrom (i.e., one depth image is acquired from one depth camera) (I.e., one analysis target image) to be subjected to image analysis for human detection.

4 and 5 show specific spatial regions. The specific space area means a space area to be analyzed for human detection. For example, it may be set as an area covering all the possible lines when a person enters and leaves a doorway.

As in the example of FIG. 4, it may happen that only one depth camera 110 can not cover the entire specific space area. That is, when one depth camera (or a pair of stereo cameras) is installed on the ceiling in a direction perpendicular to the floor as in the prior art, it may not cover the entire entrance area have.

As in the example of FIG. 5, a plurality of depth cameras may be installed to cover all the specific space areas. In the present invention, rather than merely installing and using a plurality of depth cameras, the problem of overlapping a plurality of depth images obtained by using a plurality of depth cameras while covering a specific space area by using a plurality of depth cameras is rather And to use it as information for increasing the accuracy for human detection.

Specifically, as in the example of FIG. 5, an area where a space taken by the first depth camera 110-1 overlaps with a space taken by the second depth camera 110-2 occurs. If human detection is performed individually using the depth images obtained by the respective depth cameras, there may be a problem that one person is detected twice in duplicate. When a plurality of depth images obtained from a plurality of depth cameras are used as in the present invention, a combined depth image can be generated. Further, by using the thus-synthesized depth image, the information that is erroneously recognized or unrecognized by any one of the depth cameras can be corrected by the information obtained by the other depth camera as described later, so that the accuracy of human detection can be enhanced have.

If only one camera can cover all the specific spatial regions, the analysis object height map is generated from one depth image, and candidate region determination, feature information extraction, and human object determination can be applied based on the generated height map. But does not exclude a depth image based operation within the scope of the invention. That is, in the present invention, even if only one depth image is used by one depth camera, a method of counting the area corresponding to a predetermined height or the number of pixels or pixels to be photographed is used However, since one or more candidate regions for object detection are determined, feature information is extracted from each candidate region, and a new method of detecting a human object is performed based on the extracted feature information, the accuracy of human detection .

Even if a single camera can cover all the specific spatial regions, more accurate human detection results can be expected when acquiring and using a plurality of depth images using a plurality of cameras as in the present invention.

In addition, the method using one or more depth images obtained from one or more depth cameras according to the present invention may be used when adjusting the installation position, height, or angle of the depth camera, or when increasing or decreasing the number of installed depth cameras It can be applied to all of one or more cases.

6 is a diagram for explaining the configuration and operation of a human detection apparatus using one or more depth cameras according to the present invention.

In the example of Fig. 6, there is shown a person detecting apparatus having N (N = 1, 2, 3, ...) depth cameras.

The first image receiving unit 120-1 may acquire a depth image from the first depth camera 110-1 and transmit the depth image information to the first coordinate converter 121-1. The first depth camera 110-1 may be connected to the first image receiving unit 120-1 in various ways (for example, a wired connection method such as USB, IEEE 1394, or a wireless connection method such as Bluetooth or WiFi) have. The first coordinate transforming unit 121-1 can generate real-world three-dimensional coordinates from the depth image information. The first height map generation unit 122-1 may generate a height map using coordinate values corresponding to valid values of a predetermined reference value or more based on the converted three-dimensional coordinates. And may be referred to as a first terminal preprocessor including a first image receiving unit 120-1, a first coordinate transforming unit 121-1, and a first height map generating unit 122-1. The first terminal preprocessing unit may be an embedded device corresponding to the first depth camera 110-1.

The same process as the coordinate transformation and the height map generation in the respective terminal preprocessing units can be performed on the depth images obtained from each of the second,. That is, one terminal preprocessing unit corresponding to one depth camera can be provided, and depth images from individual depth cameras can be processed in parallel or simultaneously.

The analysis target height map generation unit 130 generates the analysis target height map by a process of each terminal preprocessing unit with respect to the depth image photographed by each of the N depth cameras 110-1, 110-2, ..., 110-N .., N) height map, the height map (that is, the analysis target height map) can be generated using the first, second,.

The candidate region determining unit 140 can determine a position (i.e., a candidate region) in which it is determined that a human object can exist using the local maxima detection scheme in the analysis object height map. There may be one or more candidate regions in one analysis target height map (i.e., no candidate region may exist) (i.e., there is no possibility of existence of a human object) There is a location).

The feature information extraction unit 150 may extract feature information (e.g., a feature vector) of a human object in each candidate region if a candidate region exists.

The person determining unit 160 can determine whether the detected object in each candidate region is a human object based on the extracted feature vector.

The object estimating and counting unit 170 processes information for applications such as outbound counting, copper line prediction, security, and the like using the result of human object detection and transfers the information to an output interface (for example, a display, an audio / To another device.

A candidate region determining unit 140, a feature information extracting unit 150, and a person determining unit 160 may be referred to as a human detecting unit. The human detection unit may be implemented as a separate device physically separated from the terminal preprocessing unit or may be implemented as a functional module in which the terminal preprocessing unit and the human detection unit are distinguished in one device. In addition, the various functional units included in the human detection unit and the terminal preprocessing unit are not necessarily separately implemented, and one or more functional units may be integrated.

7 is a diagram for explaining coordinate transformation according to the present invention.

The terminal preprocessing unit (for example, image receiving unit) can receive the depth image photographed by the depth camera. That is, one depth image can be received from one depth camera, and coordinate transformation for N depth images acquired by N depth cameras can be performed. Hereinafter, the processing of one depth image received from one depth camera (for example, the first depth camera 110-1) will be described, and for depth images received from different depth cameras, Can be applied.

Specific examples of the coordinate transformation described in FIG. 7 can be performed by the terminal preprocessing unit or the coordinate transformation unit 121-1 corresponding to the first depth camera 110-1 of FIG.

In step S710, the coordinate transforming unit 121-1 may receive information on the depth image acquired by the image receiving unit 120-1 from the depth camera 110-1. The depth image may have a certain number of pixels on a two-dimensional plane of horizontal-vertical (or horizontal-vertical). Each pixel may have depth information (e.g., a distance value from the camera plane at which the depth camera is located to the target object).

In step S720, the coordinate conversion unit 121-1 may convert the three-dimensional coordinate values of the camera reference coordinate system based on the depth information of each pixel in the depth image. For example, depth information (e.g., distance values) for each pixel can be converted into three-dimensional coordinate values of the camera reference coordinate system using the angle of view of the camera and the horizontal-vertical (or horizontal-vertical) have.

If the value of each pixel of the depth camera is a three-dimensional coordinate value, the coordinate transformation may be omitted. The three-dimensional coordinate value calculated from the depth value or the three-dimensional coordinate value directly specified through the camera corresponds to the coordinate on the three-dimensional coordinate system (i.e., camera reference coordinate system) based on the camera.

The camera reference coordinate system will be described in detail with reference to Fig.

8 is a diagram for explaining the relationship between the camera reference coordinate system and the real world coordinate system.

8, the X axis and the Y axis in the coordinate system based on the camera can correspond to the horizontal axis and the vertical axis (or the horizontal axis and the vertical axis), respectively, which form a plane parallel to the camera plane. The Z axis can be orthogonal to the camera plane and correspond to the direction the camera is facing. The origin (O) of the camera reference coordinate system may correspond to the center point of the camera position.

The depth information (e.g., distance value) of each pixel of the depth image can be converted into a three-dimensional coordinate value on the camera reference coordinate system by the following equation (1).

In Equation (1), i and j denote the row and column index of the pixel of the depth image, respectively. and d represents a value of the depth information (e.g., a distance value). x , y , and z respectively correspond to values on the X-axis, Y-axis, and Z-axis of the camera reference coordinate system, respectively. ρ _hor and ρ _ver represent the horizontal resolution and the vertical resolution (or horizontal resolution and vertical resolution) of the depth image. θ _hor and θ _ver represent FOV (Field Of View) in the horizontal and vertical (or horizontal and vertical) directions of the camera.

Referring again to FIG. 7, in step S730, the coordinate transforming unit 121-1 transforms the three-dimensional coordinate values of the camera reference coordinate system (for example, x , y , and z values derived from Equation 1) It can be converted into a three-dimensional coordinate value.

The position of the object in the real space can be represented by converting the three-dimensional coordinate value on the camera reference coordinate system into the three-dimensional coordinate value of the real world coordinate system. In order to do this, information on the position, height, and angle of the camera is required, which may be a predetermined value in the process of installing the camera or may be a value derived through a calibration process after the camera is installed.

The process of converting the three-dimensional coordinate values on the camera reference coordinate system to the three-dimensional coordinate values of the real world coordinate system is a combination of rotation transformation, translation transformation, and reflection transformation in three-dimensional space Dimensional transformation (3-dimensional transformation).

For example, the three-dimensional space of the real world coordinate system consists of the X 'axis Y' axis Z 'axis, and the X' axis and Y 'axis respectively represent the horizontal and vertical axes Horizontal axis and vertical axis). Z 'axis is orthogonal to the plane formed by the bottom surface and can correspond to the ceiling direction. The origin of the real world coordinate system may correspond to a point orthogonal to the floor surface from the center point of the camera position.

Let H be the height at which the camera is installed, and assume that the installation angles in the X, Y, and Z axes of the camera reference coordinate system are (φ, θ, ψ). If the installation angle is (0, 0, 0), the X ', Y', and Z 'axes of the real world coordinate system correspond to the X, Y, and -Z axes in the camera reference coordinate system , The camera plane is parallel to the ceiling scene and the camera direction is the direction orthogonal to the bottom plane). Therefore, the X, Y, and Z axes are rotationally transformed by?,?, And?, The camera height is H, and the X-Y plane is subjected to the reflection transformation, so that the camera reference coordinate system can be transformed into the real world coordinate system.

Each of these transformation processes can be expressed by a matrix multiplication operation of a three-dimensional transformation matrix and a three-dimensional coordinate matrix having H, φ, θ, and ψ as parameters.

According to the above-described examples of the present invention, even if the depth camera is not installed in a direction perpendicular to the floor (for example, even if installed in an oblique direction), a height map can easily be generated through coordinate transformation. For example, when the depth camera is installed in an oblique direction, if the coordinate transformation is not applied, the depth information (i.e., the distance value) of the depth image corresponds to the distance from the object to the camera plane, The height may be distorted. However, according to the coordinate transformation according to the present invention, a more accurate height map reflecting the height of the actual object can be generated.

In addition, since the images captured by the plurality of depth cameras can be easily synthesized in the same coordinate system through the coordinate conversion, even if the positions and angles of the plurality of depth cameras are different from each other, The height map can be easily generated.

9 is a diagram for explaining an operation of converting coordinates of a camera reference coordinate system into coordinates of a real world reference coordinate system according to the present invention.

In step S910, the three-dimensional transformation matrix can be calculated (or determined) based on the camera installation height H and the camera installation angles?,?, And?. Such a three-dimensional transformation matrix may be predetermined at the time of installation of the camera, or may be updated and determined depending on the calibration of the camera, the angle adjustment after installation, and the like. If the position or angle of the camera is adjustable in real time, the 3D transformation matrix may be determined in real time by updating the parameters for each position and angle in real time.

In step S920, the three-dimensional coordinate value of the camera reference coordinate system for one pixel may be input. For example, the 3D coordinate value of the camera reference coordinate system can be calculated from the depth information of one pixel in the depth image obtained from the depth camera according to Equation (1).

In step S930, the three-dimensional coordinate value of the real world coordinate system can be calculated by converting the three-dimensional coordinate value of the camera reference coordinate system for one pixel by the three-dimensional conversion matrix. This transformation can be performed by multiplying a matrix.

In step S940, it is checked whether or not the conversion from the three-dimensional coordinate value of the camera reference coordinate system to the three-dimensional coordinate value of the real world coordinate system has been performed for all the pixels in the depth image. . Thus, three-dimensional coordinate values on the real world coordinate system can be obtained for all the pixels in the depth image (i.e., all the coordinates on the camera reference coordinate system). Alternatively, a three-dimensional coordinate transformation may be performed only on a specific region (for example, a region of interest excluding the background region) in the depth image. In this case, all the coordinates in step S940 refer to all the coordinates in the specific area.

Using the coordinate values thus obtained in the embodiment coordinate system, the depth from the ceiling to the floor for the object photographed by the camera (that is, the height from the floor to the ceiling) can be determined. A height map can be generated based on the depth or the height for all the pixels in the depth image (or all the pixels within the specific area). As described with reference to FIG. 6, a height map can be generated for the depth image photographed by one depth camera.

Orthographic projection may be used to project a three-dimensional object onto a two-dimensional plane for the generation of a height map. The height map may be created by setting in advance one or more specific regions (or regions of interest) for which a person is to be detected, rather than creating the entire region of the depth image, and only when the real- It is also possible to reduce the load of height map generation in a projection manner.

A particular area can be determined by six thresholds: left, right, front, back, top, and bottom. The two-dimensional plane on which the height map is displayed can be expressed as an image having a predetermined number of pixels horizontally and vertically. The range determined by the threshold values for the left, right, front, and back of the specific area may correspond to a two-dimensional plane. The threshold values above and below the specific region may correspond to the maximum height value and the minimum height value of the two-dimensional image pixels (or coordinates), respectively.

In addition, a plurality of three-dimensional coordinates may be projected on one and the same coordinate on a two-dimensional plane. For example, there may be a plurality of three-dimensional coordinates having different values only in the Z-axis direction on the three-dimensional space and having the same value on the X-Y plane. In this case, the coordinate having the highest value of the plurality of three-dimensional coordinates in the Z-axis direction (i.e., the height value) can replace the remaining coordinates. That is, by comparing the height values of the plurality of three-dimensional coordinates, only the three-dimensional coordinates having the largest height value are projected on the two-dimensional plane, and the remaining three-dimensional coordinates can be discarded.

Next, one analysis target height map can be generated from the plurality of height maps.

One or more height maps are generated for each of the N depth images, and the plurality of height maps can be synthesized into one analysis target height map. The analysis target height map may include all the height maps for a plurality of specific regions (for example, the ROIs) set in the plurality of depth images.

Further, the analysis target height map may include all threshold values of a plurality of height maps to be combined. For example, each of the height maps is generated for a depth image photographed in a different range partially overlapped with each other or not overlapped with each other, and a threshold value set for generation of a height map for each depth image (for example, All the pixels or coordinates of a specific region (for example, the region of interest) set by the right, left, front, right, front, back, top and bottom of the target object should correspond to pixels or coordinates of the target height map. In this case, one pixel (or coordinate) of the analysis target height map may correspond to different pixels (or coordinates) of different height maps, and one pixel (or coordinate) of the analysis target height map may correspond to one height (Or coordinates) of the map.

(Or coordinates) of the pixels of the respective height maps (or coordinates) in accordance with the correspondence between the pixels (or coordinates) of the respective height maps and the pixels (or coordinates) The height of the analysis target height map can be generated in such a manner that the height of the analysis target height map is substituted. If each pixel (or coordinate) of a plurality of height maps to be synthesized corresponds to one and the same pixel (or coordinate) in the analysis target height map (e.g., one pixel (or coordinate) of the first height map) (Or coordinates) of the plurality of height maps corresponds to one pixel (or coordinate) of the analysis target height map), the pixel (or the coordinates) of the second height map corresponds to one pixel (Or coordinates) having a value can replace the remaining pixels (or coordinates). That is, only the pixel (or the coordinate) having the largest height value is substituted into the pixel (or coordinate) of the analysis object height map by comparing the height values of the pixels (or the coordinates) of the plurality of height maps, ) Can be discarded.

Next, based on the analysis target height map, one or more candidate regions for detection can be determined.

A pixel (or coordinate) having the highest height value in a certain area in the analysis target height map may be referred to as a local maxima. The location corresponding to the local peak is the area where the person object is likely to exist. Therefore, for each pixel (or coordinate) in the analysis target height map, the height values are compared with other pixels (or coordinates) within a predetermined radius around the corresponding pixel (or coordinate) (Or coordinates) is higher than the corresponding pixel (or coordinate), the position of the corresponding pixel (or coordinate) can be determined as the candidate region. One or more candidate regions may be determined within the analysis target height map.

If it is determined that a human object is detected at the position only because it has a larger height value than other pixels (or coordinates), it may be misleading to refer to an object of a human height level that is not a human being as a human object. Therefore, in one example of the present invention, the feature information may be extracted for the candidate region, and the presence or absence of the person object may be determined based on the feature information.

When one or more candidate regions are determined, the feature information can be extracted for each candidate region. The feature information may include an area of a cross section depending on the height of the human body and a feature vector indicating a change in the area. For example, the feature vector may be calculated using Equation 2 below.

In Equation (2), the vector s represents a feature vector. α _k represents a weighting factor for each element of the final feature vector. That is, the final feature vector x represents a feature vector to which a weight is applied to an area change amount between adjacent height levels. s _i represents the i-th element of the feature vector. For example, the feature vector s may represent an area (for example, a convex hull area) according to the height level of the candidate area. That is, s ₁ means the area corresponding to the first height level. The vector d corresponds to the difference value (delta) between the i-th element and the (i + 1) -th element of the feature vector s. For example, d ₁ represents the difference between s _{i + 2} and s ₁ .

10 is a diagram for explaining a method of extracting feature information from each candidate region according to the present invention.

The feature information extraction for the candidate region is performed by extracting the feature information of the candidate region from the coordinates (i.e., the horizontal-vertical axis, the horizontal-vertical axis, or the coordinates in the plane on the XY axis) And generating the feature vectors by classifying them into coordinate groups.

For example, it is assumed that the maximum value and the minimum value of the height values of the pixels (or coordinates) in the analysis target height map are H _max and H _min , respectively, and are divided into K (K is a natural number) height levels. For example, the first height level is greater than or equal to H _{max and} greater than H ₁ , the second height level is greater than or equal to H _{1 and} greater than H ₂ , ..., and the Kth height level is less than or equal to H _{K -1} H _min (Or the first height level is less than H _max less than H ₁ , the second height level less than H ₁ less than H ₂ , ..., K th height level is less than H _{K -1} H _min Or more).

Here, the size of the range covered by each height level (or the number of height values belonging to each height level) need not be the same. That is, the range size of the first, second, and third height levels may be smaller than the range size of the Kth height level. In this case, for example, the pixels (or coordinates) corresponding to the height corresponding to the height of the human head to the shoulder can be further classified and grouped.

Also, the height level can be corrected by multiplying the weight according to the height value of the local highest point so that each height level is formed at a similar position in each human body according to the human key. That is, the range size of the height level applied to the candidate region having the highest local peak may be set larger than the range size of the height level applied to the candidate region having the lower local peak.

It is possible to check the height values of the adjacent coordinates starting from the coordinates corresponding to the local peak, and to include the coordinates within the predetermined height difference difference range (i.e., corresponding to the predetermined height level) in the same coordinate group, If there is no coordinate belonging to the corresponding height level, it is possible to repeat the generation of the coordinate group with respect to the next height level, thereby determining the coordinate group according to the height level. The convex hull containing all the coordinates of one coordinate group is obtained, and the area of the calculated convex hull can be determined as the area with respect to the corresponding height level. Thus, the area for the corresponding height group can be determined for each coordinate group.

Hereinafter, a method for extracting feature information for one candidate region will be described in detail with reference to FIG.

In step S1010, the information of the local highest point in the candidate area (i.e., the coordinate value of the local highest point and the height value at the position) can be input.

The coordinate value of the local highest point may be added to the first coordinate group corresponding to the first height level in step S1020. The first height level may be defined as a height value of a predetermined range including the height value of the local peak, and the first coordinate group may be a group including the coordinates of the local peak.

If there is a coordinate corresponding to the k-th height level in the analysis object height map in step S1030, the coordinate may be added to the k-th coordinate group. Here, k can have a value of 1, 2, ..., K. If k = 1, the height value of coordinates adjacent to the local maximum coordinate is checked, and if there is a coordinate having a height value corresponding to the first height level, the coordinate can be added to the first coordinate group.

In step S1040, it is determined whether there is a valid neighboring coordinate at the k-th height level. If YES, the process proceeds to step S1030 to add the corresponding coordinate to the k-th coordinate group corresponding to the k-th height level. Determining whether there is a valid adjacent contour coordinate at the k-th height level determines whether the height value of a certain coordinate is included in the range of the k-th height level and whether the coordinate falls within a predetermined distance from the center (or the coordinates of the local highest point) It can be a standard. For example, it can be determined that the coordinates in the range of the k-th height level are not valid when the coordinates are out of the human body radius from the local peak. If there is no valid adjacent coordinate at the k-th level, the process proceeds to step S1050.

In step S1050, it is possible to determine whether k is the maximum value K or not. If not (i.e., k < K), the process proceeds to step S1060 to increase the value of k by 1, and proceeds to step S1030 to correspond to the next height level (i.e., the height level corresponding to k increased by 1) (I.e., a coordinate group corresponding to k increased by 1), the coordinate can be added. If k = K, it means that grouping of valid coordinates in each of all the height levels from the height level 1 to the height level K is completed.

Convex hull can be calculated for each of the K coordinate groups in step S1070. Convex Hull means the minimum size polygon that contains all the coordinates in a two-dimensional plane.

The area for each convex hull can be calculated in step S1080. For example, the area of the convex hull can be calculated by counting the number of pixels (or coordinates) included in the convex hull.

It is possible to determine a feature vector for one candidate region (i.e., an area based on the height level) according to the method described with reference to FIG. If there are a plurality of candidate regions, the feature vectors may be determined in a similar manner for each of the remaining candidate regions.

11 is a diagram showing an example of feature information of a candidate region according to the present invention.

11 shows an example of a front view and a height map for a human body, and the right diagram exemplarily shows an area of a convex hull according to a height level. As in the example of Fig. 11, a plurality of height levels can be set closely to the head portion to increase the accuracy of human detection. For the upper body from the shoulder, the height level may be set to be less dense than the head part. From the convex hull area according to each height level, feature information of the candidate region (i.e., feature vector) can be determined.

That is, the feature vector according to the present invention may include information on the area of the cross section according to the height level and the change amount of the area based on the local maximum point. By using such a feature vector, the shape of a person's head, shoulders, and upper body can be specifically modeled. As described above, since the feature vector capable of expressing a detailed shape model of the head, shoulder, and upper body is used for human detection, it is possible to more easily and accurately determine whether the object in the candidate region is a human. Thus, the accuracy of human detection can be greatly improved.

As described above, the feature information for determining whether a person is human can include modeling information for a three-dimensional shape ranging from the head end of the person to the shoulder and the upper body. For this purpose, the area ratio of the cross section (i.e., convex hull) along the height level of the region including the head and the upper half of the candidate region, the area change amount between the adjacent height levels, and the aspect ratio of the area can be used . The aspect ratio of the cross section may correspond to the ratio of the short axis to the long axis of the minimum rectangle including the cross section.

In other words, the final feature vector x considering the image ratio of the cross section may be used in addition to the change amount of the cross section between the adjacent height levels as shown in Equation (2). The final feature vector x can be calculated as: &lt; EMI ID = 3.0 &gt;

The feature vector s in Equation (3) represents the area of the cross section according to the height level of each candidate region. The vector d represents the amount of area change between each adjacent height level. The vector r represents the image ratio of each area.

and? _k represents a weight for each element of the final feature vector. s _i represents the i-th element of the feature vector. For example, the feature vector s may represent an area (for example, a convex hull area) according to the height level of the candidate area. That is, s ₁ means the area corresponding to the first height level (i.e., the height level including the highest point of the candidate area). The vector d corresponds to the difference value (delta) between the i-th element and the (i + 1) -th element of the feature vector s. For example, d ₁ represents the difference between s _{i + 2} and s ₁ . r _i is the value obtained by dividing the length of the minor axis by the length of the major axis in each area and can have a value of more than 0 and less than 1.

Next, it is possible to determine whether the object having the feature vector of the candidate region is a person, using a classifier that has been learned in advance. That is, based on the comparison reference feature vector determined by cumulatively learning a plurality of samples of the feature vector of the actual person and the matching (or similarity) between the feature vector of the candidate region derived from the depth image obtained by the depth camera So that it can be determined that the object is a person when the matching is performed (or the degree of similarity is higher).

In addition, the classifier learning may be performed using a database that stores a plurality of samples of a depth image including a human object and a depth image that does not include a human object, and performs a learning algorithm such as a support vector machine (SVM) . &Lt; / RTI &gt; The machine learning algorithm can be applied appropriately according to the depth camera method, image characteristics, characteristics of feature vectors, and so on.

Method and apparatus for analyzing visitor entrance statistics

Visitor access statistics are very useful information for operating a store or facility. For example, a company with a large number of stores across the country could use the statistics of customer visits for each store to establish an operational strategy that fits the store's circumstances. Smaller sales compared to the number of visitors can focus on increasing sales from visitors, and if there are fewer visitors than sales, they can focus on increasing visitors. Also, in a large shopping mall such as a large shopping mall or a department store, statistical information on the number and time of customers passing through the main moving point may be used to strategically arrange the goods by analyzing the flow or movement line of the customer. In addition, we use marketing data and analysis results based on visitor's access data according to various situations such as average number of visitors per day, number of visitors per day, time period, season, event, etc. Can be utilized. Such visitor statistics and analysis information can be used to improve congestion prediction and operational efficiency even in tourism and landmark facilities where many visitors such as museums and theme parks, as well as businesses that own or operate a large number of stores such as large marts or department stores or chain stores with large merchant stores And the demand has increased greatly in recent years.

In order to generate and use customer analysis information based on the visitor's access record, it is important to accurately detect the entry and exit situation of a person entering and leaving without error. For this, a depth image-based human detection method and apparatus as described in the above embodiments of the present invention can be used. Furthermore, based on the accurate person detection information, it is possible to perform the customer analysis as described above by detecting the entrance / exit of the visitor and calculating the visitor entrance statistical information using the information.

Hereinafter, various examples of the present invention for a visitor entrance statistical information analysis method and apparatus will be described in detail.

12 is a diagram illustrating a system for analyzing visitor entrance statistics according to the present invention.

The plurality of access detectors 1210-1, 1210-2, ..., 1210-N are devices that allow a person to enter and exit from one access area. Of course, one access sensor may be installed and used for one access area. For example, each of the access sensors may include a depth-of-field-based human detection device using the three-dimensional camera described above.

Each of the plurality of access detectors 1210-1, 1210-2, ..., 1210-N may be connected to the access statistics server 1220 and the access sensor control server 1240 through the network. Each of the access detectors 1210-1, 1210-2, ..., 1210-N can transmit access information to the access statistics server 1220 when the access status is detected. The access information may include, for example, the detection time, the number of persons who have gone in and the identification information of the sensor (for example, information on the location of the store and the access area).

The access statistics server 1220 stores the access information received from the access detectors 1210-1, 1210-2, ..., 1210-N in the same group (e.g., detectors installed at various outlets of one store And various statistics such as the accumulation time, the input / output count, the time series, the day, the month, and the day of the week can be calculated and stored in the database 1230.

The access sensor control server 1240 controls access sensors 1210-1, 1210-2, ..., 1210-N for remote operation and maintenance of the access detectors 1210-1, 1210-2, ..., 1210- ., 1210-N). The access sensor control server 1240 monitors the status of each of the access detectors 1210-1, 1210-2, ..., 1210-N, stores the operation status in the database 1230, A request from the user terminal 1270 through the access control server 1260 and an administrator request through the access controller control terminal 1260. [

The access sensor control terminal 1260 is connected to the access sensor control server 1240 through the network and monitors the access sensors 1210-1, 1210-2, ..., 1210-N through the relay of the access sensor control server .

The web server 1250 can provide the access information stored in the database 1230 to the user terminal 127 connected through the network.

The access statistics server 1220, the database 1230, the access sensor control server 1240, and the web server 1250 may all be configured as one physical server, or each may be configured as a separate physical server Or may be connected via a network.

13 is a block diagram showing an example of an entrance / exit detector.

13, the depth cameras 110-1, 110-2, ..., 110-N, the terminal preprocessing units (image receiving units 120-1, 120-2, ..., 120- (122-1, 122-2, ..., 122-N), a human detection unit (analysis target height map generation unit 122-1, 122-2, ..., The candidate region determining unit 140, the feature information extracting unit 150, and the person determining unit 160 are the same as those shown in FIG. 6, and therefore, duplicate descriptions are omitted.

13, the entrance information processing unit 1300 includes a human detection unit (analysis object height map generation unit 130, candidate region determination unit 140, feature information extraction unit 150, and person determination unit 160) A determination section 1310 and an access information transfer processing section 1320. [ Here, the movement tracking and determination unit 1310 and the access information transmission processing unit 1320 may be included in the object tracking and counting unit 170 in the example of FIG.

FIG. 14 is an exemplary view illustrating generation of a height map according to the present invention. FIG.

6, it is possible to use a plurality of depth cameras with respect to one entrance / exit area when the entrance / exit area can not be covered by one depth camera, and it is also possible to use a plurality of depth cameras (110-1, 110-2 , ..., 110-N) The plurality of terminal preprocessing units can process image reception, coordinate conversion, and height map generation in parallel.

The left drawing of FIG. 14 exemplarily shows an image obtained from one depth camera, that is, a depth image before the coordinate transformation is applied. The right diagram of FIG. 14 exemplarily shows a height map generated by applying coordinate transformation to the obtained depth image.

15 is a diagram exemplarily showing generation of a target height map according to the present invention.

For example, the depth map 1 obtained by the depth camera 1 may be transformed by the terminal preprocessing unit to generate the height map 1. Likewise, the depth map 2 obtained by the depth camera 2 can be generated by the terminal preprocessing unit through the coordinate transformation. Thus, N height maps can be generated from the N depth images obtained by each of the N depth cameras.

The analysis object height map generation unit 130 can generate an analysis object height map based on a plurality of height maps (e.g., height map 1, ..., height map N). On the other hand, the candidate region can be determined based on the analysis object height map, and the feature information can be extracted to determine whether the candidate region is a human.

The movement tracking and entry determination unit 1310 receives information on the candidate region determined as a person and the position of the candidate region from the human detection unit (e.g., the person determination unit 160) (For example, the detection time, the number of persons who have gone in and the identification information of the corresponding sensor) by determining whether the incoming or outgoing situation is an incoming or outgoing situation. The specific operation of the movement tracking and entrance determination unit 1310 will be described later with reference to FIG.

The access information transmission processing unit 1320 receives access information from the movement tracking and access determination unit 1310 and transmits the access information to the server (e.g., the access statistics server 1220, the access sensor control server 1240, and the web server 1250) One or more).

The candidate determining unit 140, the feature information extracting unit 150, the person determining unit 160, the movement tracking and entry determining unit 1310, and the access information transmitting processing unit 1320, The access information processing unit 1300 and the terminal preprocessing unit may be constituted by one physical device. The input / output information processing unit 1300 and the terminal preprocessing unit may be physically separate from the terminal preprocessing unit.

16 is a diagram for explaining movement tracking and access determination operations according to the present invention.

In order to track the movement and judge the entrance / exit, it is possible to determine whether or not the moving image is analyzed by analyzing the moving trajectory for each image frame in the continuous depth image frame. Specifically, it is possible to determine whether a human detection result exists for each of successive depth image frames. If there is a human detection result in the n-th depth image frame, it is determined that there is no human detection result in each of one or more depth image frames (i.e., (n + 1) And the movement route of the person can be generated based on the detected three-dimensional position. That is, a moving path can be created by connecting a three-dimensional position where a person is detected from a depth image frame in which a person detection result is first generated to a frame immediately before a depth image frame in which no human detection result exists. When one movement route is determined (that is, when the movement route is terminated), it is possible to determine whether a person is entering or exiting.

In step S1610, it is possible to determine whether a human detection result exists in the nth depth image frame. If there is a person detection result, the flow advances to step S1620 to determine whether one or more existing paths exist. If there is no human detection result, the flow advances to step S1670 to determine whether there is an end path.

If there is more than one existing route in step S1620, the flow advances to step S1630 to determine whether there is an addable route. If the existing path does not exist in step S1620, the flow advances to step S1660 to generate a new path. That is, the person position newly detected in the n-th frame can not be determined as a person who has been previously detected, and it can be determined that a new person has started to move. After generating the new path, the process advances to step S1650 to increase the value of n by 1 in order to process the subsequent frame of the depth image. Accordingly, the process returns to step S1610 to determine whether there is a result of human detection in the subsequent frame.

In step S1630, it is determined whether or not an additional path exists. Here, in order to determine whether or not the addable route exists, the last point of the existing route and the position on the new route must be close to each other in space and time. (I.e., a spatial threshold) and a predetermined threshold in time (i.e., a time threshold), and determines whether or not the current path is within a spatial threshold based on the last position of the existing path When a person is detected within a time threshold based on the last time point when a person is detected in the existing path while the person is detected in the existing path, it can be determined that an additional path exists. If it is determined in step S1630 that the addable route exists, the flow advances to step S1640 to update the route by adding the newly detected person location to the matched existing route. After updating the path, the process advances to step S1650 to increase the value of n by 1 to process the subsequent frame of the depth image. Accordingly, the process returns to step S1610 to determine whether there is a result of human detection in the subsequent frame.

If a person is detected at a position exceeding the spatial threshold based on the last position where the person of the existing path is detected or if a person is detected at a time point exceeding the time threshold based on the last time when the person is detected in the existing path , It can be determined that the newly detected human position can not be added to the existing route. In this manner, if it is determined in step S1630 that no additional path exists, the flow advances to step S1660 to generate a new path. That is, the person position newly detected in the n-th frame can not be determined as a person who has been previously detected, and it can be determined that a new person has started to move. After generating the new path, the process advances to step S1650 to increase the value of n by 1 in order to process the subsequent frame of the depth image. Accordingly, the process returns to step S1610 to determine whether there is a result of human detection in the subsequent frame.

In step S1670, it is possible to determine whether or not the terminated path exists. That is, when it is determined in step S1610 that there is no human detection result in the depth image frame, it can be determined whether or not the movement of the person is completed. Here, in order to determine the terminated path, a predetermined threshold value (i.e., a termination threshold value) for termination determination is set, and the elapsed time from the last time point when the person is detected in the existing path to the point in time when the current frame is input If it is within the termination threshold, it can be determined that the path has not yet been terminated. That is, if it is determined in step S1670 that the completed path does not exist, the process advances to step S1650 to increase the value of n to 1 in order to process the subsequent frame of the depth image. Accordingly, the process returns to step S1610 to determine whether there is a result of human detection in the subsequent frame.

If it is determined in step S1670 that the elapsed time from the last point of time when a person is detected on the existing path to the point in time when the current frame is input exceeds the termination threshold, it can be determined that the path has ended. If the path is terminated, the flow advances to step S1680 to perform the entrance / exit determination.

In step S1680, it is possible to compare the start position and the end position of the finished path with the access baseline based on a preset access baseline, and determine whether or not the access is permitted. For example, when the start position and the end position of the finished path correspond to the external position and the internal position with respect to the entrance baseline, respectively, it can be determined that the visitor has arrived. Alternatively, when the start position and the end position of the finished path correspond to the inner position and the outer position with reference to the entry / exit reference line, it can be determined that the visitor has gone out. If the starting and ending positions of the finished path are both external or all internal with respect to the entry baseline, they may not be considered in the visitor access factor.

1210-N, entrance and exit statistics server 1220, database 1230, access sensor control server 1240, web server 1250 ), The access sensor control terminal 1260, and the user terminal 1270 will be described below.

When the access statistics server 1220 is located at the head office of a company having a large number of stores and the access detectors 1210-1, 1210-2, ..., 1210-N exist in the number of the access areas for each store I suppose. In this case, the access sensors 1210-1, 1210-2, ..., 1210-N can transmit the access information accumulated every time the access situation occurs or the set time unit to the access statistics server 1220. [ For example, access information may be transmitted whenever a person enters or exits, or access information accumulated within a predetermined time interval such as 10 seconds or 1 minute may be transmitted to the access statistics server 1220.

The access information transmitted from the access detectors 1210-1, 1210-2, ..., 1210-N to the access statistics server 1220 includes, for example, an access sensor ID, an access occurrence time, And the like. The IDs of the access detectors 1210-1, 1210-2, ..., 1210-N may be identification information that can identify the store where the access detector is installed and the access area. The entrance / exit time is the time when the entrance / exit situation occurred, and the number of entries and the number of departures are the number of persons entering or leaving at the same time or within the set time interval.

In the access statistics server 1220, access information transmitted from the access sensor may be recorded in the database 1230, and the statistical information calculated from the access information may be recorded in the database 1230 at predetermined time intervals. The statistical information recorded by the entrance statistics server 1220 in the database 1230 may include, for example, the number of entries, the number of exits, the number of residence, the average residence time, etc. according to the time zone according to each store.

The access statistics server 1220 can also calculate additional information that can be calculated from such statistical information and write it in the database 1230. [ The additional information may include, for example, entry / exit statistics for each entrance station, store or specific store group according to time and period, such as monthly, weekly, and yearly.

The access sensor control server 1240 may support the following operations. For example, a user corresponding to the system administrator who installs and builds or maintains the access detectors 1210-1, 1210-2, ..., 1210-N may use a dedicated access sensor control terminal 1260 And 1210-N connected to the access sensor control server 1240 to check the status of each of the access sensors 1210-1, 1210-2, ..., and 1210-N installed in the entire store, and the access sensors 1210-1, 1210-2, ..., 1210-N. Alternatively, the system administrator may access the web page for the administrator of the visitor access statistical analysis service using a general web browser, not through the access sensor control terminal 1260, and access the web server 1250 linked with the access sensor control server 1240 1210-N by using the access sensors 1210-1, 1210-2, ..., 1210-N.

17 exemplarily shows an access sensor control screen using the access sensor control terminal 1260 or a web browser. The access sensor control terminal 1260 and the access sensor control server 1240 are connected to a list of the access detectors 1210-1, 1210-2, ..., 1210-N and a list of the access detectors 1210-1, 1210-2 The height map, the analysis object height map, and the operation setting values of the access sensors 1210-1, 1210-2, ..., and 1210-N, You can send and receive. For example, you can connect to a specific access detector by setting the address and port of a specific access detector and selecting the device ID. In addition, the camera parameter setting of the connected access detector includes color, mirroring, up / down reversal, and out-side setting, and the camera height and angle can be set. Also, the boundaries covered by the access sensor can be set by using the values of Top, Bottom, Front, and Left. In addition, the entry baseline (In / Out Line) can be set separately from the entry baseline and the exit baseline.

The access sensor control server 1240 may access the access sensor control server 1240 from the access sensor control terminal 1260 to transmit the client ID of the access sensor control terminal 1260 to the access sensor control server 1240, May perform client ID authentication and transmit the authentication result to the access sensor control terminal 1260. [ Also, the access sensor control terminal 1260 may request the access sensor list to the access sensor control server 1240, and the access sensor control server 1240 may transmit the access sensor list to the access sensor control terminal 1240. Also, the access controller control server 1240 requests the control of the specific access sensor from the access sensor control terminal 1260, determines whether the access controller control server 1240 has authorized the access, And notifies the terminal 1240 of this. In addition, the access sensor control terminal 1260 may request the access sensor control server 1240 to transmit the video streaming of the specific access sensor, request the operation state and the set value of the specific access sensor, You can also request to change the setting value. In the above description, the operation of the access sensor control terminal 1260 can also be applied to the operation of the management client through the manager web page of the visitor entrance statistical analysis service using a general web browser.

18 and 19 are views showing exemplary visitor entrance statistical analysis information.

The web server 1250 may obtain information necessary for a user or an administrator from other entities (e.g., the entrance statistics server 1220, the database 1230, the access sensor control server 1240, etc.) And provides a function of checking and managing access statistics information, access detector information, store information, and the like by providing a web page as shown in FIG. 18 to a user or an administrator.

For example, the web server 1250 may store information on the number of visitors, the change in the number of visitors compared to the reference value, the number of departing guests, the change amount of the number of departed customers relative to the reference value, the current stay number, the average stay time, the maximum stay time, And can provide and provide.

Also, the web server 1250 can generate and provide information on the ratio of the attendance rate to the plurality of cameras corresponding to the plurality of access sensors, and information on the exit ratio.

In addition, the web server 1250 displays the information on the sex and the age of the visitor based on the information on the sex and age of the visitor (for example, information obtained by interlocking with the face detection information or in association with previously stored visitor identification information) And the age-specific attendance ratio and the like can be generated and provided.

In addition, the web server 1250 may provide an image photographed by each of a plurality of cameras corresponding to a plurality of access sensors in real time.

In addition, the web server 1250 can generate and provide detailed statistics on the number of visitors, the number of departed guests, the number of guests, the number of cumulative visitors, the number of cumulants removed, and the average residence time by time slot.

The matters described in the various embodiments of the present invention described above may be applied independently or two or more embodiments may be applied at the same time.

Although the exemplary methods described in the various embodiments of the invention described above are represented by a series of acts for clarity of illustration, they are not intended to limit the order in which the steps are performed, Or may be performed in different orders. In addition, not all illustrated steps are necessary to implement the method proposed by the present invention.

The scope of the present invention includes an apparatus for processing or implementing operations according to the methods proposed by the present invention.

The scope of the present invention includes software (or an operating system, an application, a firmware, a program, and the like) that causes an operation according to the present invention to be executed on a device or a computer, And includes an executable medium.

110 depth camera 120 image receiving unit
121 Coordinate transformation unit 122 Height map generation unit
130 Analysis object height map generation unit 140 Candidate area determination unit
150 Feature information extraction unit 160 Person &lt; RTI ID = 0.0 &gt;
170 Object tracking and counting

Claims (19)

A method for detecting a person using a depth image,
Obtaining one or more depth images from the one or more depth cameras;
Generating one analysis target height map using the one or more depth images;
Determining one or more candidate regions in the one analysis target height map;
Extracting feature information from each of the one or more candidate regions; And
Determining if a human object is detected in each of the one or more candidate regions based on the feature information,
Wherein the feature information includes information on an area according to a height level in one candidate region and an amount of change in an area according to the height level. The method according to claim 1,
Wherein the feature information includes a feature vector including elements representing the area occupied by coordinates belonging to each of a plurality of coordinate groups classified according to a height level. 3. The method of claim 2,
Wherein one element of the feature vector represents an area of a convex hull with respect to the coordinates belonging to one coordinate group corresponding to one height level. 3. The method of claim 2,
The final feature vector x to which the weight is applied to the area change amount between adjacent height levels is defined by the following equation,

s _i denotes an i-th element of the feature vector,
and? _k represents a weight for the k-th element of the final feature vector. 3. The method of claim 2,
The final feature vector x to which the weights are applied to the area ratio between the adjacent height levels and the image ratio of each area is defined by the following equation,

s _i denotes an i-th element of the feature vector,
alpha _k denotes a weight for the k-th element of the final feature vector,
r _i is a value obtained by dividing the length of the minor axis by the length of the major axis in an area corresponding to the i-th level, and has a value of more than 0 and less than 1. The method according to claim 1,
Wherein the feature information comprises information on an area according to one or more height levels corresponding to a head portion of a person and an area according to one or more height levels corresponding to a part of an upper half of the person from a shoulder of a person. The method according to claim 1,
Wherein the one analysis target height map is generated based on a height map generated for each of the one or more depth images. 8. The method of claim 7,
Wherein the one analysis target height map is generated by combining a plurality of height maps. The method according to claim 1,
Wherein a coordinate transformation is applied to each of the one or more depth images, and a height map is generated based on the depth image on which the coordinate transformation is performed. 10. The method of claim 9,
In the coordinate transformation,
The depth information of one depth image obtained from one depth camera is converted into three-dimensional coordinates of the camera reference coordinate system,
And converting the three-dimensional coordinates of the camera reference coordinate system into three-dimensional coordinates on a real world coordinate system. 11. The method of claim 10,
Wherein the conversion of the depth information of each pixel of the one depth image to the three-dimensional coordinates of the camera reference coordinate system is defined according to the following equation,

i represents a row index and a column index of a pixel of the depth image,
d represents a value of the depth information,
x , y , and z represent values on the X-axis, Y-axis, and Z-axis of the camera reference coordinate system,
ρ _hor , ρ _ver Represents a horizontal resolution and a vertical resolution of the one depth image,
and θ _hor and θ _ver represent the horizontal and vertical field of view (FOV) of the one depth camera, respectively. 11. The method of claim 10,
Wherein the conversion from the three-dimensional coordinates of the camera reference coordinate system to the three-dimensional coordinates on the real world coordinate system is performed,
The X-axis, the Y-axis, and the Z-axis of the camera reference coordinate system are rotationally transformed by?,?, And?
H,
And performing reflection transformation on the XY plane,
H denotes the height at which one camera is installed,
and? represents an angle at which one camera is installed in the X, Y, and Z axes of the camera reference coordinate system, respectively. The method according to claim 1,
Wherein each of the one or more candidate regions comprises one local peak. The method according to claim 1,
Wherein at least one of the installation position or the installation angle of each of the one or more depth cameras is adjustable. The method according to claim 1,
Wherein at least one of installation positions or installation angles of the plurality of depth cameras is different. The method according to claim 1,
Wherein the determination as to whether the human object is detected is based on whether the characteristic information matches the characteristic information of the human object determined using the classifier that has been learned in advance. An apparatus for detecting a person using a depth image,
An image receiving unit for acquiring one or more depth images from at least one depth camera;
An analysis object height map generation unit for generating an analysis object height map using the at least one depth image;
A candidate region determining unit for determining one or more candidate regions in the one analysis target height map;
A feature information extracting unit for extracting feature information from each of the one or more candidate regions; And
And a person judging section that judges whether a human object is detected in each of the one or more candidate regions based on the feature information,
Wherein the feature information includes information on an area according to a height level in one candidate region and an amount of change in an area according to the height level. A computer-readable medium having stored thereon software executable by an apparatus for detecting a person using a depth image,
The executable instructions causing the apparatus to obtain one or more depth images from one or more depth cameras; Generate one analysis target height map using the one or more depth images; Determine one or more candidate regions in the one analysis target height map; Extracting feature information from each of the one or more candidate regions; Determine, based on the feature information, whether a human object is detected in each of the one or more candidate regions,
Wherein the feature information comprises information on an area according to a height level in one candidate area and an amount of change in area according to the height level. A method for determining a person's access using a depth image,
Determining whether a human detection result exists in an nth frame of the depth image;
Determining if an existing path exists if a human detection result exists in the nth frame;
If the existing path exists, determining whether an addable path exists based on a time threshold and a spatial threshold;
Updating the existing path if an addable path exists; And
And determining the entry and exit of the person based on the start position and the end position of the updated route,
Wherein a difference between a last detection position of the existing path and a position of a human detection result in the nth frame is less than or equal to the spatial threshold and a difference between a last detection point of the existing path and a human detection point in the nth frame is less than or equal to the time threshold , It is determined that the addable path exists.

Images