KR102055276B1 - Method and Apparatus for Counting People - Google Patents

Abstract

Disclosed are a human counting device and method.
The human counting device and method for installing the camera in accordance with the characteristics of the place where the human counting is to be freed from the limitation of the installation of the camera that the main axis of the camera should be perpendicular to the floor, and minimizing the occlusion to provide.

Description

Person and Apparatus for Counting People

This embodiment relates to an apparatus and method for counting people by analyzing an image.

The contents described in this section merely provide background information on the present embodiment and do not constitute a prior art.

The conventional general counting system (People Counting System) is a way that when a visitor passes through a mechanical gate (Gate), a person manually controls the gate to count the visitor. In addition, a conventional counting system includes a counting method for detecting a person using an infrared sensor, a counting method for analyzing a temperature distribution of a person using a thermal image sensor, and a specific form after analyzing image information using an image processing technology. Alternatively, when a feature is detected, there is a counting method that is regarded as an event.

In general, counting the number of people entering a predetermined area is an important measure of the convergence ratio, which is one of the important factors of marketing. However, it is currently counting the number of people entering a specific area manually set. Performing the counting by hand, not only takes a lot of time to perform the counting with human eyes, but also generates a lot of labor costs, but there is a problem that can not guarantee the accuracy of the counting.

Conventional counting techniques for solving the above problems are to separate the foreground and background of the image by using the motion information of the object, and to determine the number and direction of the tracked objects by tracking the object that is determined to be a person in the separated foreground It was the way. Conventional human counting techniques operate reliably in the case of relatively low traffic or simple background. However, in a complex environment where a large number of people pass or a system that monitors a large area, many variables operate, which causes a problem of decreasing accuracy.

In order to solve the above-mentioned problems, there has been a method of analyzing and counting two-dimensional images obtained from a single camera as a conventional human counting technique. However, limitations in an outdoor environment have been revealed, such as recognizing a shadow as a human. Accordingly, recently, methods for counting people by analyzing three-dimensional images acquired from a 3D camera have been actively studied.

Conventional methods of analyzing and counting three-dimensional images have cameras mounted on the ceiling, etc. to minimize occlusion between people present in the images. By creating an overhead view image by making the principal axis of the camera perpendicular to the floor, the body part between people existing in the image is blocked or overlapped.

However, this method is environmentally limited to install the camera perpendicular to the floor.

Embodiments of the present invention, while the camera is installed in accordance with the characteristics of the place to be a person count to escape from the installation constraints of the camera to be installed so that the main axis of the camera perpendicular to the floor, it can minimize the overlap (Occlusion) The main purpose is to provide a person counting device and method.

According to an embodiment of the present invention, a three-dimensional model generator for generating a three-dimensional model from an image photographed using a three-dimensional (3D) camera, a tilt estimator for estimating the tilt of the 3D camera, the three-dimensional model A 3D model reconstructor which obtains a Top View based 3D model by rotating it according to the tilt of the 3D camera, and generates a 2D projection image by projecting the restored Top View based 3D model onto a horizontal plane And a counting unit for counting a person by detecting a person region using the 2D projection image.

According to an exemplary embodiment of the present invention, a process of generating a 3D model based on depth information obtained from one or more input images by combining with hardware, and obtaining sensing information from a tilt sensor mounted on a camera, based on a horizontal plane A process of estimating an inclined tilt of the camera, a process of obtaining a restored 3D model by rotating the 3D model to correspond to the tilt, and projecting the restored 3D model in a direction perpendicular to the horizontal plane Generating a two-dimensional projection image representing density information in a vertical direction on a coordinate system; and detecting and tracking a human region using the two-dimensional projection image, and counting a person passing through a predetermined region of interest. In order to provide a computer program stored in a recording medium.

As described above, according to the embodiments of the present invention, the camera is installed in accordance with the characteristics of the place where the human factor is to be freed from the limitation of the installation of the camera, which should be installed so that the main axis of the camera is perpendicular to the floor. It has the effect of minimizing occlusion and accurately counting people.

According to this embodiment, by using the sensor mounted on the camera to estimate the inclination of the camera, and by restoring the three-dimensional model according to the inclination of the camera to reduce the human detection error that can occur due to the tilting of the camera has an effect have.

According to the present embodiment, the image obtained from the camera is analyzed to more accurately count the number of people, which can be applied to the field of video security such as intrusion or abnormal situation detection and also applied to the field of business intelligence (BI). Retailers can quickly and accurately analyze the floating population and congestion in stores.

1 is an exemplary view for explaining a problem of the prior art.
2 is a schematic structural diagram of an image analyzing apparatus according to an embodiment of the present invention.
3 is a conceptual diagram illustrating a tilt estimation unit of an image analyzing apparatus according to an exemplary embodiment of the present invention.
4A to 4C are exemplary diagrams for describing a 3D model reconstructor and a projection unit of an image analyzing apparatus according to an exemplary embodiment of the present invention.
5 is a flowchart illustrating a person counting method performed by an image analyzing apparatus according to an exemplary embodiment of the present invention.

Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings.

1 is an exemplary view for explaining a problem of the prior art.

FIG. 1 (a) shows a case where the camera 10-1 is installed on the ceiling of a specific space so that the main axis is perpendicular to the floor surface. In such a case, occlusion between people existing in the image is small, so that each person can be detected and tracked relatively accurately.

However, the proper installation position of the camera may vary depending on the environmental characteristics of the space where the human factor is to be applied and the application field using the human factor. In some cases, it is more appropriate to install the camera at an angle to the bottom surface than to install it so that the main axis of the camera is perpendicular to the floor.

FIG. 1 (b) shows a case where the camera 10-2 is installed so that the main shaft is inclined with the bottom surface. Even in this case, if the same counting method as in FIG. 1 (a) is applied, occlusion between people in the image is increased, thereby reducing accuracy in detecting people from the image.

Embodiments of the present invention provide an image analysis apparatus and method for accurately counting people without installation restrictions that a camera must be installed to obtain a top view image. That is, an image analysis apparatus capable of counting a person robust to occlusion even in a slanted view image obtained by installing the camera obliquely from a ceiling or a wall, and Provide a method.

Hereinafter, an image analyzing apparatus 100 according to an exemplary embodiment of the present invention will be described in detail with reference to FIG. 2.

2 is a schematic structural diagram of an image analyzing apparatus 100 according to an embodiment of the present invention.

The image analyzing apparatus 100 according to the present exemplary embodiment includes a preprocessor 210, a detector 220, and a counter 230. The detector 220 includes a 3D model generator 222, a tilt estimator 224, a 3D model reconstructor 226, and a projection unit 228. However, the components included in the image analyzing apparatus 100 are not necessarily limited thereto, and may further include other components.

Each component included in the image analyzing apparatus 100 may be connected to a communication path connecting a software module or a hardware module inside the apparatus to operate organically with each other. These components communicate using one or more communication buses or signal lines.

In addition, although FIG. 2 illustrates that each component exists as a separate module, the present invention is not limited thereto and may be implemented as one integrated module including all of the functions.

The image analyzing apparatus 100 executes (i) a communication device such as a communication modem for performing communication with various devices or a wired / wireless network, (ii) a memory for storing various programs and data, and (iii) executes various programs for operation. And a microprocessor for controlling and the like. According to at least one embodiment, the memory may be a computer such as random access memory (RAM), read only memory (ROM), flash memory, optical disk, magnetic disk, solid state disk (SSD), or the like. It may be a readable recording / storage medium. According to at least one embodiment, the microprocessor may be programmed to selectively perform one or more of the operations and functions described in the specification. According to at least one embodiment, the microprocessor may be implemented as hardware, such as an Application Specific Integrated Circuit (ASIC), in whole or in part.

The preprocessor 210 performs preprocessing on one or more input images input from a camera (not shown). Here, the camera is a generic name for equipment capable of measuring three-dimensional distance information including a depth camera such as a stereo camera. Hereinafter, the preprocessor 210 will be described using the stereo camera as an example for convenience of description.

In detail, the preprocessing unit 210 includes noise reduction, rectification, calibration, color enhancement, color space conversion (CSC), and the like for stereo images. Interpolation, camera gain control, and the like can be performed. Accordingly, a clearer stereo image may be obtained than the stereo image photographed by the stereo camera.

The preprocessor 210 may obtain disparity information by performing stereo matching on the stereo images. In this case, the disparity information may refer to a map representing numerically parallax information of a stereo image, that is, left and right images.

The preprocessor 210 may separate the background and the foreground from at least one of the stereo images based on the disparity information.

The detector 220 detects a person from one or more input images received from the preprocessor 210. As described above, the detector 220 includes a 3D model generator 222, a tilt estimator 224, a 3D model reconstructor 226, and a projection unit 228. Hereinafter, each component is demonstrated concretely.

The 3D model generator 222 generates a 3D model based on depth information obtained from one or more input images. Here, the 3D model may be in the form of a set of a plurality of points belonging to the 3D coordinate system. The points P constituting the 3D model according to the present exemplary embodiment may be defined as X coordinates, Y coordinates, and Z coordinates in the 3D coordinate system, and may be expressed as shown in Equation 1, for example.

The tilt estimator 224 estimates the tilt of the camera by obtaining sensing information from a sensor mounted in the camera. This is to solve the problem of overlapping when the 3D model is projected on the 2D plane to detect the human area when the camera is installed at an angle rather than perpendicular to the ground. . That is, the inclination of the camera is estimated using a sensor to obtain a three-dimensional model corresponding to the top view image-based three-dimensional model by correcting and reconstructing the three-dimensional model as much as the inclination of the camera. In this embodiment, by using a three-dimensional model corresponding to the top-view image-based three-dimensional model, it is possible to improve the accuracy of human detection by generating a two-dimensional projection image minimized occlusion (Occlusion).

The sensor mounted on the camera may include, for example, at least one of a tilt sensor, an acceleration sensor, or a gyro sensor. However, the present invention is not limited thereto and may include various types of sensors capable of detecting the tilt of the camera. In the case of using the acceleration sensor, the sensing information may be acceleration values in the X, Y, and Z directions based on the gravity direction.

The tilt estimator 224 estimates the tilt of the camera using the sensing information. A tilt estimation method according to the present embodiment will be described with reference to FIG. 3.

3 is a conceptual diagram for describing a tilt estimator 224 of the image analysis apparatus 100 according to an exemplary embodiment.

는 X축 방향으로 카메라가 기울어진 각도,

는 Y축 방향으로 카메라가 기울어진 각도,

는 Z축 방향으로 카메라가 기울어진 각도를 의미한다.The tilt of the camera based on the X-axis, Y-axis, and Z-axis of the three-dimensional coordinate space where the camera is installed can be displayed as shown in FIG. 3 by the coordinate system (Xcam, Ycam, Zcam) of the camera body. here

Is the angle at which the camera is tilted in the x-axis direction,

Is the angle at which the camera is tilted in the Y-axis direction,

Is the angle of inclination of the camera in the Z-axis direction.

The tilt of the camera can be estimated according to [Equation 2].

,

,

는 카메라에 장착된 센서로부터 획득된 센싱정보로서 각각 X축, Y축, Z축 방향의 센싱값을 의미한다.here,

,

,

Is sensing information obtained from a sensor mounted in a camera, and means sensing values in X, Y, and Z-axis directions, respectively.

The 3D model restorer 226 rotates and restores the 3D model generated by the 3D model generator 222 to correspond to the camera tilt estimated by the tilt estimator 224. That is, the 3D model restoring unit 226 may restore the 3D model by rotating the 3D model by the angle in which the camera is inclined in the opposite direction to the camera tilt.

In detail, the 3D model reconstructor 226 generates a rotation matrix based on the sensing information, and converts the coordinates by applying the rotation matrix to each of the 3D coordinates of the plurality of points constituting the 3D model. Can be.

,

,

)에 기초한 회전 행렬(R)은 [수학식 3]과 같다.Camera tilt estimated by the tilt estimator 224 (

,

,

) Is based on Equation (3).

The 3D model P ′ reconstructed by the 3D model reconstructor 226 may be represented by Equation 4.

Here, P means [Equation 1] described above.

In an embodiment of the present invention, the 3D model generator 222 may generate a 3D model by scattering a plurality of points in a virtual 3D space by using depth information obtained from an input image. Since the three-dimensional model generated by the above process is obtained by an algorithm, there may be an error with the three-dimensional information in the view according to the camera tilt estimated by the sensing information.

,

,

)를 3차원 모델 복원 시 그대로 적용할 경우, 오차가 발생하여 탑 뷰(Top View)의 3차원 모델로 복원되지 않을 수 있다. 따라서 센싱정보를 기반으로 3차원 모델을 복원함에 있어서, 해당 오차를 최소화할 필요가 있다.In other words, the camera tilt (estimated from the tilt estimation unit 224)

,

,

) Is applied as it is when restoring the 3D model, an error may occur and may not be restored to the 3D model of the top view. Therefore, in restoring the 3D model based on the sensing information, it is necessary to minimize the corresponding error.

Accordingly, the 3D model reconstructor 226 may reconstruct the 3D model based on a previously stored error table related to error information of the camera tilt estimated by the tilt estimator 224 in response to the change in the tilt of the camera.

The error table according to the present embodiment may be defined through the following process. In fact, the camera is inclined at various angles (tilts) to generate three-dimensional models according to the angle, and the angles of the camera are estimated from the sensing information at the angle. Then, the error of the angle is checked by comparing the geometry information of the 3D model generated according to the angle with the angle of the camera estimated from the angle. The error table generated based on the error of the checked angle (tilt) may be stored in advance in the image analyzing apparatus 100.

When the camera tilt estimated based on the sensing information is corrected using an error table and applied to reconstructing the 3D model, it is possible to more accurately detect a person.

The projection unit 228 projects the three-dimensional model reconstructed by the three-dimensional model reconstructor 226 in a direction perpendicular to the horizontal plane (surface) to display two-dimensional density information in the vertical direction (Z-axis direction) on the three-dimensional coordinate system. Create a projected image.

Specifically, when the three-dimensional model is projected on the X-Y plane of the three-dimensional coordinate system, it can be seen that the cross-sections of the three-dimensional model parallel to the X-Y plane are accumulated and projected. Accordingly, an area in which projection points overlap may appear on the X-Y plane, and the overlapping area is denser than the surrounding area. That is, the two-dimensional projection image represents density information in the vertical direction (Z-axis direction) of the three-dimensional model. At this time, the higher the Z-axis height of the three-dimensional model, the greater the density in the two-dimensional projection image.

For example, the 2D projection image may be a 2D gray scale image in which the pixel brightness corresponds to the height. In this case, the higher the density of the projection points, the darker the pixel brightness, and the darker the pixel brightness, the greater the density of the plurality of points by the height in the Z-axis direction. In other words, the density information can be displayed as a gray level.

Hereinafter, a detailed description will be given with reference to FIGS. 4A to 4C.

4A to 4C are exemplary diagrams for describing a 3D model reconstructor and a projection unit of an image analyzing apparatus according to an exemplary embodiment of the present invention.

4A shows (a) the three-dimensional model and (b) the two-dimensional projection image when the main axis of the camera is perpendicular to the horizontal plane (surface). In general, the points corresponding to the human area in the 3D model may accumulate in the 2D projection and may appear darker than the surroundings in the 2D projection image. Therefore, the human area can be detected using the pixel brightness of the 2D projection image.

4B shows (a) the three-dimensional model and (b) the two-dimensional projection image when the main axis of the camera is inclined with respect to the horizontal plane. In this case, as in the embodiment of the present invention, the sensing information is used to directly project the 2D plane without restoring the 3D model. As shown in (b) of FIG. 4B, when the 3D model is projected onto the 2D plane without restoration, the human area may be incorrectly detected by the overlap between the objects.

4C illustrates (a) a three-dimensional model, (b) a restored three-dimensional model, and (c) a two-dimensional projection image when the main axis of the camera is inclined with respect to the horizontal plane. According to the present embodiment, the accuracy of the detection of the human region can be improved by restoring the 3D model according to the camera tilt estimated using the sensing information of the sensor mounted in the camera.

The counting unit 230 counts a person passing through the preset ROI by detecting and tracking a human area using the 2D projection image.

5 is a flowchart illustrating a person counting method performed by the image analyzing apparatus 100 according to an exemplary embodiment of the present invention.

The image analysis apparatus 100 generates a 3D model based on depth information obtained from at least one input image (S510). Here, the 3D model may be in the form of a set of a plurality of points belonging to the 3D coordinate system. The points P constituting the 3D model according to the present exemplary embodiment may be defined as X coordinates, Y coordinates, and Z coordinates in the 3D coordinate system, and may be expressed as shown in Equation 1 above.

In operation S520, sensing information is obtained from an inclination sensor mounted on the camera to estimate the tilt of the camera (S520). This is to solve the problem of overlapping when the 3D model is projected on the 2D plane to detect the human area when the camera is installed at an angle rather than perpendicular to the ground. . That is, the inclination of the camera is estimated using a sensor to obtain a three-dimensional model corresponding to the top view image-based three-dimensional model by correcting and reconstructing the three-dimensional model as much as the inclination of the camera. In this embodiment, by using a three-dimensional model corresponding to the top-view image-based three-dimensional model, it is possible to improve the accuracy of human detection by generating a two-dimensional projection image minimized occlusion (Occlusion).

The sensor mounted on the camera may include, for example, at least one of a tilt sensor, an acceleration sensor, or a gyro sensor. However, the present invention is not limited thereto and may include various types of sensors capable of detecting the tilt of the camera. In the case of using the acceleration sensor, the sensing information may be acceleration values in the X, Y, and Z directions based on the gravity direction.

A detailed method of estimating the camera tilt is as described with reference to FIG. 3.

Thereafter, the image analyzing apparatus 100 rotates and restores the 3D model generated in step S510 to correspond to the camera tilt estimated in step S520 (S530). That is, the 3D model may be restored by rotating the 3D model by the tilt angle of the camera in a direction opposite to the tilt of the camera.

In detail, a rotation matrix may be generated based on sensing information, and coordinates may be converted by applying a rotation matrix to each of three-dimensional coordinates of a plurality of points constituting the three-dimensional model. Detailed description thereof is as described above.

On the other hand, since the three-dimensional model generated in step S510 is obtained by the algorithm, there may be an error with the three-dimensional information in the view (View) according to the camera tilt estimated by the sensing information.

,

,

)를 3차원 모델 복원 시 그대로 적용할 경우, 오차가 발생하여 탑 뷰(Top View)의 3차원 모델로 복원되지 않을 수 있다. 따라서 센싱정보를 기반으로 3차원 모델을 복원함에 있어서, 해당 오차를 최소화할 필요가 있다.In other words, the camera tilt estimated in step S520 (

,

,

) Is applied as it is when restoring the 3D model, an error may occur and may not be restored to the 3D model of the top view. Therefore, in restoring the 3D model based on the sensing information, it is necessary to minimize the corresponding error.

Accordingly, the process S530 may include restoring the 3D model based on a previously stored error table related to the error information of the camera tilt estimated in response to the change in the tilt of the camera.

The error table according to the present embodiment may be defined through the following process. In fact, the camera is inclined at various angles (tilts) to generate three-dimensional models according to the angle, and the angles of the camera are estimated from the sensing information at the angle. Then, the error of the angle is checked by comparing the geometry information of the 3D model generated according to the angle with the angle of the camera estimated from the angle. The error table generated based on the error of the checked angle (tilt) may be stored in advance in the image analyzing apparatus 100.

As such, when the camera tilt estimated based on the sensing information is corrected using an error table and then applied to the 3D model reconstruction, it is possible to more accurately detect a person.

In step S540, the restored three-dimensional model is projected in a direction perpendicular to a horizontal plane (surface) to generate a two-dimensional projection image representing density information in a vertical direction (Z-axis direction) on the three-dimensional coordinate system.

Specifically, when the three-dimensional model is projected on the X-Y plane of the three-dimensional coordinate system, it can be seen that the cross-sections of the three-dimensional model parallel to the X-Y plane are accumulated and projected. Accordingly, an area in which projection points overlap may appear on the X-Y plane, and the overlapping area is denser than the surrounding area. That is, the two-dimensional projection image represents density information in the vertical direction (Z-axis direction) of the three-dimensional model. At this time, the higher the Z-axis height of the three-dimensional model, the greater the density in the two-dimensional projection image.

For example, the 2D projection image may be a 2D gray scale image in which the pixel brightness corresponds to the height. In this case, the higher the density of the projection points, the darker the pixel brightness, and the darker the pixel brightness, the greater the density of the plurality of points by the height in the Z-axis direction. In other words, the density information can be displayed as a gray level.

Using the two-dimensional projection image generated in step S540, the image analysis apparatus 100 detects and tracks a human area, and counts a person passing through a preset ROI (S550).

In FIG. 5, steps S510 to S550 are sequentially executed, but the present disclosure is not limited thereto. In other words, since the steps described in FIG. 5 may be applied by changing the execution or performing one or more steps in parallel, FIG. 5 is not limited to the time series order.

The person counting method according to the present embodiment described in FIG. 5 may be implemented in a program and recorded in a computer-readable recording medium. The computer-readable recording medium having recorded thereon a program for implementing the person counting method according to the present embodiment includes all kinds of recording devices storing data that can be read by a computer system.

The above description is merely illustrative of the technical idea of the present embodiment, and those skilled in the art to which the present embodiment belongs may make various modifications and changes without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment but to describe the present invention, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

As described above, this embodiment is a useful invention that is applied to the field of video security or business intelligence (BI) to generate an effect of accurately counting a person using a sensor mounted on a camera.

100: image analysis device 210: preprocessor
220: detection unit 222: 3D model generation unit
224: slope estimation unit 226: 3D model restoration unit
228 projection unit 230 counting unit

Claims (5)

A three-dimensional model generator for generating a three-dimensional model from an image photographed using a stereoscopic image (3D) camera;
A tilt estimator estimating a tilt of the 3D camera;
A 3D model reconstructing unit which obtains a Top View based 3D model by rotating the 3D model to correspond to the tilt of the 3D camera;
A projection unit configured to project the restored top view-based three-dimensional model to a horizontal plane to generate a two-dimensional projection image; And
Counting unit for counting people by detecting the human area using the two-dimensional projection image
Image analysis device comprising a. The method of claim 1,
The 3D model reconstruction unit,
And restore the 3D model by rotating the camera by an inclined angle so as to correspond to the inclination estimated by the inclination estimator. The method of claim 1,
The 3D model reconstruction unit,
And a reconstructed three-dimensional model based on a previously stored error table associated with error information of a camera tilt estimated by the tilt estimator in response to a tilt change of the camera. The method of claim 1,
The two-dimensional projection image,
And a plurality of points constituting the three-dimensional model represent the density of the plurality of points in the Z-axis direction on the three-dimensional coordinate system corresponding to the density of the projected points. Combined with hardware,
Generating a 3D model from an image photographed using a 3D camera;
Estimating the tilt of the 3D camera;
Obtaining a Top View based 3D model by rotating the 3D model to correspond to the tilt of the 3D camera;
Generating a 2D projection image by projecting the top view based 3D model onto a horizontal plane; And
Counting people by detecting a human area using the 2D projection image
A computer program stored on a recording medium to execute the program.

Images