KR101839827B1 - Smart monitoring system applied with recognition technic of characteristic information including face on long distance-moving object - Google Patents

Abstract

According to the present invention, an intelligent monitoring system comprises: an object information generating unit configured to generate moving line data of a detected objet by using a detection camera and remote image data and generate behavior pattern information of the object by using a location relation of the generated moving line data and a situation recognition area; an image control unit configured to control a tracking camera capable of PTZ driving to photograph a near image of the object when movement of the object is stopped; a receiving unit configured to receive near image data from the tracking camera; a face information detecting unit configured to detect face area information in the near image data; and a feature information generating unit configured to generate feature information including a gender, an age, and a worn article of the object by using the face area information.

Description

TECHNICAL FIELD [0001] The present invention relates to an intelligent surveillance system for recognizing facial feature information (age, gender, worn tool, face facial identification) for a remote dynamic object,

More particularly, the present invention relates to an intelligent monitoring system, and more particularly, to an intelligent monitoring system that more accurately detects and tracks an object moving at a long distance by organically fusing two or more detection tracking algorithms, If the motion pattern of the recognized dynamic object corresponds to a stop, it recognizes the behavior pattern of the dynamic object accurately by linking with the pattern, and performs the face recognition of the dynamic object more precisely using the high resolution image data of the object, And an intelligent surveillance system implemented so as to accurately recognize the age, sex, worn tool, facial identification, and the like.

In order to realize various purposes such as crime prevention, prevention of accidents, disaster monitoring, monitoring of buildings and facilities, identification of passengers using public transportation, parking, traffic monitoring, etc., CCTV and related systems or devices are installed , And these devices and systems are gradually increasing due to various social needs.

Such CCTV and related systems are used in various ways. Generally, a CCTV is installed in a plurality of locations in a specific area, and the CCTV is output to a screen display unit for controlling video data photographed from the plurality of CCTVs. Etc.) to monitor specific events while watching them, or to use images of various places or locations as a database and to use post-test evidence.

Conventionally, a CCTV or a control system has been applied to a method of tracking an object of interest in a photographed image and monitoring the movement of the tracked object using a video analysis (VS) technique, etc. Recently, When an abnormal behavior or an action of an object is detected, a method of transmitting alarming information to the manager is also used little by little.

Methods for detecting and tracking such objects include a method of detecting an object included in image data using an image difference technique, an optical flow technique, and the like, and further, a method of detecting an object included in the image data, (Kalman Filter) are used to track objects.

However, since the techniques related to object detection and tracking are designed to be processed by limited specific environment or feature parameters that can be detected and tracked by the corresponding technique, However, in situations or environments that are not suitable for these techniques, the efficiency of object detection and tracking is significantly degraded.

For this reason, the conventional techniques and methods are sensitive to the noise change of the external environment, so that the reliability of detection / tracking is degraded. Moreover, since it is difficult to continuously detect and track the object accurately, There is a limit.

Even if the dynamic object is detected or tracked, the conventional method is at a level of simply expressing the result to the manager or the like, so that the result is still applied in a human-dependent manner It is almost impossible to implement effective interfacing between system and person (administrator).

On the other hand, a method of detecting / recognizing a face in an object included in image data includes an algorithm based on skin color, an algorithm using motion generated in real-time image, and an algorithm using statistical cluster information.

However, since this method is an algorithm designed to be specialized under specific conditions, it is difficult to have versatility in various environments. Especially, when the image is not secured on the front face of the object such as the rotation of the object in the image data, There is a problem.

In addition, the face recognition algorithm of the conventional object is implemented only in an indoor environment in which a predetermined angle and a view are precisely secured in a specific place (for example, an airport entry and exit examination office, a building security desk, a police station, etc.) So that the utilization thereof is significantly restricted.

In addition, the conventional face recognition algorithm can not be integrated with a system for detecting and monitoring a specific object such as a distant object, and is implemented only as an independent driving method or apparatus of its own, thereby failing to implement an integrated object monitoring and management system .

The object of the present invention is to solve the above-mentioned problems, and it is an object of the present invention to provide a method and apparatus for detecting and tracking an object, It is possible to precisely and sustainably detect and track the object, as well as precisely define the motion, behavior, or movement pattern of the tracked object by a time series combination of predefined behavior patterns, The object of the present invention is to provide a system infrastructure that can more effectively implement subsequent utilization according to an action pattern of an object.

In addition, according to the present invention, it is possible to automatically acquire local high-resolution object image data when an object exhibits a specific behavior pattern and organically apply a more generalized face recognition method to the object, thereby improving the efficiency of face recognition of the object, The sex, age, wearing tool, facial identification, etc. of the patient.

Other objects and advantages of the present invention will become apparent from the following description, and it will be apparent from the description of the embodiments of the present invention. Further, the objects and advantages of the present invention can be realized by a combination of the constitution shown in the claims and the constitution thereof.

According to an aspect of the present invention, there is provided an intelligent surveillance system using facial feature information recognition for a remote dynamic object. The intelligent surveillance system generates motion data of a detected object using image data of a sensing camera, An object information generating unit for generating behavior pattern information of the object using a positional relationship with the object; An image controller for controlling a tracking camera capable of PTZF driving such that a near image of the object is captured when the movement of the object is stopped; An input unit for receiving the near vision image data from the tracking camera; And a face information detector for detecting face area information in the near vision image data; And a feature information generation unit for generating feature information including the age, wearing object, or facial identification information of the object using the face area information.

Here, the face information detection unit of the present invention may include a first detection unit for detecting first face region information by applying a haar feature-based algorithm to the near vision image data; A second detection unit for detecting second face area information by applying a skin color-based algorithm to the near vision image data; And a detection arithmetic unit for generating face area information on a hybrid area using the first and second face area information.

Also, the image control unit of the present invention may include a spatial information storage unit storing spatial model information including position information of the detection camera, position information of the tracking camera, characteristic information of the tracking camera, and three-dimensional position information of each point; A position information conversion unit for generating three-dimensional coordinates corresponding to MER center coordinates of an object whose movement is stopped using the spatial model information; And a transmitting unit for calculating a PTZ control value of the tracking camera so that the generated three-dimensional coordinate image is captured and transmitting the PTZ control value to the tracking camera.

Preferably, the present invention includes a feature information extracting unit for extracting a feature region of a face including nose, mouth, and two-eye region from the face region information; And a rotation information generator for generating length information of each of the nose and eye and length information between the nose and mouth and generating rotation direction and rotation angle information of the face using the length information, The feature information generating unit may further include rotation direction and rotation angle information of the face to generate the feature information.

More preferably, the present invention further includes an image restoration unit for symmetrically applying the opposite image region corresponding to the rotation direction and rotation angle information of the face to generate the restored face region information based on the front face as the face region information In this case, the feature information generating unit of the present invention is configured to generate feature information using the restored face region information.

In addition, the feature information generation unit of the present invention combines the presence or absence of each of the eye, nose, or mouth regions with the pixel characteristics of the eye region in the face region information to generate wear tool information worn on the face And generates skin crease histogram data for areas other than the eyes, nose, and mouth areas of the face area information according to the embodiment, generates texture data on the size or depth of the skin crease line using the generated skin crease line histogram data, And to generate age information of the object by comparing the data with the age-based reference texture data.

According to an embodiment of the present invention, when image data of the detection camera or the tracking camera is input, a human object is extracted from the input image data, and the MER region of the extracted human object is classified into a head, a chest, A coordinate setting unit for setting a coordinate having a maximum brightness value in the resultant data as a skeleton coordinate by applying a distance transformation algorithm to an ROI area; Estimating the position of the hand using the skeleton coordinates, designating the area around the hand centered on the skeleton coordinate mapped with the position of the estimated hand, detecting the outline of the area around the designated hand, An object zone generating unit for generating an object zone which is an excluded zone; And an object information generating unit for generating object information included in the object area by comparing feature information of the object area with reference information for object determination.

Preferably, the feature information generating unit of the present invention divides the face area and the hair area in the face area information, and generates the feature information on the sex of the object using the vertical ratio of the divided face area and the hair area .

The object information generating unit of the present invention may further include: a receiving unit that receives image data from n (n is a natural number equal to or greater than 1) detecting cameras; A first detection unit for detecting an object by applying a difference image detection algorithm to the image data and generating first candidate MER information of the detected object; A second detecting unit detecting an object by applying an optical flow algorithm to the image data and generating second candidate MER information of the detected object; A detection determination unit for generating final MER information of the object by applying a hog algorithm to the first and second candidate MER information; A tracking unit for tracking movement of final MER information of the object to generate copper line data of the object; And a behavior pattern determiner for determining a behavior pattern of the object by using a positional relationship between the context aware area as a basis of the behavior pattern analysis and the copper line data of the object.

Here, the first detection unit of the present invention is configured to generate a background image every frame using statistical images accumulated for a predetermined time and to generate the first candidate MER information by a difference between the generated background image and a current image .

The behavior pattern determination unit of the present invention may further include an event generation unit for generating event information of an object using the relationship between the coordinates of the final MER center point position of the object of each time of the copper line data and the position coordinates of the boundary line of the context- ; And a pattern information generating unit for generating behavior pattern information for the object using the time-series characteristic of the event information of the object.

The intelligent surveillance system according to the present invention is configured to be organically combined with processing for detecting, tracking, and generating behavior patterns of an object, so that when the object corresponds to a specific behavior pattern, the local high resolution image data is automatically generated It is possible to realize a more integrated object monitoring system.

The intelligent monitoring system according to the present invention can enhance the versatility of face recognition by using a plurality of face recognition algorithms in a binary manner and using the generated recognition areas integrally for each result, have.

According to another embodiment of the present invention, the rotation angle information about the face region of the object is accurately generated, and the face region that is not secured by the image can be estimated and restored using the rotation angle information, Can be implemented.

The intelligent monitoring system according to the present invention processes object detection algorithms in a binary manner and processes the object detection algorithms organically by using a third algorithm that is optimized for intra- By hierarchically applying it, it is possible to adapt to the noise of the external environment, to further improve the probability of object detection, and to increase the computation efficiency.

Further, according to another embodiment of the present invention, a Kalman filter algorithm and a camshit algorithm are organically combined using an object tracking algorithm, and the process of using the result of the camshit algorithm as a parameter of the Kalman filter algorithm is repeated It is possible to improve the accuracy and reliability of the object tracking and to further optimize the tracking of the dynamic object.

According to still another embodiment of the present invention, dynamic data of a dynamic object, that is, dynamic characteristics or behavioral characteristics generated by a dynamic object, may be classified into an event of the predefined object and a behavior pattern Information can be precisely defined and data can be generated. Therefore, it is possible to create an effect of precisely interfacing the subsequent processing to the user as well as the behavior pattern of the dynamic object.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate preferred embodiments of the invention and, together with the detailed description of the invention given below, serve to better understand the technical idea of the invention, And shall not be construed as limited to such matters.
1 is a block diagram showing a detailed configuration of an intelligent monitoring system according to a preferred embodiment of the present invention;
FIG. 2 is a block diagram showing the detailed configuration of the face information detecting unit of the present invention shown in FIG. 1;
3 is a flow chart illustrating a process for an overall processing method of the present invention,
FIG. 4 is a flowchart illustrating a face recognition processing according to an exemplary embodiment of the present invention. FIG.
FIG. 5 is a flowchart illustrating a process for an example of the present invention for processing for estimating and recognizing the age of an object;
FIG. 6 is a block diagram showing a detailed configuration of the object information generating unit of the present invention shown in FIG. 1;
FIG. 7 is a block diagram showing the detailed configuration of the behavior pattern determining unit of FIG. 6;
FIG. 8 and FIG. 9 are flowcharts illustrating a detailed processing procedure of object detection according to a preferred embodiment of the present invention;
FIG. 10 is a flowchart illustrating a detailed process of object tracking according to an exemplary embodiment of the present invention.
FIG. 11 illustrates an embodiment of an image recognition area and a context recognition area according to the present invention.
12 is a view for explaining a characteristic of the HOG algorithm used for object detection,
13 is a view for defining six event information of an object according to an exemplary embodiment of the present invention;
14 is a view for explaining eight behavior pattern information according to a preferred embodiment of the present invention,
FIG. 15 is a flowchart illustrating a processing procedure according to an exemplary embodiment of the present invention for generating information on objects existing in a hand region of a human object; FIG.
FIG. 16 is a diagram for explaining the processing of FIG. 15; FIG.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, terms and words used in the present specification and claims should not be construed as limited to ordinary or dictionary terms, and the inventor should appropriately interpret the concepts of the terms appropriately It should be interpreted in accordance with the meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined.

Therefore, the embodiments described in the present specification and the configurations shown in the drawings are merely the most preferred embodiments of the present invention and are not intended to represent all of the technical ideas of the present invention. Therefore, various equivalents It should be understood that water and variations may be present.

FIG. 1 is a block diagram illustrating a detailed configuration of an intelligent monitoring system (hereinafter referred to as a monitoring system) 1000 to which a face feature information recognition method for a remote dynamic object according to an exemplary embodiment of the present invention is applied.

1, the monitoring system 1000 includes an object information generating unit 100, an image controlling unit 200, an input unit 300, a face information detecting unit 400, a feature information generating unit 500, A feature information extracting unit 600, a rotation information generating unit 700, and an image restoring unit 800.

Prior to the description of the present invention, the monitoring system 100 of the present invention shown in Fig. 1 and each component of the present invention shown in Figs. 2, 6 and 7 are logically divided rather than physically separated components It should be understood as a component.

That is, since each constitution corresponds to a logical constituent element for realizing the technical idea of the present invention, even if each constituent element is constituted integrally or separately, if the function performed by the logical constitution of the present invention can be realized, It is to be understood that any component that performs the same or similar function should be interpreted as falling within the scope of the present invention regardless of the consistency of the name.

The monitoring system 100 of the present invention accurately detects the object of interest in the image data by raw-data of the image data photographed by one or more detection cameras 50 And tracking the movement and behavior characteristics of the detected object to generate motion data or behavior pattern information of the object and perform face recognition of the object using the generated motion data or behavior pattern information.

As shown in FIG. 1, the monitoring system 100 of the present invention can be implemented in a hardware configuration such as a system device, a server, or the like, as well as a detection camera 50, a zoom camera (a tracking camera / ) 70 and the like, and may be embodied in the form of software for carrying out the processing of the present invention which will be described later.

3 is a block diagram illustrating the structure of a semiconductor memory device according to an embodiment of the present invention. Referring to FIG.

As shown in FIG. 3, when image data is received from the detection camera 50, the image data is parsed or analyzed to detect an object of interest in the image data (S300).

 An object of interest is an object of interest or an object of interest, such as a person, an animal, or a vehicle. Various methods such as an image difference algorithm, an optical flow algorithm, and a model matching algorithm may be applied as methods for extracting objects from image data, including binarization, labeling processing, and edge detection.

The method of representing the detection result of the object uses a minimum enclosing rectangle (MER), which is referred to as a minimum adjacent rectangle or a minimum containing rectangle. Specifically, the width of the MER, Height, center point coordinates, density information, and width / height ratio information, and a method of detecting / distinguishing a specific object of interest by comparing the characteristics of the MER and the width and height of the object included in the image data .

When the detection of the object is completed, processing for tracking the detected object proceeds (S310). Tracking of the detected object corresponds to the process of generating path data, behavioral characteristic information, and behavioral pattern information of the object as to which direction the detected object moves to.

When the tracking of the object is completed, the behavior pattern information of the object is generated using the mutual positional relationship between the movement data of the created object and the predetermined context-aware area (S320).

As shown in FIG. 11 (a), in the image generation area, which is an area generated by the camera 50, an area that does not need to monitor a human object or the like It is necessary to reduce the amount of computation by excluding the region in which the detection of the object is not required in the image generation region, as shown in Fig. 11 (a).

Furthermore, the area set for the purpose of automatically recognizing the situation of the system (apparatus) as the area included in the image processing area becomes the situation recognition area (FIG. 11 (b)) without user intervention.

When the situation recognition area is set, position coordinates such as the boundary (outline) and center point coordinates of the situation recognition area (image generation area) are generated. Based on the position coordinates of the situation recognition area, The center point coordinate information is compared with the position coordinates of the situation aware area to generate information on the behavior characteristics or behavior patterns of the object.

When the behavior pattern information of the dynamic object is generated, the subsequent processing according to the generated behavior pattern proceeds (S330). When the behavior pattern of an object is defined as an intrusion, processing such as sending information to an administrator or a related organization (such as a police station), or automatically activating a warning light, a siren, or the like installed at a specific location is a part of such subsequent processing.

If it is determined that the behavior pattern of the dynamic object is " paused " (S340), that is, if it is determined that the velocity of the object is 0 (or close to 0) (Pan, Tilt, Zoom, and Focal length) of the tracking camera 70 installed in the tracking camera 70 to capture a high-resolution zoom image at a position where the object is stopped, Sex, wearing ornaments, age range, etc.) (S350). Various information related to the object generated in the processing, history information, and the like may be stored together with the time information (S360).

Hereinafter, the configuration and processing of the present invention for a face recognition process in which a behavior pattern of an object is stopped, that is, when a moving object stops, will be described first with reference to FIGS. 4 and 5, Details of the object detection processing during processing, the tracking processing of the detected object, the behavior pattern generation processing of the object, etc. will be described later.

The object information generating unit 100 of the present invention detects the object using the remote image data when the sensing camera 50 generates the remote image data, generates the moving object data of the object by tracking the detected object, And generates event information and behavior pattern information of the object as shown in FIGS. 13 and 14 by organically applying the positional relationship between the copper line data and the context-aware area. Detailed processing and specific configuration performed by the object information generation unit 100 of the present invention will be described later.

In the case where the motion pattern of the generated object is the stop motion pattern, that is, when the movement of the object is stopped, the image control unit 200 of the present invention is a zoom camera capable of driving PTZF (Pan, Tilt, Zoom, Focal length) Tracking camera) 70 so that a close-up image (zoomed image / high-resolution image) of the object whose movement is stopped is photographed.

Specifically, the image control unit 200 of the present invention may include a spatial information storage unit 210, a location information conversion unit 220, and a transmission unit 230.

The spatial information storage unit 210 stores the position information of the detection camera 50 and the tracking camera 70 as well as the characteristic information PTZF of each camera. And spatial model information including three-dimensional position information of each of the points included therein is stored.

When the movement of the object is stopped, that is, when the behavior pattern of the object corresponds to " stop " (S600), the position information conversion unit 220 of the present invention extracts the coordinates of the center point of the MER of the object, Dimensional coordinates corresponding to the MER center point of the object whose movement has been stopped using the information stored in the spatial information storage unit 210 (S610 ).

When the three-dimensional coordinate is generated, the transmitting unit 230 of the present invention transmits P, T, Z, and F information of the tracking camera 70 corresponding to the generated three-dimensional coordinates so that a high- (S620) and transmits the generated PTZF control information to the tracking camera 70 (S630).

Preferably, the PTZF value is determined such that the upper half of the object, which is 1/3 of the total height of the MER, is photographed by calculating the width or height of the MER of the object.

When the PTZF value is transmitted to the tracking camera 70, the tracking camera 70 is driven according to the transmitted PTZF value to capture a near-field image (high-resolution zoom image) of the object (S640) and transmits the photographed near-field image data (S650).

When the upper half region including the face region of the object does not appear in the local image data generated by the tracking camera 70, the PTZF value of the tracking camera 70 is corrected and updated by performing cyclic processing, It is preferable that the re-photographing of the image is performed.

Thus, the present invention does not use the image data that includes the object, but rather uses a camera having a relatively wide angle of view for detecting and tracking the object by binarizing the camera device, thereby detecting the omni-directional object When the object stops moving, a high resolution clear image can be obtained for the object by using a camera having a relatively high resolution and a variable PTZF.

When the local image data of the object is input through the input unit 300 through the processing as described above, the face information detection unit 400 of the present invention detects the face region information in the transmitted local image data at step S660.

Various known methods can be applied to the method of detecting face area information. However, when a single detection algorithm is selectively used, the reliability of detecting face area information may be degraded due to problems such as being limited to a specific environment.

The present invention overcomes such a problem and is implemented to have higher adaptability and versatility in an outdoor environment. 2, the face information detection unit 400 includes a first detection unit 410, a second detection unit 420, and a detection calculation unit 430 .

Specifically, when the near vision image data is received through the input unit 300, the first detection unit 410 of the present invention applies the face area detection algorithm based on the haar feature to the near vision image data, (S661).

In parallel with this, the second detection unit 420 of the present invention detects the second face area information by applying the skill color based face area detection algorithm to the near vision image data (S662).

When the first face area information and the second face area information are detected, the detection operation unit 430 of the present invention generates a hybrid area using the first and second face area information, and the face area information (S663).

When the face area information is detected or extracted, the feature information generating unit 500 of the present invention parses the detected face area information to determine the sex, age or face of the object (wearing object) And generates characteristic information including identification information (S670).

Gender discrimination processing of the object can be realized by dividing the facial region and the hair region again in the face region information, and comparing the height ratio of the divided hair region and the face region, thereby recognizing the sex. And generates characteristic information on the sex of the object.

 The characteristic information generating unit 500 of the present invention sequentially or concurrently detects an eye, a nose, or an mouth area, which is a facial body organ, from facial area information as shown in Table 1 below, The organ of the facial organs detected and the undetected organs can be applied in combination to generate the wearing tool information worn by the object (S670).

Further, by determining whether or not an object other than the body organs (eye) is detected around the eye by using combination of the detection (search / detection) of the eye and the pixel characteristic of the area around the eye, Can be configured to distinguish between sunglasses and glasses.

In addition, the feature information generating unit 500 of the present invention can generate age information of the object.

5, when the face area information is input (S500), the feature information generating unit 500 performs image size / brightness normalization processing on the inputted face area information, processing for matching the center of the image , LBP (Local Binary Pattern) conversion processing, and the like (S510).

Through such processing, histogram data of the pixel values of the area other than the eye, nose, and mouth areas (such as the forehead, the forehead, or the neck) Texture data on the size or depth of the line is generated (S520)

This texture data means distribution data on the depth, size, and the like of the groove line representing the wrinkles on the forehead, forehead or neck, and performs processing for matching the texture data with reference texture data DB S530), and may generate the age (age) information of the corresponding object (S540, S670).

More preferably, the monitoring system 1000 of the present invention may further include a feature information extracting unit 600 and a rotation information generating unit 700. The rotation of the face of the face region information through the processing of these configurations Direction and / or rotation angle information may be additionally generated and utilized.

Specifically, the feature information extracting unit 600 of the present invention extracts regions of the nose, mouth, and eyes (both eyes) from the inputted face region information, and the rotation information generating unit 700 extracts coordinates of the center of gravity of the extracted regions Setting. Assuming the line connecting the left eye, the right eye, and the mouth with respect to the center of gravity of the nose, these segments will assume a Y shape.

Then, the separation distance between the mouth, the left eye, and the right eye is calculated based on the coordinates of the center of gravity of the nose. As described above, when the distance D1 between the nose and the left eye, the distance D2 between the nose and the right eye, and the distance D3 between the nose and mouth are calculated, And the size information of the rotation angle can be estimated.

When the rotation direction or rotation angle information is generated as described above, the feature information generation unit 500 of the present invention can generate the feature information on the object by further including information about the rotation direction and the rotation angle of the face.

When the rotation direction information and the rotation angle information of the face are generated by the rotation information generation unit 700 of the present invention, the image restoration unit 800 of the present invention generates the rotation direction and rotation angle information corresponding to the rotation direction and the rotation angle The opposite face image region is symmetrically applied to generate the input face region information and the restored face region information based on the front face.

When the image reconstruction unit 800 of the present invention generates restored face region information, the feature information generation unit 500 of the present invention can generate feature information of the object using the restored face region information.

The monitoring system 1000 of the present invention may further include a coordinate setting unit 810, an object zone generating unit 820 and an object information generating unit 830 as shown in FIG.

The coordinate setting unit 801 of the present invention generates a background image from the input image data when the image data (FIG. 16A) is inputted from the detection camera 50 or the tracking camera 70 (S800) The generated background image is used to separate the background image and the foreground image from the input image data (S805).

When the foreground image is separated, it is detected whether or not the object is included in the separated foreground image. When the object is detected in the foreground image (FIG. 16B), the MER of the detected object is extracted, The feature information including height, center point coordinates (a, b), density information (implication ratio of the number of constituent pixels of the object with respect to the entire width), width / height ratio information, and the like are calculated (S810).

An object corresponding to the extracted MER is classified into a person, a vehicle, or the like by using the feature information.

When the object corresponding to the extracted MER is human (human object), the coordinate setting unit 810 of the present invention divides the MER area of the extracted human object into head, chest, lower body and leg area (S815) (C) of FIG.

Since the head, the chest, the lower body, and the leg have a certain range of ratios based on the upright person, the MER area of the human object can be divided into four areas using the ratio of this specific range. For example, the ratio of the upper body to the lower body of a human being can be represented by a ratio of 1: 1.618, and in the corresponding view, the head, the chest, the lower body and the leg are 25/13, 40/13, 64/13, 40/13 .

When the ROI (Region of Interest) is generated, the ROI is applied to each ROI (S820).

The distance conversion algorithm is a matrix having a closest distance from a current pixel to a pixel having a value of 0. As shown in FIG. 16 (d), for example, when a distance conversion algorithm is applied to a hand area , An image in which the size value (brightness value) of each pixel is different is generated, and the coordinate having the maximum size value (brightness value) may be set as the center coordinate of the hand region.

The coordinate setting unit 810 of the present invention sets a skeleton coordinate which can represent the coordinates having the maximum brightness value as the center of each body part in the resultant data obtained by applying the distance transformation algorithm to each ROI (S825). In FIG. 16 (e), points representing the center point of each body part correspond directly to the skeleton coordinates.

When the skeleton coordinates are generated as described above, the object region generating unit 820 of the present invention estimates the position of the hand based on the skeleton coordinates and the general bone structure of a person (S830), and sets the skeleton coordinates mapped with the estimated hand position (Area around the hand) (S835) (area indicated by a red color in (e) of Fig. 16).

Then, the object region generating unit 820 of the present invention detects an outline (edge) of the corresponding region by applying a Canny operation, a Sobel operation, or the like to the designated hand peripheral region (S840) f)).

Since the center color of the skeleton coordinates may typically include the skin color of a person, this information can be used to exclude the area of the hand itself from the area around the hand, The object region in the hand can be generated based on the information of the outline (edge) of the area (S845) (the upper portion of Fig. 16 (g)).

When the object region is generated as described above, the object information generating unit 830 of the present invention performs processing for comparing feature information such as edge information, color, and local pattern of the object region with reference feature information for determining the object (S850) (See FIG. 16 (g)), object information included in the object zone is generated (S855).

Hereinafter, the configuration and processing of the object information generation unit 100 of the present invention for performing object detection, object tracking, and behavior pattern information generation of objects, etc. will be described in detail with reference to FIGS. 6 to 8 attached hereto. FIG. 6 is a block diagram showing a detailed configuration of the object information generating unit 100 of the present invention 1000 shown in FIG. 1. FIG. 8 and FIG. 9 are diagrams for explaining the object detecting method according to a preferred embodiment of the present invention. Fig. 3 is a flowchart showing a detailed processing procedure. Fig.

6, the object information generating unit 100 includes a receiving unit 110, a first detecting unit 130, a second detecting unit 140, a detection determining unit 140, (150), a tracking unit (170), and a behavior pattern determination unit (190).

First, as shown in FIG. 6, the receiving unit 110 of the present invention is a receiving unit 110 that receives signals from n detection cameras 50-1, 50-2, ..., 50- The video data is received through a wired or wireless communication network (S400).

Since the object information generating unit 100 is focused on the detection and tracking of the object itself, the detection camera 50 can detect a view angle such that a remote area can be photographed rather than a kind of zoom camera, It is preferable that a wide camera is suitable and that the detection camera 50 is implemented as a plurality of images for each direction so that the image of the omnidirectional region can be captured.

It is needless to say that image data of a specific area generated by a plurality of cameras covering a single or specific direction other than the omnidirectional area may be used according to the embodiment.

If the sensing camera 50 transmits the image data in this way, the first sensing unit 130 of the present invention applies the differential image sensing algorithm robust to the external environmental noise to the transmitted image data, And generates first candidate MER information of the detected object (S420).

The second detection unit 140 detects an object by applying an optical flow algorithm to the transmitted image data independently of the processing of the first detection unit 130, 2 candidate MER information (S430).

As mentioned above, the MER information is information on the minimum rectangular area in which the object is included, and includes width, height, center point coordinates (a, b), density information (Implication) ratio), width / height ratio information, and the like.

Since there may be a plurality of objects in one image data and a plurality of objects estimated as objects in the image data may exist for each algorithm, the first candidate MER information or the second candidate MER information may include a plurality of MERs Lt; / RTI > may be a collection of datasets related to information about < RTI ID = 0.0 >

In this case, when a plurality of image data are input from the plurality of detection cameras 50 in order to more effectively detect an object in a further enlarged area, the foreground image unit 120 of the present invention merges merged to generate panorama image data (S405).

In order to improve the efficiency of the image processing as described above, the image data generated by the camera 50 may be moved (S410) to reduce the image data in consideration of the area where the object can move .

When the panoramic image data is generated as described above, the first detection unit 130 and the second detection unit 140 of the present invention generate first and second candidate MER information for the panorama image data, respectively.

(2) an optical flow algorithm that computes an optical flow velocity vector between a previous image and a current image, and (3) And a method of using a minutiae point information or a method of comparing a direction pattern between pixels in a set search area, and a model matching method of generating and utilizing a model for detection.

The image difference algorithm can be subdivided into a background image difference algorithm, a previous image difference algorithm, or an adaptive background image difference algorithm to the noise change of the external environment. The optical flow algorithm includes a horn & Schunk algorithm, a block matching Matching algorithm, the Lucas & Kanade algorithm, or the Pyramid Lucas & Kanade algorithm.

The model matching method can be subdivided into a shift algorithm, a Surf algorithm, or a Histogram (Hogg of Oriented Gradient) algorithm.

These methods are designed to be optimized for each specific environment and parameters, and thus can be used for various purposes such as hardware resource usage, computation speed, ease of logic implementation, noise removal performance, rapid environmental change adaptability, , There is a considerable variation in the detection performance of each method depending on the sudden change of the illumination environment, the sudden change of the illumination environment, and the shadow effect.

The present invention is configured to maximize detection efficiency by configuring the detection algorithms to be hybridized so as to optimize the disadvantages and advantages of such conventional detection techniques and to organically combine them.

For this purpose, the first and second detection units 130 and 140 of the present invention are adapted to apply detection algorithms corresponding to different categories, respectively. First, in order to increase the probability of object detection, a difference image detection algorithm And generate first candidate MER information using a difference image detection algorithm robust against external environment noise and second candidate MER information using an optical flow algorithm, respectively.

Specifically, the first detection unit 130 of the present invention performs processing S421 of updating / generating a background image every frame using statistical images accumulated for a predetermined time, And generates at least one first candidate MER information through processing (S423).

The second detection unit 140 calculates an optical flow velocity vector between the two images (the previous image and the current image) based on the difference caused by the relative movement of the camera and the object.

Specifically, the second detection unit 140 detects an object by processing an optical flow algorithm distinguished from the image difference algorithm. In order to improve the probability and efficiency of object detection, the second detection unit 140 constructs an image pyramid according to the scale of the optical flow algorithm A pyramid Lucas and Canard algorithm, which is an algorithm for calculating an optical flow velocity vector between a previous image and a current image, is processed (S431) and one or more second candidate MER information is generated using the window matching method (S433) .

When the first candidate MER information and the second candidate MER information are generated by applying the different or biased algorithm in this way, the detection determination unit 150 of the present invention determines the first candidate MER information and the second candidate MER information The final MER information of the object is generated by applying a hog algorithm, which is a third algorithm for object tracking (S440).

Specifically, the detection determination unit 150 of the present invention determines whether or not one or more MER information items belonging to the first candidate MER information item and one or more MER information items belonging to the second candidate MER information item are mutually positioned The MER information having a lower deviation of the reference deviation is selected and a hybrid area composed of only the selected MER information is calculated (S441).

When the hybrid region is calculated as described above, the detection determination unit 150 of the present invention performs HOG algorithm processing on the hybrid region (S443) and generates final MER information of the object (S445).

12, a hog operation is performed by dividing an image into blocks, dividing a block into cells, and then using a histogram of feature vectors of each cell, Since the method of detecting an object by calculating a direction pattern between pixels in the area, the search area is limited to the hybrid area, and the calculation efficiency can be further improved.

When the final MER information is generated, the final MER information about the object is compared with the predetermined characteristic information (width, height, center point coordinate, density, width / height ratio, etc.) (S450). The object corresponding to the final MER is classified into a person (S460) or another specific object (vehicle, animal, bag, etc.) according to the result (S470).

In order to further improve the reliability of object detection by more effectively filtering the noise due to the external environment, the second detection unit 140 of the present invention determines an external environmental noise factor as shown in FIG. 9 according to the embodiment (S431 It is determined whether the generated external environmental noise is caused by the shaking of the detection camera (S432).

In this case, the first candidate MER information of the object and the second candidate MER information of the object are used as the difference between the detection camera shake and the second candidate MER information because the difference image detection algorithm is relatively difficult to generate accurate MER information. It is preferable that candidate candidates are calculated (S434) only for the second candidate MER information generated by the optical flow algorithm having a relatively small influence due to the influence of the second candidate MER information.

On the other hand, if it is not the external environment noise due to the shaking of the detection camera or the like, the candidate region is calculated on the basis of the first candidate MER information and the second candidate MER information as described above (S433).

As described above, the present invention utilizes a single algorithm for object detection or an object detection algorithm that is implemented in a different manner, rather than simply using a plurality of algorithms in parallel, in an organic and hierarchical manner The probability of object detection can be further improved, and the computational efficiency can be increased.

Hereinafter, object detection processing (S310) implemented through the configuration of the present invention will be described in detail with reference to FIG. 10, which is a flowchart illustrating a detailed processing process of object tracking according to the present invention.

When the processing for detecting the object is completed by generating the final MER information of the object as described above, the tracking unit 170 of the present invention tracks the movement of the final MER information of the object, Movement path data) (S310).

 The algorithm for tracking an object is to specify a target area of the tracked object, obtain a histogram corresponding to the color distribution characteristic of the designated area, and then calculate the histogram of the area corresponding to the target area size in the continuous input image A Cam-Shift algorithm for tracking an object by actively adjusting the size of a search window to solve problems caused by using a fixed-size search window, an error (P) And a Kalman-filter algorithm for tracking the object by predicting future information (x '& P`) based on the current system measurement value (x) and observation value (z).

These object tracking algorithms have different characteristics according to hardware resource, easiness of logic implementation, and computation amount. However, since MinShitP algorithm does not control the size of search window during object tracking, There is a problem that object tracking is no longer valid since the object is lost.

In the case of the camshift algorithm, the size of the search window is designed to be actively adjusted so that it can adapt to the environment where the size of the object changes. However, since the object can not accurately reflect the moving direction or speed of the object, The object can not be traced.

The Kalman filter algorithm can be applied when there is a stochastic error in the measured value of an object (object), and when the state of the object at a certain point in time has a linear relationship with the state at the previous point of time. And each state vector is related only to the vectors of the previous time.

In the case of the Kalmar filter algorithm, it is a method to track the object by predicting the future information based on the measured value and the observed value, and it has the characteristic feature that tracking of the moving object can be optimized because it can reflect the moving direction or speed of the object during tracking However, since the observation value is required for the prediction of the next state, it can be said that there is a difficulty in applying the observation value always.

The present invention uses a Kalman filter algorithm for predicting future information using current measurement data and observation data for the final MER information among the object tracking algorithms as a basic object tracking algorithm to track a moving object, In the Kalman filter algorithm, the observation data required for the next state prediction is calculated by applying the cam shift algorithm to the final MER information, and the process is applied cyclically, And to perform effectively.

That is, as shown in FIG. 10, the tracking unit 170 of the present invention is configured such that the pre-processing (S500), the processing by the Kalman filter algorithm (S510) and the processing by the camshit algorithm (S550) (Step S310). The movement data of the object is generated by tracking the movement of the final MER information of the object by performing the circulating processing.

Specifically, the tracking unit 170 of the present invention first initializes a Kalman filter parameter for performing a Kalman filter algorithm (S501), and sets a color space of a basic window area set as an area corresponding to the final MER information of the object as an HSV color (S502).

The HSV color space corresponds to a color space or color model representing a color based on hue, saturation, and brightness as one of the methods of expressing colors such as an RGB color space and a CMY color space .

After changing or changing the color space, the histogram of the base window area is calculated using the HSV data (S503).

When the pre-processing (S500) is completed, the tracking unit 170 of the present invention calculates the parameters (state vector, measured value, state matrix, system noise (Kalmar error), measurement noise (Step S511). In step S511, the state vector of the center of the object is predicted.

Thereafter, the search window is set to have the same width and height as the center and base window corresponding to the position of the predicted state vector (S512).

 If the center of the object is searched, the search window is located at the center of the searched object and the region is searched (S552) .

If it is determined that the object to be searched is the object to be searched using the correspondence between the histogram calculated in the searched region and the histogram calculated in the pre-processing (S503) (S553), the result of the calculated histogram The rotation angle and the center coordinates of the object are calculated (S555).

The rotation angle and center coordinate information of the calculated object are used as observation values used for prediction of future information in the Kalman filter.

When the rotation angle of the object and the center coordinate information (observation value information) are calculated as described above, the tracking unit 170 of the present invention updates the state vector using the calculated rotation angle and center coordinate information (S513).

When the state vector is updated, the control vector (object rate) of the Kalman filter algorithm is calculated (S514) and returns to the initial stage of the Kalman filter algorithm to apply the above described state vector prediction step (S511) cyclically.

As described above, according to the present invention, not only a single algorithm is used as an algorithm for tracking an object but a Kalman filter algorithm and a camshit algorithm are combined so as to be organically combined, thereby improving the reliability of object tracking .

According to embodiments of the present invention, the present invention can classify objects according to characteristics of objects such as a person, a vehicle, and an animal, that is, MER characteristic information (ratio of width, height, width height, density, center point coordinates, And an object classifier 160 for classifying the object corresponding to the final MER information by comparing the final MER information with the reference information.

In this case, the tracking unit 170 of the present invention may be configured to generate the moving-line data of the object through the above-described processing only when the object classifying unit 160 classifies the object corresponding to the final MER as a person . The path data generated by the tracking unit 170 and the like can be stored in the DB unit 180 of the present invention together with time information.

Hereinafter, the detailed configuration and processing of the present invention for generating a behavior pattern of an object using the copper line data of the finally generated object will be described.

When the tracking unit 170 of the present invention generates the copper line data of the object as described above, the behavior pattern determining unit 190 of the present invention determines the behavior pattern of the object based on the context- Or the final MER information for each view included in the copper line data to determine the behavior pattern of the object and generate corresponding behavior pattern information at step S320.

Specifically, the behavior pattern determination unit 190 of the present invention includes an event generation unit 191, an event DB unit 192, a pattern information storage unit 193, and a pattern information generation unit 194, as shown in FIG. 7 And the like.

In order to more accurately define the behavior pattern information of the object more accurately and to implement accurate interfacing between the system and the user, the present invention predefines six basic events that can be caused by the object as illustrated in FIG. 13, And a method of determining a behavior pattern of an object using time-series characteristics generated by combining one or more kinds of events in combination.

The six event information of the object illustrated in FIG. 13 and the position information on the boundary of the situation recognition area or the image processing area can be stored and utilized in the event DB unit 192 of the present invention.

The event generation unit 191 of the present invention extracts the coordinates of the center point position of the final MER information of the object of each time of the input copper line data when the copper line data of the object is input from the tracking unit 170, The event information for the object is generated using the relationship between the position coordinates of the boundary line of the situation recognition area stored in the event DB unit 192. [

In this regard, in FIG. 13, an entry event refers to an event that an object comes in from outside the context area, and is determined using the coordinates of the center of the object (final MER information) and the boundary of the context area.

An entry event may be an event that the object has moved out of the context aware area according to the embodiment, the object has moved from the context aware area to the image processing area outside the context aware area, , The case where the boundary of the image and the boundary of the context-aware area are in contact, or the case where the object is moved out of the image in the context-aware area.

The movement event is a motion event in which the object is located in the image processing region (which may be larger than the context-aware region) and there is a change in the center coordinates of the MER or a movement of the center coordinates of the object during the reference time within the context- Pixel) / reference time (s) and the calculated speed is not 0 (px / s).

The stop event corresponds to a case in which the amount of change (moving distance) of the central coordinate during a specific reference time is zero or substantially equal to zero, or when the velocity of the central coordinate is substantially zero.

The separation event is an event corresponding to the case where the first detected object is detected for the first time when another object (person, object, etc.) is detected in the object and the adjacent area when the object is inside the context area, Refers to when the center coordinates of the MERs of the different objects approach each other and when the regions of the MER overlap and then move to the same displacement when the different objects (person vs. person or person vs. object, etc.) are within the context area Event.

When one or more event information is generated as described above, the pattern information generator 194 of the present invention generates behavior pattern information for the object using the time series characteristic of the generated one or more event information.

The behavior pattern determination unit 190 of the present invention may further include a pattern information storage unit 193 which stores six patterns of the objects A plurality of reference behavior pattern information defined by a time series combination of a plurality of event information among events, i.e., entry, advance, movement, stop, separation, and association events can be stored.

In this regard, among the reference behavior pattern information shown in Fig. 14, the " path passing " pattern means a situation in which an object moves beyond a set direction and a set boundary line (an outer boundary line or another inner boundary line) Quot; movement " pattern means a pattern in which a positional variation of an object is generated and moved in a direction opposite to a predetermined direction when a position of an object of a current frame is compared with a position of an object of a previous frame.

When a plurality of reference behavior pattern information is stored in the pattern information storage unit 193 as described above, the pattern information generation unit 194 of the present invention generates the pattern information of the reference MER The reference behavior pattern information corresponding to event information of feature information (center point coordinates, width, height, width / height ratio, density, etc.) of information is selected, Information. ≪ / RTI >

By combining the six basic event information in a time series manner, it is possible to precisely define various behavior patterns of an object as well as to form an entire block using a piece, By associating in order, the complex behavior of the object can also be precisely defined.

Therefore, according to the present invention, the behavior pattern information of an object is continuously and accurately generated by an accurate and automated method, so that the situation of the object can be accurately and automatically recognized. Therefore, the generated behavior pattern information can be accurately interfaced to the user, It is possible to provide a system infrastructure in which follow-up or counter-processing optimized for a given situation (behavior pattern) can be induced.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not to be limited to the details thereof and that various changes and modifications will be apparent to those skilled in the art. And various modifications and variations are possible within the scope of the appended claims.

In the above description of the present invention, the first and second modifiers are merely terms of a tool concept used for relatively separating the components of each other, so that they are used to indicate a specific order, priority, etc. It should be interpreted that it is not a terminology.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed. It should be understood that various modifications may be made in the ordinary skill in the art.

1000: Intelligent monitoring system of the present invention
50: index camera 70: tracking camera (zoom camera)
100: object information generating unit 200:
210: Spatial information storage unit 220: Position information conversion unit
230:
300: input unit 400: face information detection unit
410: first detection unit 420: second detection unit
430:
500: feature information generating unit 600: feature information extracting unit
700: rotation information generating unit 800:
810: Coordinate setting unit 820: Object zone creation unit
830: Object information generating unit

Claims (11)

An object information generation unit for generating motion data of the detected object using the image data of the detection camera and generating motion pattern information of the object using the positional relationship between the generated motion data and the context aware area;
An image controller for controlling a tracking camera capable of PTZF driving such that a near image of the object is captured when the movement of the object is stopped;
An input unit for receiving the near vision image data from the tracking camera;
A first detection unit for detecting first face area information by applying a haar feature based algorithm to the near vision image data, a second detection unit for detecting second face area information by applying a skin color based algorithm to the near vision image data, A facial information detector for detecting facial region information in the near vision image data, the detection information being generated by generating a facial region information on a hybrid region using the first and second facial region information;
A feature information extracting unit for extracting a feature region of a face including nose, mouth, and two-eye region from the face region information;
A rotation information generation unit for generating length information of each of the nose and eye and length information between the nose and mouth and generating rotation direction and rotation angle information of the face using the length information;
The method comprising the steps of: generating feature information including the age, wearable object, or facial identification information of the object using the face area information, rotation direction and rotation angle information of the face, A feature information generation unit for generating wear tool information worn on the face by applying a combination of the presence or absence of the face and the pixel characteristics of the area around the eyes;
When the image data of the detection camera or the tracking camera is input, a human object is extracted from the input image data, and the MER region of the extracted human object is divided into a head region, a chest region, a lower body region, A coordinate setting unit for setting a coordinate having a maximum brightness value in the data as a skeleton coordinate by applying the distance conversion algorithm;
Estimating a position of a hand using skeleton coordinates, designating an area around the hand centered on a skeleton coordinate mapped with the estimated position of the hand, detecting an outline of the area around the designated hand, An object zone generating unit for generating an object zone which is an excluded zone; And
And an object information generating unit for generating object information included in the object region by comparing feature information of the object region with reference information for object determination. Intelligent surveillance system. delete The image processing apparatus according to claim 1,
A spatial information storage unit for storing spatial model information including position information of the detection camera, position information of the tracking camera, characteristic information of the tracking camera, and three-dimensional position information of each point;
A position information conversion unit for generating three-dimensional coordinates corresponding to MER center coordinates of an object whose movement is stopped using the spatial model information; And
And a transmitter for calculating a PTZ control value of the tracking camera and transmitting the PTZ control value to the tracking camera so that the generated three-dimensional coordinate image is captured. Intelligent surveillance system with adaptive technique. delete delete delete delete 2. The apparatus according to claim 1,
Wherein the feature information for the gender of the object is generated by distinguishing the face region and the hair region from the face region information and using the vertical ratio of the divided face region and the hair region, Intelligent Monitoring System with Recognition Technique. The apparatus of claim 1,
a receiving unit for receiving image data from n detection cameras (n is a natural number of 1 or more);
A first detection unit for detecting an object by applying a difference image detection algorithm to the image data and generating first candidate MER information of the detected object;
A second detecting unit detecting an object by applying an optical flow algorithm to the image data and generating second candidate MER information of the detected object;
A detection determination unit for generating final MER information of the object by applying a hog algorithm to the first and second candidate MER information;
A tracking unit for tracking movement of final MER information of the object to generate copper line data of the object; And
And a behavior pattern determiner for determining a behavior pattern of the object by using a positional relationship between a situation recognition area serving as a basis of the behavior pattern analysis and the motion data of the object, Intelligent Monitoring System with Recognition Technique. 10. The image pickup apparatus according to claim 9,
Wherein the first candidate MER information is generated by generating a background image for each frame using a statistical image accumulated for a predetermined period of time and generating a difference between the generated background image and the current image, Intelligent Monitoring System with Recognition Technique. 10. The apparatus according to claim 9,
An event generation unit for generating event information of an object by using a relation between coordinates of a final MER center point position of an object of each time of the copper line data and a position coordinate of a boundary line of the situation recognition area; And
And a pattern information generation unit for generating behavior pattern information for the object using the time series characteristic of the event information of the object. The intelligent monitoring system according to claim 1, further comprising:

Images