KR102614895B1 - Real-time object tracking system and method in moving camera video - Google Patents

Abstract

A system for real-time tracking of objects in a dynamic camera image is disclosed. The tracking system is capable of horizontal movement, vertical movement, and rotation, and is capable of tracking a video camera that generates moving video frames taken while moving a specific area, and the moving video frames generated from the video camera. By inverse homography transforming the movement vectors according to the movement, compensation image frames are generated in which the movement of the video capture camera is removed, and objects detected according to key points from the generated compensation image frames are tracked according to location similarity and appearance similarity. Includes video analysis server.

Description

System and method for real-time tracking objects in dynamic camera images {REAL-TIME OBJECT TRACKING SYSTEM AND METHOD IN MOVING CAMERA VIDEO}

An embodiment according to the concept of the present invention relates to a technology for tracking an object in an image captured from a dynamic camera in real time. More specifically, the real-time removal of camera movement components from an image captured from a moving camera and the detection of the detected object are performed in real time. This relates to a technology for real-time tracking objects in dynamic camera images that can track objects more quickly and accurately by extracting and comparing feature vectors for each key point.

In modern times, interest in personal and public safety is increasing as social anxiety increases due to the frequent occurrence of various violent crimes. Accordingly, the number of closed-circuit televisions (CCTVs) being installed in urban residential areas, schools, and roads is gradually increasing in order to prevent and quickly resolve various incidents and accidents. And these CCTV systems are evolving from simple recording devices to intelligent CCTVs that can serve as real-time surveillance and reporters by extracting key features from images and recognizing objects. In addition, recent CCTV systems have begun to capture images using dynamic cameras such as PTZ (Pan-Tilt-Zoom) cameras to overcome limitations in the shooting range and resolution of CCTV cameras. However, conventional or recent CCTV systems only provide a method of recognizing and tracking objects in images captured by fixed cameras, but do not provide a method of recognizing and tracking objects in real time for images captured by moving cameras. .

The technical problem that the present invention aims to solve is that object tracking is possible not only for images captured from fixed cameras but also from moving cameras, and object tracking is possible not only in low-density environments but also in high-density complex environments, and not only in previously captured images, but also in images captured by moving cameras. The goal is to provide a system for real-time tracking objects in dynamic camera images that enables real-time object tracking even for images currently being shot.

Another technical problem that the present invention aims to solve is that object tracking is possible for images captured not only from fixed cameras but also from moving cameras, and object tracking is possible not only in low-density environments but also in high-density and complex environments, and is capable of tracking only images that have already been captured. In addition, it provides a method of real-time tracking objects in dynamic camera images that enables real-time object tracking even for images currently being shot.

A system for real-time tracking of objects in a dynamic camera image according to an embodiment of the present invention is capable of horizontal movement, vertical movement, and rotation, and includes an image capture camera that generates moving image frames captured while moving in a specific area, and the image capture camera. Compensation image frames that remove the movement of the video capture camera are generated by inverse homography transformation of movement vectors according to the movement of the video capture camera with respect to the moving image frames generated from the camera, and key points are generated from the generated compensation image frames. It includes a video analysis server that tracks objects detected according to location similarity and appearance similarity.

At this time, the video analysis server performs homography inverse transformation on the calculated motion vectors for the moving video frames and a camera motion estimation module for calculating motion vectors according to the movement of the video capture camera for the moving video frames. A camera motion compensation module that generates compensation image frames by removing the movement of the video capture camera, an object detection module that detects the object by recognizing the key points according to each part of the object from the compensation image frames, and the location similarity and appearance. It includes an object tracking module that tracks the detected object according to similarity.

Depending on the embodiment, the camera motion estimation module converts the pixel values of the moving image frames into gray scale, and generates a grid composed of grid points arranged at regular intervals in the previous image frame among the moving image frames. For each of the grid points, the grid point whose gray scale value change in the current image frame compared to the previous image frame is greater than a predetermined standard value is selected as a feature point, and the prediction of the selected feature points in the next image frame is performed through the pyramid Lucas Canae algorithm. It is characterized by estimating the position, obtaining the motion vectors of the feature point pair, calculating a two-dimensional histogram for the motion vectors of the feature point pair, and calculating the highest frequency motion vectors as motion vectors according to the movement of the video capture camera. .

Depending on the embodiment, the motion compensation module is a homography matrix for homography converting feature points in a previous image frame among the moving image frames to a specific position of the current image frame based on movement vectors according to the movement of the image capturing camera. Calculate and apply the inverse transformation of the calculated homography to all pixels of the current image frame to generate compensated image frames in which the video capture camera movement in the current image frame is removed compared to the previous image frame. .

According to an embodiment, the object detection module is characterized in that it extracts a feature vector representing the appearance of the detected object according to the recognized key point.

Depending on the embodiment, the object tracking module uses a Kalman filter to estimate the position predicted to exist in the current image frame for the tracking object that was being tracked until the previous image frame among the objects detected by the object detection module, and detects the object. The location similarity is determined by comparing the positions of the candidate objects detected in the current image frame by the module with the positions of the tracked objects estimated using the Kalman filter, and the feature vectors of the tracked objects and the feature vectors of the candidate objects. The appearance similarity is determined by comparing, and the tracking object is tracked by determining the identity between the tracking object and the candidate object according to the determined location similarity and the appearance similarity.

At this time, the object tracking module determines the identity between the tracking object and the candidate object by assigning a weight according to a predetermined ratio to each of the determined location similarity and the appearance similarity, and determining the identity between the tracking object and the candidate object within a predetermined range adjacent to the estimated location of the tracking object. When the candidate objects are detected within the location similarity, a greater weight than the appearance similarity is given to the location similarity, and when the candidate objects are detected outside a predetermined range adjacent to the estimated estimated location, the location similarity is assigned to the location similarity. It is characterized by assigning a greater weight than similarity.

A method of real-time tracking an object in a dynamic camera image according to an embodiment of the present invention includes transmitting moving image frames captured while a video capture camera moves to a camera motion estimation module, and the camera motion estimation module transmits the transmitted motion. estimating the motion of the video capture camera by calculating a motion vector for the video capture camera in video frames; and a compensation video frame in which a camera motion compensation module removes the estimated motion of the video capture camera from the moving video frames. generating and transmitting to an object detection module, the object detection module detecting a tracking object in a corresponding compensation image frame among the transmitted compensation image frames, and extracting a feature vector of the detected tracking object; and object tracking. a module estimating the expected position of the tracked object in a compensation image frame subsequent to the corresponding compensation image frame using a Kalman filter; and the object detection module detecting a candidate object in a compensation image frame subsequent to the corresponding compensation image frame. , extracting feature vectors of the detected candidate objects, and positional similarity between the candidate objects detected within or outside the predetermined range adjacent to the estimated expected position by the object tracking module and the tracked objects, and It includes the step of determining whether or not there is a match by comparing the appearance similarity according to the extracted feature vector.

Depending on the embodiment, the step of the camera motion estimation module estimating the motion of the video capture camera includes converting pixel values of the transmitted moving video frames into gray scale and arranging the moving video frames at regular intervals. A step of generating a grid composed of grid points, and for each of the grid points, selecting grid points whose gray scale value change in the current image frame compared to the previous image frame is greater than a predetermined standard value as feature points, and using a predetermined optical flow method. estimating the positions of the feature points for the next image frame; obtaining movement vectors of feature point pairs using feature points selected from the current image frame and feature points in the next frame of the estimated next image; and calculating movement vectors of the feature point pair. Calculating a two-dimensional histogram for the vectors to determine the highest frequency motion vectors and calculating a motion vector for the video camera.

Depending on the embodiment, the step of generating the compensated image frames by the camera motion compensation module and transmitting them to the object detection module involves placing feature points in the previous image frame at a specific location in the current image frame through the determined highest frequency motion vectors. Calculating a homography matrix for conversion to pixels in the moving image frame and applying inverse transformation of the calculated homograph matrix to the pixels in the moving image frame to generate the compensation image frames from which the image capturing camera movement is removed. and transmitting one compensated image frame to the object detection module.

As described above, the system and method for real-time tracking an object in a dynamic camera image according to an embodiment of the present invention can calculate the movement of the object itself within the image frame, excluding the movement of the video camera from the movement of the object within the image frame. Therefore, it is effective in stably tracking objects not only from fixed camera images but also from non-fixed camera images.

In addition, the system and method for real-time tracking of objects in a dynamic camera image according to an embodiment of the present invention minimizes system load and can exclude camera movement very quickly, so it can be applied not only to previously captured images but also to images currently being captured. It has the effect of tracking objects in real time.

Furthermore, the system and method for real-time tracking of objects in a dynamic camera image according to an embodiment of the present invention uses a method of recognizing each key point for a person's body part, so that it can be used not only in low-density environments but also in high-density complex environments where objects overlap. It is also effective in tracking objects precisely.

A detailed description of each drawing is provided to more fully understand the drawings cited in the detailed description of the present invention.
Figure 1 is a block diagram showing the configuration of a system for real-time tracking of objects in a dynamic camera image according to an embodiment of the present invention.
FIG. 2 is an example diagram in which the camera motion estimation module shown in FIG. 1 displays grid points and feature points in the corresponding image frame.
FIG. 3 is an example of how the camera motion estimation module shown in FIG. 1 determines a high-frequency motion vector by calculating a two-dimensional histogram for the motion vector.
Figure 4 is an exemplary diagram in which inverse homography transformation is applied according to an embodiment of the present invention.
Figure 5 is a flow chart to explain a method for real-time tracking an object in a dynamic camera image according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can make various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, and similarly The second component may also be named the first component.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "immediately between" or "neighboring" and "directly adjacent to" should be interpreted similarly.

The terms used in this specification are merely used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this specification, terms such as “comprise” or “have” are intended to indicate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, but are not intended to indicate the presence of one or more other features or numbers. It should be understood that this does not preclude the existence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains.

Terms as defined in commonly used dictionaries should be interpreted to include meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, be interpreted in an ideal or excessively formal sense. It doesn't work.

Hereinafter, the present invention will be described in detail by explaining preferred embodiments of the present invention with reference to the accompanying drawings.

Figure 1 is a block diagram showing the configuration of a system for real-time tracking an object in a dynamic camera image (hereinafter referred to as 'tracking system 10') according to an embodiment of the present invention.

Referring to FIG. 1, the tracking system 10 includes a video capture camera 100 that captures a desired area of a specific area, removes the camera's movement component from the image of the captured area, and tracks the object according to the object's movement component. It includes a video analysis server 200 that does.

The video capture camera 100 can be implemented not only as a general CCTV camera with a fixed area, but also as a dynamically moving camera, such as a Pan-Tilt-Zoom (PTZ) camera capable of horizontal movement, vertical movement, and zoom adjustment. there is.

That is, the video capture camera 100 can move to photograph a specific area or a specific object, and transmits moving video frames (1_frame to n_frame) captured while moving to the video analysis server 200.

The video analysis server 200 includes a camera motion estimation module 230, a camera motion compensation module 250, an object detection module 270, and an object tracking module 290, and images transmitted from the video capture camera 100. It performs the role of identifying and tracking the object movement state for each frame by removing the camera movement component.

First, when the camera motion estimation module 230 of the video analysis server 200 acquires moving video frames (1_frame to n_frame) through the video capture camera 100, it compares them to the previous video frame (e.g., k-1_frame). It serves to estimate the movement of the video capture camera 100 in the current video frame (eg, k_frame).

At this time, the camera motion estimation module 230 determines how the background (e.g., building, store, etc.) is located in the current video frame (k_frame) compared to the previous video frame (k-1_frame) based on a real-time video stitching algorithm. The movement of the video camera 100 is estimated by determining whether there has been a change.

When there is movement of objects in the current image frame (e.g., k_frame) compared to the previous image frame (e.g., k-1_frame), this means that there is movement of the object itself or movement of the video capture camera 100 (e.g., the position of the background) It may be said that there is a change), and it may also be the case that both the movement of the object itself and the movement of the video capture camera 100 exist.

There may be object movement for various reasons as described above, and the camera motion estimation module 230 excludes the movement of the object within the image frame and estimates only the camera movement within the image frame.

The reason why the camera motion estimation module 230 estimates the camera movement within the video frame is that what can be directly obtained from the video frame is the 'movement of the object within the video frame', and from this 'movement of the object within the video frame' This is because by excluding the ‘movement of the video camera’, the ‘movement of the object itself within the video frame’ can be calculated.

The process by which the camera motion estimation module 230 estimates the motion of the video capture camera 100 in the moving video frame is as follows.

First, the camera motion estimation module 230 generates a grid composed of grid points arranged at regular intervals in the corresponding image frame, and each of the grid points forming the grid is not only the pixel value in the corresponding image frame, but also x It holds location information with axis coordinate values and y-axis coordinate values.

Afterwards, the camera motion estimation module 230 extracts feature points that serve as a standard for determining movement within the image frame using the generated grid points.

At this time, to extract the feature points, a general feature point extraction algorithm such as SIFT (Scale Invariant Feature Transform) using an image pyramid or SURF (Speed-Up Robust Feature) using a Hessian determinant may be used, but the SIFR or The SURF algorithm is not suitable for real-time environments because it requires a lot of computation to extract and match feature points.

Therefore, the camera motion estimation module 230 uses a feature point extraction method that can achieve a significantly faster calculation speed than the conventional method.

The method by which the camera motion estimation module 230 extracts the feature points will be described in more detail.

First, the camera motion estimation module 230 converts the pixel values of the image frames into an achromatic scale, that is, gray scale, rather than a color value.

Depending on the embodiment, the gray scale may be set within the range of 0.0 to 1.0 or within the range of 0 to 255.

Afterwards, the camera motion estimation module 230 calculates the gray scale value change in the current image frame (k_frame) compared to the previous image frame (e.g., k-1_frame) for each grid point, and the calculated gray scale value change. Only when it is greater than this predetermined standard value, the corresponding grid point is selected as the feature point.

For example, the gray scale value of the first grid point whose x-coordinate value is 10 and y-coordinate value is 10 in the previous image frame (k-1_frame) is 1.0 (white), and in the current image frame (k_frame), the gray scale value of the first grid point is 1.0 (white) and has the same coordinate value in the current image frame (k_frame). When the gray scale value of the first grid point is 0.0 (black), the amount of change in the gray scale value of the first grid point is 1.0.

At this time, if the predetermined reference value is 0.5, 1.0, which is the amount of change in the gray scale value of the first grid point, is greater than the predetermined reference value 0.5, so the camera motion estimation module 230 selects the first grid point as the feature point. .

In this way, the camera motion estimation module 230 selects the feature point by calculating the amount of change in gray scale value for each grid point.

FIG. 2 is an example diagram in which the camera motion estimation module 230 shown in FIG. 1 displays grid points and feature points in the corresponding image frame.

Referring to FIG. 2, it can be seen that a grid is created in the corresponding image frame, and the generated grid is composed of grid points arranged at predetermined intervals.

In addition, among the grid points, the grid points shown in yellow represent the final selected feature points, and the grid points shown in red represent grid points that cannot be selected as feature points because they do not meet the above predetermined standard.

Referring again to FIG. 1, the camera motion estimation module 230 estimates where the feature points extracted from the current image frame (k_frame) have moved to in the next frame (k+1_frame) through a predetermined optical flow method. do.

Depending on the embodiment, the optical flow method may be a block matching method, Horn-Shunck algorithm, Lucas-Kanade algorithm, Gunnar Farenback's algorithm, etc. The method may be used, but the camera motion estimation module 230 according to an embodiment of the present invention can analyze the optical flow for the extracted feature points, and has high accuracy compared to the time required for the algorithm, so it is most suitable for real-time imaging. It is desirable to use an algorithm (Iterative Lucas-Kanade Method with Pyramids).

That is, the camera motion estimation module 230 can estimate where the feature points selected as above have moved to in the next image frame (k+1_frame) through the Pyramid Lucas Canae algorithm.

Through this, the camera motion estimation module 230 uses the feature point location information in the current image frame (k_frame) and the location information of the feature point estimated for the next image frame (k+1_frame) to create a feature point pair (feature point in the current image frame). and find the movement vector of the feature point estimated in the next video frame.

For example, among several feature points existing in the current image frame (k_frame), the location information of the first feature point is (41, 25), and the location information of the first feature point estimated in the next image frame (k+1_frame) is (73, 81), the movement vector of the first feature point is V1 = (32, 56).

In this way, the camera motion estimation module 230 estimates the positions in the next image frame (k+1_frame) for all feature points existing in the current image frame (k_frame) and obtains movement vectors for these feature point pairs. You can.

Meanwhile, among the movement vectors of these feature point pairs, there may be a movement vector generated only by the movement of the object within the video frame, and there may also be a movement vector generated solely by the movement of the video capturing camera 100.

Moving objects such as people and cars occupy a small area within the video frame, while background parts such as buildings occupy most of the area within the video frame. Therefore, the movement vector generated from the background part is generated by the video capture camera (100 ) can be seen as being caused by the movement of.

Therefore, the motion vector with the highest frequency can be determined by calculating a 2D histogram for all motion vectors, and the motion vectors corresponding to the highest frequency are the background motion vectors, that is, according to the movement of the video capture camera 100. Judging by the movement vector.

In reality, there may be more than hundreds of motion vectors of the feature point pair in an image frame. However, in the detailed description of the present invention, for convenience of explanation, it is assumed that the total number of motion vectors is 4, and the storage (bin) of the two-dimensional histogram is assumed to be 4. explains that x and y are classified into 10 units.

For example, the first movement vector (V1) is (32, 56), the second movement vector (V2) is (41, 25), the third movement vector (V3) is (37, 51), and the fourth movement vector (V4) ) is (35, 53).

At this time, if the first to fourth movement vectors (V1 to V4) are input to storage of x and y units of 10, respectively, the first movement vector (V1) is input to the storage of (3, 5), The second motion vector (V2) is input into the storage of (4, 2), the third motion vector (V3) is input into the storage of (3, 5), and the fourth motion vector (V4) is ( It is entered into the repositories of 3 and 5).

At this time, the storage where the most movement vectors are input is the (3, 5) storage where the first movement vector (V1), the third movement vector (V3), and the fourth movement vector (V4) are stored, and the (3) , 5) The first, third, and fourth motion vectors stored in the storage can be said to be motion vectors (i.e., background motion vectors) corresponding to the highest frequency.

Accordingly, the camera motion estimation module 230 uses the first motion vector (V1), the third motion vector (V3), and the fourth motion vector (V4) related to the background motion vector to the motion of the video capture camera 100. It is judged based on the movement vector.

Since the video capture camera 100 can simply move in the horizontal direction (or vertical direction), but can also rotate (moving in the horizontal and vertical directions simultaneously), the movement vector according to the movement of the video capture camera 100 is as described above. It may appear as a plurality of movement vectors (eg, V1, V3, and V4) with similar sizes and directions.

Figure 3 is an example diagram in which the camera motion estimation module 230 determines a high-frequency motion vector by calculating a two-dimensional histogram for the motion vector.

Referring to FIG. 3, the displayed red line becomes a movement vector corresponding to the final selected camera movement, and it can be determined that there was movement roughly in the direction and size of the red line in the current frame compared to the previous frame.

Referring again to FIG. 1, the camera motion compensation module 250 calculates feature points in the previous image frame (k-1_frame) through movement vectors according to the movement of the video capture camera 100 determined from the camera motion estimation module 230. Calculate a homography matrix to convert the images to a specific location in the current image frame (k_frame).

In general, homography refers to a method of converting two images viewed from two different perspectives in 3D space by projecting an image in 3D space into 2D space. In this case, a matrix expressing the relationship between the two different images is used. It is called a homography matrix.

In other words, the homography matrix can be said to be a matrix that transforms specific feature points located in the previous image frame (k-1_frame) so that they exist at a specific position in the current image frame (k_frame).

At this time, the specific feature point means a feature point for which the movement vector of the feature point pair is determined to be a movement vector according to the movement of the video capture camera 100, and the specific position of the current image frame (k_frame) is the optical flow Through the method, the specific feature point refers to a position where the specific feature point is estimated to exist in the current image frame (k_frame).

Therefore, specific feature points located in the previous image frame (k-1_frame) exist at specific positions in the current frame (k_frame) when the homography transformation is applied.

In other words, the homography in this specification can be said to be movement of an object based on the movement of the video capture camera 100 excluding the movement of the object itself.

Meanwhile, other feature points other than the specific feature points located in the previous image frame (k-1_frame) are located at a specific location in the current image frame (k_frame) when the homography transformation is applied and the current image frame (k_frame) through the optical flow method described above. ), the estimated location may be different.

This difference can be seen as the fact that other feature points other than the specific feature points are not only moved due to the movement of the video capture camera 100 (i.e., the homograph transformation), but also because there is movement of the object itself.

Therefore, all feature points in the current image frame (k_frame) have position information estimated through the optical flow method, and when the inverse homograph transformation is performed on them, the position information from which the movement of the video capture camera 100 is removed will hold.

That is, when performing the inverse transformation of the homograph, objects related to the background have the same position information compared to the previous image frame (k-1_frame) because the movement of the video capture camera 100 has been removed, and objects unrelated to the background ( For example, people, cars, etc.) have location information that reflects only the movement of the object itself.

For this reason, the camera motion compensation module 250 applies the inverse transformation of the homography calculated as above to all pixels forming the current image frame (k_frame), and captures the image in the current image frame (k_frame) compared to the previous image frame. Compensatory image frames (eg, 1_frame_am to n_frame_am) from which movement of the camera 100 is removed are generated.

As a result, all pixels in the current compensated image frame (k_frame_am) to which the inverse homography transformation has been applied have location information in which the movement of the video capturing camera 100 has been removed.

Figure 4 is an exemplary diagram in which inverse homography transformation is applied according to an embodiment of the present invention.

At this time, (a) in Figure 4 shows the moving image frames (k-1_frame and k_frame) before the inverse homography transformation is applied, and (b) in Figure 4 shows the compensation image frames (k-1_frame_am and k-1_frame_am) after the inverse homography transformation is applied. k_frame_am).

Referring to (a) of FIG. 4, due to the movement of the video capture camera 100 in the right direction, the background object (e.g., electric pole) in the current video frame (k_frame) compared to the previous video frame ((k-1_frame)) You can see that it has moved to the left.

In contrast, referring to (b) of FIG. 4, it can be seen that even though the video capture camera 100 moves, the absolute position of the background object (eg, electric pole) in the video is fixed.

Thereafter, the camera motion compensation module 250 transmits the compensated image frames (1_frame_am to n_frame_am) generated by removing the movement of the video capture camera 100 from the video frame as described above to the object detection module 270.

Referring again to FIG. 1, the object detection module 270 detects an object (e.g., a person) that is the target of tracking and counting within the compensated image frames (1_frame_am to n_frame_am) transmitted from the camera motion compensation module 250. And extract a feature vector expressing the appearance of the detected object.

In the case of a general person detection model, the entire area occupied by the person in the image is recognized and a bounding box surrounding the person is returned, but the object detection module 270 according to an embodiment of the present invention uses the human body part A method is used to recognize each key point (e.g., body parts such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, ankle, etc.) and inversely estimate the entire area of the person.

This method can detect people more precisely by extracting and recognizing information about human body parts using the concept of key points.

For example, if only the head of a specific person in an image is visible and the area below the head is obscured by a passing car, there is a high possibility that a general person detection model that calculates a score for the entire area of the person will not recognize the specific person as a person.

However, since the object detection module 270 according to an embodiment of the present invention can calculate a score for each key point of a person, even if the score for the entire human body is calculated low, a high score is calculated for the head area, so only the head is calculated. Exposed people can also be accurately detected.

Afterwards, the object detection module 270 extracts a feature vector representing the appearance of the detected object.

The feature vector is an appearance vector extracted from the pixel value of the target human area (eg, area for each key point), and the object detection module 250 extracts the feature vector using information for each key point.

In other words, the object detection module 250 can determine which body part is visible or invisible within the image frame by using information for each key point, and extracts a feature vector using the visible part of the object.

Accordingly, the object detection module 250 can accurately recognize a person and extract the corresponding feature vector even when the density of multiple people overlapping within the corresponding image frame is high.

Meanwhile, the object tracking module 290 uses a detection-based tracking method (Track by Detect) to determine the identity of the object detected in the previous image frame (e.g., k-1_frame_am) and the object detected in the current image frame (e.g., k_frame_am). ) to track the object.

At this time, the compensation image frames (e.g., 1_frame_am to n_frame_am) for the object tracking module 290 to track the corresponding object are generated by the camera motion compensation module 250 by removing the movement of the video capture camera 100. These are video frames.

In relation to the detection-based tracking method, the first image frame (1_frame_am) is the frame where the image first starts, and since there is no object being tracked in the first image frame (1_frame_am), an initialization step of starting tracking for all detected objects It can be said.

In addition, the k-th video frame (k_frame_am) matches the objects being tracked up to the k-1-th video frame (k-1_frame_am) with the objects detected in the k-th video frame (k_frame_am), thereby converting the existing tracking to the current video frame. It can be said to be a step to continue or to start new tracking for objects that do not match the existing tracking.

At this time, when tracking the object using the detection-based tracking method, the object tracking module 290 determines the location similarity and appearance similarity (similarity of feature vectors for each key point) between the objects detected by the object detection module 250. You can track the object accordingly.

Hereinafter, a detailed description will be given of how the object tracking module 290 tracks the corresponding object.

First, the object tracking module 290 uses a Kalman Filter to track an object (e.g., tracking object (Tr_obj)) that was being tracked up to the previous image frame (e.g., k-1_frame_am) in the current frame (e.g., k_frame_am). Estimate the location where it is expected to exist.

Thereafter, the object tracking module 290 compares an object (e.g., a candidate object (Ca_obj)) detected within a predetermined range adjacent to the estimated location from the object detection module 270 with the object (Tr_obj) being tracked. .

At this time, the position estimated from the object tracking module 290 or the position of the object detected from the object detection module 270 is the position in the image frame for which the camera movement has already been compensated by the camera motion compensation module 250, so there is a direct relationship between them. Location comparison is possible.

That is, the object tracking module 290 determines the positional similarity between the candidate objects detected by the object detection module 250 depending on whether they exist within a certain range of the estimated position, and determines the positional similarity between the candidate objects detected by the object detection module 250 and determines the positional similarity between the candidate objects detected by the object detection module 250. The feature vector is compared with the feature vector of the detected candidate object (Ca_obj) to determine the similarity in appearance between them.

As explained earlier, the feature vector is extracted using information for each key point (body parts such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, ankle, etc.), so multiple people overlap within the video frame. Even when density is high, accurate comparison is possible.

In addition, the object tracking module 290 compares the feature vector for each key point of the object being tracked (Tr_obj) with the feature vector for each key point of the detected candidate object (Ca_obj) to determine the similarity in appearance, and determines the degree of similarity according to each key point. By setting priorities, you can determine the similarity between objects.

For example, among key points such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, and ankle, the highest priority can be set to the feature vector of key points related to the human face, such as eyes, nose, and mouth. .

In this way, the object tracking module 290 determines the identity of the candidate object (Ca_obj) and the tracking object (Tr_obj) according to the determination of the location similarity and the appearance similarity, and determines the identity of the candidate object (Ca_obj) and the tracking object (Tr_obj). ) are defined as mutually matched if they are judged to be the same object.

On the other hand, when the density is high enough for multiple objects to overlap in the corresponding image frame (e.g., k_frame_am), that is, there may be cases where multiple candidate objects (e.g., Ca_obj1 to Ca_obj3) are detected within a certain range of the estimated position. You can.

In this case, the object tracking module 290 determines whether there is a match by comparing the appearance similarity and the location similarity with the tracking object (Tr_obj) for each of the plurality of candidate objects (Ca_obj1 to Ca_obj3).

At this time, a weight is given to the appearance similarity and the location similarity according to a predetermined ratio, and the result of applying the given weight to the determined appearance similarity and the location similarity is combined to select one of the plurality of candidate objects (Ca_obj1 to Ca_obj3). Match candidate objects that are determined to be the same as the tracking object (Tr_obj).

Depending on the embodiment, a predetermined value greater than 1 may be set as the weight for the appearance similarity, and a predetermined value less than 1 may be set as the weight for the location similarity.

This is because when judging the identity between objects, the similarity in appearance between objects can be viewed as more important comparison information than the similarity in location between objects. However, in cases where it is difficult to give high weight to similarity in appearance due to the influence of the video frame environment, etc. (the object itself is recognized Of course, in cases where this is not possible or when all feature vectors are not extracted for each key point, changes in the lighting environment within the image, changes in the posture of the object, etc.), a higher weight can be set for the position similarity.

For example, when no object is detected within a predetermined range adjacent to the estimated location, the object tracking module 290 tracks the tracking object (Tr_obj) for each of the candidate objects (e.g., Ca_obj4 to Ca_obj) detected outside the predetermined range. ) is determined by comparing the appearance similarity and the location similarity. However, for the location similarity, a predetermined value greater than 1 may be set as the weight, and for the appearance similarity, a predetermined value less than 1 may be set as the weight. You can set it.

Meanwhile, there may be an object (Un_obj) that was detected in previous video frames (1_frame_am to k-1_frame_am) but continues to fail to track, and the estimated position of the object (Un_obj) that failed to track is in the current video frame (k_frame_am) There may be cases where no object is detected within a predetermined adjacent range.

In this case, the object tracking module 290 can set the weight for the location similarity to 0 and perform matching between objects based only on the appearance similarity, regardless of the location similarity.

For an object (Un_obj) that fails to match even through the above matching process, the position predicted through the Kalman filter is updated to the object position in the current image frame (k_frame_am).

Additionally, if matching fails more than a certain number of times in succession, the object is determined to have disappeared within the video frame (e.g., boarding a vehicle such as a bus or going down to a subway station) and is no longer tracked.

Figure 5 is a flow chart to explain a method for real-time tracking an object in a dynamic camera image according to an embodiment of the present invention.

Referring to FIGS. 1 to 5, moving video frames captured while the video capture camera 100, implemented as a Pan-Tilt-Zoom (PTZ) camera capable of horizontal movement, vertical movement, and zoom adjustment, moves. (1_frame to n_frame) is transmitted to the video analysis server 200 (S100).

The camera motion estimation module 230 of the video analysis server 200 receives moving video frames (1_frame to n_frame) transmitted through the video capture camera 100 (S150), and places the moving video frames (1_frame to n_frame) at regular intervals in the video frames. A grid consisting of grid points is created (S200).

Afterwards, the camera motion estimation module 230 converts the pixel values of the corresponding image frames into an achromatic scale, that is, gray scale, rather than a color value (S210).

Sequentially, the camera motion estimation module 230 calculates the amount of change in gray scale value in the current image frame (k_frame) compared to the previous image frame (e.g., k-1_frame) for each of the grid points, and calculates the calculated gray scale value. Only when the amount of change is greater than a predetermined standard value, the corresponding grid point is selected as the feature point (S230).

Afterwards, the camera motion estimation module 230 estimates where the feature points extracted from the current image frame (k_frame) have moved to in the next image frame (k+1_frame) using a predetermined optical flow method (S250).

Depending on the embodiment, the optical flow for the feature points extracted by the optical flow method can be analyzed, and the Pyramid Lucas Canae algorithm, which is most suitable for real-time imaging due to its high accuracy compared to the time required for the algorithm, can be used.

Sequentially, the camera motion estimation module 230 uses the feature point location information in the current image frame (k_frame) and the location information of the feature point estimated for the next image frame (k+1_frame) to pair feature points (feature points in the current frame). and the motion vector of the feature point estimated in the next frame (S270).

Next, the camera motion estimation module 230 calculates a two-dimensional histogram for all of the obtained motion vectors and determines the motion vectors with the highest frequency (S290).

At this time, the motion vectors corresponding to the determined highest frequency become background motion vectors, that is, motion vectors related to the movement of the video capture camera 100.

Meanwhile, the camera motion compensation module 250 converts specific feature points in the previous image frame (k-1_frame) to specific positions in the current image frame (k_frame) through the highest frequency motion vectors determined from the camera motion estimation module 230. Calculate the homography matrix (S300).

At this time, the specific feature point means a feature point whose motion vectors of the feature point pair are included in the determined highest frequency motion vectors, and calculating the homography matrix means estimating the motion of the video capture camera 100. It has the same meaning as

Sequentially, the camera motion compensation module 250 applies the inverse transformation of the calculated homograph matrix to the pixels in the current image frame (k_frame) to compensate for the movement of the image capture camera 100 by removing the compensated image frames (e.g., 1_frame_am). to n_frame_am) is generated (S330).

The camera motion compensation module 250 transmits the compensated image frames (1_frame_am to n_frame_am) generated by removing the movement of the video capture camera 100 to the object detection module 270 (S350).

The object detection module 270 detects an object (e.g., a person) to be tracked within the compensated image frames (1_frame_am to n_frame_am) transmitted from the camera motion compensation module 250 (S400), and A feature vector representing the appearance is extracted (S430).

The object detection module 270 according to an embodiment of the present invention detects each key point of a person's body part (e.g., body parts such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, ankle, etc.). A person can be detected more precisely by using a method that recognizes and inversely estimates the entire area of the person (S400).

Afterwards, the object detection module 270 extracts a feature vector (appearance vector) obtained from the pixel value of each key point of the detected object (S430).

Sequentially, the object tracking module 290 uses a Kalman Filter to track the current frame for the object being tracked (e.g., the tracking object (Tr_obj)) up to the previous image frame (e.g., k-1_frame_am) as described above. A position predicted to exist in (e.g., k_frame_am) is estimated (S500).

Thereafter, the object tracking module 290 selects candidate objects detected within a predetermined range adjacent to the estimated position from the object detection module 270 (e.g., Ca_obj1 to Ca_obj3) or candidate objects detected outside the predetermined range ( For example, Ca_obj4 to Ca_obj6) is compared with the object being tracked (Tr_obj) to determine whether there is a match (S550).

At this time, the matching refers to a case where the object tracking module 290 determines that the candidate object (Ca_obj) and the tracking object (Tr_obj) are the same object during the comparison.

The method (S550) in which the object tracking module 290 compares the candidate object and the tracked object to determine whether they match has been previously described, so redundant description will be omitted.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations can be made by those skilled in the art without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but rather to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

10: Tracking system
100: Video recording camera
200: Video analysis server
230: Camera movement estimation module
250: Camera movement compensation module
270: object detection module
290: Object tracking module

Claims (10)

A video capture camera capable of horizontal movement, vertical movement, and rotation, and generating moving video frames captured while moving a specific area, and
For the moving image frames generated from the video capturing camera, compensation image frames are generated by removing the movement of the video capturing camera by inverse homography transforming movement vectors according to the movement of the video capturing camera, and the generated compensation image frames are It includes a video analysis server that tracks objects detected according to key points according to location similarity and appearance similarity,
The video analysis server,
The pixel values of the moving image frames are converted to gray scale, a grid composed of grid points arranged at regular intervals in the previous image frame among the moving image frames is generated, and each of the grid points is compared to the previous image frame. Grid points whose gray scale value change in the current image frame is more than a predetermined standard value are selected as feature points, and the expected positions of the selected feature points in the next video frame are estimated through the pyramid Lucas Canae algorithm to obtain movement vectors of feature point pairs. , a camera motion estimation module that calculates a two-dimensional histogram for the motion vectors of the feature point pair and calculates the highest frequency motion vectors as motion vectors according to the motion of the video capture camera;
a camera motion compensation module that generates compensation image frames in which movement of the video capture camera is removed by performing inverse homography transformation on the calculated motion vectors for the moving video frames;
an object detection module that detects the object by recognizing the key points for each part of the object from the compensated image frames; and
A system for real-time tracking an object in a dynamic camera image, including an object tracking module that tracks the detected object according to the location similarity and appearance similarity. delete delete The method of claim 1, wherein the camera motion compensation module,
Calculate a homography matrix for homography conversion of feature points in previous image frames among the moving image frames to specific positions of the current image frame based on movement vectors according to the movement of the image capture camera,
In a dynamic camera image, the inverse transform of the calculated homography is applied to all pixels of the current image frame to generate compensated image frames in which the image capture camera movement in the current image frame is removed compared to the previous image frame. A system that tracks objects in real time. The method of claim 1, wherein the object detection module:
A system for real-time tracking an object in a dynamic camera image, characterized in that a feature vector representing the appearance of the detected object is extracted according to the recognized key point. The method of claim 1, wherein the object tracking module:
Using the Kalman filter, among the objects detected by the object detection module, estimate the position expected to exist in the current video frame for the tracking object that was being tracked until the previous video frame,
Determine the position similarity by comparing the positions of candidate objects detected in the current image frame by the object detection module with the positions of the tracked objects estimated using the Kalman filter,
Determine the appearance similarity by comparing feature vectors of the tracked object and feature vectors of the candidate objects,
A system for real-time tracking an object in a dynamic camera image, characterized in that the tracking object is tracked by determining the identity between the tracking object and the candidate object according to the determined location similarity and the appearance similarity. The method of claim 6, wherein the object tracking module,
Determine the identity between the tracking object and the candidate object by assigning a weight according to a predetermined ratio to each of the determined location similarity and the appearance similarity,
When the candidate objects are detected within a predetermined range adjacent to the estimated location of the tracked object, a greater weight is given to the location similarity than the appearance similarity, and the candidate objects are assigned to the candidate outside the predetermined range adjacent to the estimated location. A system for real-time tracking objects in a dynamic camera image, wherein when objects are detected, greater weight is given to the appearance similarity than the location similarity. Transmitting moving video frames captured while the video capture camera moves to a camera motion estimation module;
estimating the motion of the video capture camera by the camera motion estimation module calculating a movement vector for the video capture camera from the transmitted moving video frames;
A camera motion compensation module generating compensation image frames in which the estimated motion of the video capture camera is removed from the moving image frames and transmitting the compensation image frames to an object detection module;
The object detection module detecting a tracking object in a corresponding compensation image frame among the transmitted compensation image frames and extracting a feature vector of the detected tracking object;
estimating, by an object tracking module, the expected position of the tracked object in a compensation image frame subsequent to the corresponding compensation image frame using a Kalman filter;
detecting, by the object detection module, at least one candidate object in a compensation image frame subsequent to the corresponding compensation image frame, and extracting a feature vector of the detected candidate object; and
The object tracking module determines whether or not a match is made by comparing the position similarity between the candidate objects detected within or outside the predetermined range adjacent to the estimated expected position and the tracked object and the appearance similarity according to the extracted feature vector. Including a judgment step;
The step of the camera motion estimation module estimating the movement of the video capture camera,
Converting pixel values of the transmitted moving image frames into gray scale, generating a grid composed of grid points arranged at regular intervals with respect to the moving image frames, and converting the previous image frame for each of the grid points. Selecting grid points whose gray scale value change in the current image frame is greater than a predetermined standard value as feature points; estimating the positions of the feature points for the next image frame through a predetermined optical flow method; and selecting the feature points in the current image frame. Obtaining motion vectors of a feature point pair using one feature point and a feature point in the next frame of the estimated next image; calculating a two-dimensional histogram for the obtained motion vectors to determine the highest frequency motion vectors; and transmitting the motion vectors to the video camera. A method for real-time tracking of an object in a dynamic camera image, including calculating a movement vector for the object. delete The method of claim 8, wherein the camera motion compensation module generates the compensated image frames and transmits them to the object detection module,
calculating a homography matrix for converting feature points in the previous image frame to specific positions in the current image frame through the determined highest frequency motion vectors;
generating the compensated image frames from which the video capturing camera movement is removed by applying an inverse transform of the calculated homograph matrix to pixels in the moving video frame; and
A method of real-time tracking an object in a dynamic camera image, comprising: transmitting the generated compensation image frames to the object detection module.