KR20220114819A - Real-time object tracking system and method in moving camera video - Google Patents

Abstract

A real-time object tracking system and method in a moving camera video are disclosed. The tracking system includes: a video camera that can move horizontally, vertically and rotate, and generates moving video frames captured while moving in a specific area; and a video analysis server that generates compensation video frames from which the motion of the video camera is removed by homographically inversely transforming motion vectors according to the motion of the video camera with respect to the moving video frames generated by the video camera, and tracks an object detected according to a key point from the generated compensation video frames according to location similarity and appearance similarity.

Description

Real-time OBJECT TRACKING SYSTEM AND METHOD IN MOVING CAMERA VIDEO

An embodiment according to the concept of the present invention relates to a technology for tracking an object in an image captured by a dynamic camera in real time, and more particularly, it removes a camera motion component from an image captured by a moving camera in real time, and It relates to a technology for real-time tracking of objects in dynamic camera images that can track objects faster and more accurately by extracting and comparing feature vectors for each key point.

In modern times, as social anxiety increases due to the frequent occurrence of various violent crimes, interest in personal and public safety is increasing. Accordingly, closed-circuit television (CCTV) is increasingly installed in residential areas, schools, and roads in downtown for proactive prevention and prompt resolution of various incidents and accidents. And this CCTV system is evolving into an intelligent CCTV that can serve as both a real-time monitoring and a reporter, rather than a simple recording device, through a method of recognizing an object by extracting key feature points from the image. In addition, the recent CCTV system began to shoot images using a dynamic camera such as a PTZ (Pan-Tilt-Zoom) camera to overcome the limitations of the shooting range and resolution of the CCTV camera. However, conventional or recent CCTV systems only suggest a method for recognizing and tracking an object for an image captured by a fixed camera, but do not provide a method for recognizing and tracking an object in real time for an image captured by a moving camera. .

The technical problem to be solved by the present invention is that object tracking is possible not only in a fixed camera but also in an image captured by a moving camera, object tracking is possible not only in a low-density environment but also in a high-density complex environment; It is to provide a system for real-time tracking of an object in a dynamic camera image capable of real-time object tracking even for an image currently being photographed.

Another technical problem to be solved by the present invention is that object tracking is possible not only in a fixed camera but also in an image taken by a moving camera, object tracking is possible not only in a low-density environment but also in a high-density complex environment, Rather, it is to provide a method for real-time tracking of an object in a dynamic camera image, which enables real-time object tracking even for an image currently being photographed.

A system for tracking an object in a dynamic camera image in real time according to an embodiment of the present invention is capable of horizontal movement, vertical movement, and rotation, and an image capturing camera that generates moving image frames captured while moving a specific area, and the image capturing Compensation image frames from which the motion of the imaging camera is removed are generated by inverse homography transformation of motion vectors according to the motion of the imaging camera with respect to the motion image frames generated by the camera, and a key point is obtained from the generated compensation image frames. and an image analysis server that tracks the detected object according to the location similarity and the shape similarity.

In this case, the image analysis server includes a camera motion estimation module that calculates motion vectors according to the motion of the image capturing camera with respect to the moving image frames, and homography inversely transforms the calculated motion vectors with respect to the moving image frames. A camera motion compensation module for generating compensated image frames from which the motion of an image capturing camera is removed, an object detection module for detecting the object by recognizing the key point according to each part of the object from the compensated image frames, and the positional similarity and shape and an object tracking module for tracking the detected object according to the degree of similarity.

According to an embodiment, the camera motion estimation module converts pixel values of the moving image frames into gray scale, and generates a lattice composed of lattice points arranged at regular intervals in a previous image frame among the moving image frames to generate the For each of the grid points, a grid point in which the gray scale value change in the current image frame compared to the previous image frame is greater than or equal to a predetermined reference value is selected as a feature point, and the selected feature points are expected in the next image frame through the Pyramid Lucas Kanae algorithm. It is characterized by calculating the motion vectors of the feature point pair by estimating the position, calculating the two-dimensional histogram for the motion vectors of the feature point pair, and calculating the highest frequency motion vectors as motion vectors according to the motion of the imaging camera. .

According to an embodiment, the motion compensation module is a homography matrix for homography-converting feature points in a previous image frame among the moving image frames to a specific position of the current image frame based on the motion vectors according to the motion of the image capturing camera. , and applying the calculated inverse transform of the homography to all pixels of the current image frame to generate compensated image frames in which the image capturing camera movement in the current image frame is removed compared to the previous image frame .

According to an embodiment, the object detection module extracts a feature vector representing the shape of the detected object according to the recognized key point.

According to an embodiment, the object tracking module estimates a position predicted to exist in a current image frame with respect to a tracking object being tracked up to a previous image frame among objects detected by the object detection module using a Kalman filter, and detecting the object The position similarity is determined by comparing the positions of the candidate objects detected in the current image frame by the module with the positions of the tracking objects estimated using the Kalman filter, and the feature vector of the tracking object and the feature vectors of the candidate objects to determine the shape similarity, and to track the tracking object by determining the sameness between the tracking object and the candidate object according to the determined location similarity and the shape similarity.

At this time, the object tracking module determines the sameness between the tracking object and the candidate object by assigning a weight according to a predetermined ratio to each of the determined position similarity and the shape similarity, but within a predetermined range adjacent to the estimated position of the tracking object. When the candidate objects are detected within It is characterized in that a weight greater than the similarity is given.

A method for tracking an object in a dynamic camera image in real time according to an embodiment of the present invention includes the steps of: transmitting moving image frames captured by an image capturing camera while moving to a camera motion estimation module; estimating the motion of the imaging camera by calculating a motion vector for the imaging camera from image frames, and a compensation image frame obtained by removing the estimated motion of the imaging camera from the moving image frames by a camera motion compensation module generating and transmitting them to an object detection module; detecting, by the object detection module, a tracking object in a corresponding compensation image frame among the transmitted compensation image frames, and extracting a feature vector of the detected tracking object; and object tracking estimating, by a module, an expected position of the tracking object in a compensation image frame after the corresponding compensation image frame by using a Kalman filter, and the object detection module detecting a candidate object in a compensation image frame after the corresponding compensation image frame, , extracting a feature vector of the detected candidate object and the position similarity between the candidate objects and the tracking object detected within or outside the predetermined range adjacent to the predicted position estimated by the object tracking module; and determining whether to match by comparing the degree of similarity in appearance according to the extracted feature vector.

According to an embodiment, the step of estimating, by the camera motion estimation module, the motion of the image capturing camera includes converting pixel values of the transmitted moving image frames into gray scale and arranging the moving image frames at regular intervals. Generating a lattice composed of lattice points, and for each of the lattice points, selecting lattice points in which the gray scale value change in the current image frame compared to the previous image frame is greater than or equal to a predetermined reference value as a feature point, and a predetermined optical flow method estimating the position of the feature points with respect to the next image frame through and calculating a motion vector for the imaging camera by calculating a two-dimensional histogram with respect to the vectors to determine the motion vectors of the highest frequency.

According to an embodiment, the step of generating, by the camera motion compensation module, the compensated image frames and transmitting the compensated image frames to the object detection module may include, through the determined highest frequency motion vectors, feature points in the previous image frame to a specific position of the current image frame. calculating a homography matrix for transforming into , and generating the compensated image frames from which the imaging camera motion is removed by applying an inverse transformation of the calculated homography matrix to pixels in the moving image frame; and transmitting one compensation image frame to the object detection module.

As described above, the system and method for real-time tracking of an object in a dynamic camera image according to an embodiment of the present invention can calculate the movement of the object itself in the image frame by excluding the movement of the video camera from the movement of the object in the image frame. Therefore, there is an effect of stably tracking an object not only from a fixed camera image but also from a non-fixed camera image.

In addition, the system and method for real-time tracking of an object in a dynamic camera image according to an embodiment of the present invention minimizes system load and can exclude camera movement very quickly, so that not only a previously captured image but also an image currently being captured It has the effect of being able to track real-time objects.

Furthermore, a system and method for real-time tracking of an object in a dynamic camera image according to an embodiment of the present invention uses a method of recognizing each key point for a human body part, so that not only a low-density environment but also a high-density complex environment in which objects are overlapped It also has the effect of precisely tracking objects.

A detailed description of each drawing is provided in order to more fully understand the drawings recited in the Detailed Description of the Invention.
1 is a block diagram showing the configuration of a system for tracking an object in a dynamic camera image in real time according to an embodiment of the present invention.
FIG. 2 is an exemplary diagram in which the camera motion estimation module shown in FIG. 1 displays grid points and feature points in a corresponding image frame.
FIG. 3 is an exemplary diagram in which the camera motion estimation module shown in FIG. 1 determines a high frequency motion vector by calculating a two-dimensional histogram with respect to the motion vector.
4 is an exemplary view to which inverse homography transformation according to an embodiment of the present invention is applied.
5 is a flowchart illustrating a method of tracking an object in a dynamic camera image in real time according to an embodiment of the present invention.

Specific structural or functional descriptions for the embodiments according to the concept of the present invention disclosed in this specification are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one element from other elements, for example, without departing from the scope of the present invention, a first element may be called a second element, and similarly The second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Other expressions describing the relationship between elements, such as "between" and "immediately between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise.

As used herein, terms such as “comprises” or “having” are intended to designate that the described feature, number, step, operation, component, part, or combination thereof is present, and includes one or more other features or numbers. , it is to be understood that it does not preclude the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as including meanings consistent with the meanings in the context of the related art, and unless explicitly defined in the present specification, they should be interpreted in an ideal or excessively formal meaning. doesn't happen

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a system (hereinafter, referred to as a 'tracking system 10') for tracking an object in a dynamic camera image in real time according to an embodiment of the present invention;

Referring to FIG. 1 , the tracking system 10 removes a moving component of the camera from an image for a captured area and an image capturing camera 100 for capturing a desired area of a specific area, and tracks the object according to the moving component of the object. and an image analysis server 200 that

The video recording camera 100 can be implemented as not only a general CCTV camera with a fixed area, but also a dynamically moving camera, for example, a Pan-Tilt-Zoom (PTZ) camera capable of horizontal movement, vertical movement, and zoom adjustment. have.

That is, the image capturing camera 100 is movable to photograph a specific area or a specific object, and transmits the moving image frames 1_frame to n_frame taken while moving to the image analysis server 200 .

The image analysis server 200 includes a camera motion estimation module 230 , a camera motion compensation module 250 , an object detection module 270 , and an object tracking module 290 , and the image transmitted from the image capturing camera 100 . It plays a role of tracking by identifying the moving state of the object from which the camera's moving component is removed for the frames.

First, when the camera motion estimation module 230 of the image analysis server 200 acquires moving image frames 1_frame to n_frame through the image capturing camera 100, compared to the previous image frame (eg, k-1_frame) It serves to estimate the motion of the image capturing camera 100 in the current image frame (eg, k_frame).

At this time, the camera motion estimation module 230 determines how the position of the background (eg, buildings, shops, etc.) in the current image frame (k_frame) compared to the previous image frame (k-1_frame) is determined based on a real-time video stitching algorithm. The movement of the imaging camera 100 is estimated by determining whether there is a change.

When there is movement of objects in the current image frame (eg, k_frame) compared to the previous image frame (eg, k-1_frame), this means that there was a movement of the object itself or the movement of the imaging camera 100 (eg, the position of the background). change) may exist, or it may be a case in which both the movement of the object itself and the movement of the image capturing camera 100 exist.

There may be object motion for various reasons as described above, and the camera motion estimation module 230 estimates only the camera motion within the image frame while excluding the motion of the object within the image frame.

The reason that the camera motion estimation module 230 estimates the camera motion in the image frame is that the 'movement of an object in the image frame' is directly obtained from the image frame, and in this 'movement of the object in the image frame', ' This is because if 'movement of the video camera' is excluded, 'movement of the object itself within the image frame' can be calculated.

A process in which the camera motion estimation module 230 estimates the motion of the image capturing camera 100 in the moving image frame is as follows.

First, the camera motion estimation module 230 generates a grid composed of grid points arranged at regular intervals in a corresponding image frame, and each of the grid points constituting the grid is x as well as a pixel value in the corresponding image frame. It holds position information having an axis coordinate value and a y-axis coordinate value.

Thereafter, the camera motion estimation module 230 extracts a feature point, which is a reference for finding out a motion within an image frame, from the generated grid points.

In this case, for extracting the feature points, a general feature point extraction algorithm such as SIFT (Scale Invariant Feature Transform) using an image pyramid or SURF (Speed-Up Robust Feature) using a Hessian determinant may be used, but the SIFR or The SURF algorithm is not suitable for a real-time environment because there is a large amount of computation for extracting and matching feature points.

Accordingly, the camera motion estimation module 230 uses a feature point extraction method that can achieve a significantly faster operation speed than the conventional general method.

A method for the camera motion estimation module 230 to extract the feature point will be described in more detail.

First, the camera motion estimation module 230 converts pixel values of image frames into an achromatic scale, that is, a gray scale, rather than a color value.

According to an embodiment, the gray scale may be set within a range of 0.0 to 1.0 or within a range of 0 to 255.

Then, the camera motion estimation module 230 calculates the gray scale value change amount in the current image frame (k_frame) compared to the previous image frame (eg, k-1_frame) for each of the grid points, and the calculated gray scale value change amount Only when it is equal to or greater than the predetermined reference value, the corresponding grid point is selected as the feature point.

For example, in the previous image frame (k-1_frame), the gray scale value of the first lattice point having an x-coordinate value of 10 and a y-coordinate value of 10 is 1.0 (white), and has the same coordinate value in the current image frame (k_frame). When the gray scale value of the first grid point is 0.0 (black), the change amount of the gray scale value of the first grid point becomes 1.0.

At this time, if the predetermined reference value is 0.5, the gray scale value change amount of the first grid point of 1.0 is greater than the predetermined reference value 0.5, so the camera motion estimation module 230 selects the first grid point as the feature point. .

In this way, the camera motion estimation module 230 selects the feature point by calculating the gray scale value change for each grid point.

FIG. 2 is an exemplary diagram in which the camera motion estimation module 230 shown in FIG. 1 displays grid points and feature points in a corresponding image frame.

Referring to FIG. 2 , it can be seen that a grid is generated in a corresponding image frame, and the generated grid is composed of grid points arranged at predetermined intervals.

Also, among the grid points, the grid points indicated in yellow indicate the final selected feature points, and the grid points indicated in red indicate the grid points that cannot be selected as the feature points because they do not satisfy the predetermined reference value.

Referring back to FIG. 1 , the camera motion estimation module 230 estimates to which position in the next frame (k+1_frame) the feature points extracted from the current image frame (k_frame) have moved through a predetermined optical flow method. do.

According to an embodiment, the optical flow method is a block matching method, a Horn-Shunck algorithm, a Lucas-Kanade algorithm, or a Gunnar Farenback's algorithm, etc. may be used, but the camera motion estimation module 230 according to an embodiment of the present invention can analyze the optical flow for the extracted feature points, and the accuracy is high compared to the algorithm required time, so the pyramid Lucas Canae most suitable for real-time images It is preferable to use an algorithm (Iterative Lucas-Kanade Method with Pyramids).

That is, the camera motion estimation module 230 may estimate to which position in the next image frame (k+1_frame) the selected feature points are moved through the Pyramid Lucas Canae algorithm.

Through this, the camera motion estimation module 230 uses the key point position information in the current image frame (k_frame) and the position information of the key point estimated for the next image frame (k+1_frame) in a key point pair (feature point in the current image frame). and the motion vector of the estimated feature point in the next image frame).

For example, the position information of the first feature point among several feature points existing in the current image frame (k_frame) is (41, 25), and the position information of the first feature point estimated in the next image frame (k+1_frame) is (73, 81), the movement vector of the first feature point becomes V1 = (32, 56).

In this way, the camera motion estimation module 230 estimates positions in the next image frame (k+1_frame) for all feature points existing in the current image frame (k_frame), and obtains motion vectors for these feature point pairs. can

Meanwhile, among the motion vectors of the pair of feature points, there may be motion vectors generated only by the motion of the object in the image frame, and there may be motion vectors generated solely by the motion of the image capturing camera 100 .

A moving object such as a person or a car is a partial area within the image frame, whereas a background such as a building will occupy most of the area within the image frame. ) can be attributed to the movement of

Therefore, it is possible to determine a motion vector with the highest frequency by calculating a 2D histogram for the entire motion vector, and the motion vectors corresponding to the highest frequency are the motion vectors of the background, that is, according to the motion of the imaging camera 100 . It is judged as a movement vector.

In practice, the number of motion vectors of the pair of feature points in the image frame may be several hundred or more, but in the detailed description of the present invention, for convenience of explanation, it is assumed that the total number of motion vectors is 4, and the two-dimensional histogram bin is explained by classifying x and y into units of 10.

For example, the first motion vector V1 is (32, 56), the second motion vector V2 is (41, 25), the third motion vector V3 is (37, 51), and the fourth motion vector V4 is ) is (35, 53).

At this time, if the first to fourth motion vectors V1 to V4 are input to the storage of x and y 10 units, respectively, the first motion vector V1 is input to the storage of (3, 5), The second motion vector V2 is input to the storage of (4, 2), the third motion vector V3 is input to the storage of (3, 5), and the fourth motion vector V4 is ( 3, 5) are entered into the storage.

In this case, the storage to which the largest number of motion vectors is input is the (3, 5) storage in which the first motion vector V1, the third motion vector V3, and the fourth motion vector V4 are stored, and the above (3) , 5) The first, third, and fourth motion vectors stored in the storage may be referred to as motion vectors (ie, motion vectors of the background) corresponding to the highest frequency.

Accordingly, the camera motion estimation module 230 applies the first motion vector (V1), the third motion vector (V3), and the fourth motion vector (V4) related to the motion vector of the background to the motion of the image capturing camera 100 . It is determined as a movement vector according to the

Since the video camera 100 can simply move in the horizontal direction (or in the vertical direction), but can also rotate (when moving in the horizontal and vertical directions at the same time), the motion vector according to the movement of the video camera 100 is may be represented by a plurality of motion vectors (eg, V1, V3, and V4) having similar magnitudes and directions.

3 is an exemplary diagram in which the camera motion estimation module 230 determines a high frequency motion vector by calculating a two-dimensional histogram with respect to the motion vector.

Referring to FIG. 3 , the displayed red line becomes a motion vector corresponding to the finally selected camera motion, and it can be determined that there was movement in the direction and magnitude of the red line in the current frame approximately compared to the previous frame.

Referring back to FIG. 1 , the camera motion compensation module 250 uses the motion vectors according to the motion of the imaging camera 100 determined by the camera motion estimation module 230 , the feature points in the previous image frame k-1_frame. A homography matrix is calculated for homography-transforming the images to a specific position of the current image frame (k_frame).

In general, homography is a transformation in which an image in 3D space is projected into 2D space, and refers to a method of transforming two images viewed from two different viewpoints in 3D space. It is called a homography matrix.

That is, the homography matrix can be referred to as a matrix that transforms specific feature points located in the previous image frame (k-1_frame) so that they exist at specific positions in the current image frame (k_frame).

In this case, the specific feature point means a feature point in which a movement vector of a pair of feature points is determined as a movement vector according to the movement of the image capturing camera 100, and the specific position of the current image frame (k_frame) is the optical flow It means a position at which the specific feature point is estimated to exist in the current image frame (k_frame) through the method.

Accordingly, specific feature points located in the previous image frame (k-1_frame) exist at a specific position in the current frame (k_frame) when the homography transformation is applied.

That is, in the present specification, the homography may be a movement according to the movement of the image capturing camera 100 excluding the movement of the object itself in the movement of the object.

On the other hand, other feature points except for the specific feature points located in the previous image frame (k-1_frame) are located at a specific position in the current image frame (k_frame) when the homography transformation is applied and the current image frame (k_frame) through the optical flow method described above. ) may be different.

This difference can be seen because other feature points other than the specific feature points are not only moved according to the movement of the image capturing camera 100 (ie, the homograph transformation) but also the object itself has a movement.

Accordingly, all feature points in the current image frame (k_frame) retain the position information estimated through the optical flow method, and when the inverse transformation of the homograph is performed on them, the position information from which the motion of the imaging camera 100 is removed. will hold

That is, when the inverse transformation of the homograph is performed, objects related to the background have the same position information as compared to the previous image frame (k-1_frame) because the motion of the image capturing camera 100 is removed, and objects not related to the background ( For example, a person, a car, etc.) has location information in which only the movement of the object itself is reflected.

For this reason, the camera motion compensation module 250 applies the inverse transform of the homography calculated as described above to all pixels constituting the current image frame (k_frame) to capture an image in the current image frame (k_frame) compared to the previous image frame Compensation image frames (eg, 1_frame_am to n_frame_am) from which the camera 100 movement is removed are generated.

As a result, all pixels in the current compensated image frame k_frame_am to which the inverse transformation of the homography is applied have position information from which the motion of the image capturing camera 100 is removed.

4 is an exemplary view to which inverse homography transformation according to an embodiment of the present invention is applied.

At this time, (a) of FIG. 4 shows the moving image frames (k-1_frame and k_frame) before the inverse homography transformation is applied, and (b) of FIG. 4 is the compensation image frame (k-1_frame_am and k_frame_am).

Referring to (a) of FIG. 4 , due to the movement in the right direction of the image capturing camera 100, a background object (eg, a power pole) in the current image frame (k_frame) compared to the previous image frame ((k-1_frame) It can be seen that has moved to the left.

In contrast, referring to FIG. 4B , it can be confirmed that the absolute position of the background object (eg, a power pole) in the image is fixed even though the image capturing camera 100 moves.

Thereafter, the camera motion compensation module 250 transmits the compensation image frames 1_frame_am to n_frame_am generated by removing the motion of the imaging camera 100 from the image frame as described above to the object detection module 270 .

Referring back to FIG. 1 , the object detection module 270 detects an object (eg, a person) to be tracked and counted within the compensation image frames 1_frame_am to n_frame_am transmitted from the camera motion compensation module 250 . and extracts a feature vector representing the shape of the detected object.

In the case of a general human detection model, the entire area occupied by a person in an image is recognized and a bounding box surrounding the person is returned, but the object detection module 270 according to an embodiment of the present invention is a human body part. A method of recognizing each key point (eg, body parts such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, ankle, etc.) for

In this way, by extracting and recognizing information on a body part of a person using the concept of the key point, a person can be detected more precisely.

For example, when only the head of a specific person in the image is visible and the lower part of the head is obscured by a passing car, there is a high probability that a general person detection model that calculates a score for the entire area of the person may not recognize the specific person as a person.

However, since the object detection module 270 according to an embodiment of the present invention can calculate a score for each key point of a person, even if the score for the entire human body is calculated low, a high score is calculated for the head, so only the head Even exposed people can be accurately detected.

Then, the object detection module 270 extracts a feature vector representing the shape of the detected object.

The feature vector is a feature vector extracted from pixel values of a target human region (eg, a region for each key point), and the object detection module 250 extracts the feature vector using information for each key point.

That is, the object detection module 250 may determine which body part is a visible part or an invisible part in the image frame by using the information for each key point, and extracts a feature vector using the visible part of the object.

Accordingly, the object detection module 250 can accurately recognize a person and extract a corresponding feature vector even when the density of overlapping multiple people within the corresponding image frame is high.

On the other hand, the object tracking module 290 determines the identity of an object detected in a previous image frame (eg, k-1_frame_am) and an object detected in a current image frame (eg, k_frame_am) is a detection-based tracking method (Track by Detect). ) to track the object.

At this time, the compensation image frames (eg, 1_frame_am to n_frame_am) for the object tracking module 290 to track the corresponding object are generated by the camera motion compensation module 250 removing the motion of the imaging camera 100. video frames.

In relation to the detection-based tracking method, the first image frame (1_frame_am) is a frame from which an image is first started, and since there is no object being tracked in the first image frame (1_frame_am), an initialization step of starting tracking of all detected objects can be said

In addition, the kth image frame (k_frame_am) matches the objects being tracked until the k-1th image frame (k-1_frame_am) with the objects detected in the kth image frame (k_frame_am) to convert the existing tracking into the current image frame. It can be said that it is a step of continuing or starting a new tracking for objects that do not match the existing tracking.

At this time, when the object tracking module 290 tracks the corresponding object using the detection-based tracking method, the position similarity and shape similarity between the objects detected by the object detection module 250 (similarity of feature vectors for each key point) You can track the object accordingly.

Hereinafter, a method for the object tracking module 290 to track a corresponding object will be described in detail.

First, the object tracking module 290 uses a Kalman filter to track an object (eg, a tracking object (Tr_obj)) that was being tracked until a previous image frame (eg, k-1_frame_am) in the current frame (eg, k_frame_am). Estimate where it is expected to exist.

Thereafter, the object tracking module 290 compares an object (eg, a candidate object Ca_obj) detected within a predetermined range adjacent to the estimated position from the object detection module 270 with the object being tracked (Tr_obj). .

At this time, the position estimated by the object tracking module 290 or the position of the object detected by the object detection module 270 is a position in the image frame for which the camera movement has already been compensated by the camera movement compensation module 250 , so they are directly related to each other. Location comparison is possible.

That is, the object tracking module 290 determines the position similarity between the candidate objects detected by the object detection module 250 according to whether or not they exist within a predetermined range of the estimated position, and By comparing the feature vector and the feature vector of the detected candidate object Ca_obj, the similarity in appearance is determined.

As described above, the feature vector is extracted using information for each key point (body parts such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, ankle, etc.), so that multiple people overlap within the image frame. Even when the density is high, accurate comparison is possible.

In addition, the object tracking module 290 compares the feature vector for each key point of the object being tracked (Tr_obj) with the feature vector for each key point of the detected candidate object (Ca_obj) to determine the similarity in appearance, but according to each key point It is possible to determine the degree of similarity between objects by setting priorities.

For example, the highest priority may be set to a feature vector of a key point related to a human face, such as an eye, nose, or mouth, among key points such as eyes, nose, mouth, shoulder, elbow, wrist, waist, knee, and ankle. .

In this way, the object tracking module 290 determines the identity of the candidate object Ca_obj and the tracking object Tr_obj according to the location similarity and the shape similarity determination, and the candidate object Ca_obj and the tracking object Tr_obj ) as the same object, it is defined as mutually matching.

On the other hand, when the density is high enough to overlap several objects in the corresponding image frame (eg, k_frame_am), that is, a plurality of candidate objects (eg, Ca_obj1 to Ca_obj3) are detected within a certain range of the estimated position. can

In this case, the object tracking module 290 determines whether to match each of the plurality of candidate objects Ca_obj1 to Ca_obj3 by comparing the similarity in appearance with the tracking object Tr_obj and the similarity in location.

At this time, weights are given according to a predetermined ratio to the degree of appearance similarity and the degree of location similarity, and the results of applying the weights to the determined degree of similarity in appearance and location are combined, and among the plurality of candidate objects (Ca_obj1 to Ca_obj3). A candidate object determined to be the same as the tracking object (Tr_obj) is matched.

According to an embodiment, a predetermined value greater than 1 for the appearance similarity may be set as the weight, and a predetermined value less than 1 for the location similarity may be set as the weight.

This is because, when determining the sameness between objects, the similarity in appearance between objects can be viewed as more important comparative information than the position similarity between objects. It goes without saying that a higher weight may be set for the position similarity in the case of not being able to do so, or when all feature vectors are not extracted for each key point, a change in the lighting environment in the image, a change in the posture of an object, etc.).

For example, when no object is detected within a predetermined range adjacent to the estimated position, the object tracking module 290 may detect a tracking object Tr_obj for each of the candidate objects (eg, Ca_obj4 to Ca_obj) detected outside the predetermined range. ) to determine whether a match is made by comparing the degree of similarity in appearance with the degree of location similarity, and a predetermined value greater than 1 may be set as the weight for the degree of location similarity, and a predetermined value smaller than 1 may be used as the weight for the degree of similarity in appearance. can be set.

On the other hand, there may be an object (Un_obj) that has been detected in the previous image frames (1_frame_am ~ k-1_frame_am) but has failed to be tracked, and the estimated position of the object (Un_obj) that has failed to be tracked in the current image frame (k_frame_am) and There may be a case where no object is detected within an adjacent predetermined range.

In this case, the object tracking module 290 sets the weight for the location similarity to 0, so that the object-to-object matching can be performed only with the appearance similarity regardless of the location similarity.

For the object Un_obj that has failed to match even through the above matching process, the position predicted through the Kalman filter is updated with the object position in the current image frame k_frame_am.

In addition, if matching fails a certain number of times in succession, it is determined that the object has disappeared within the corresponding image frame (eg, boarding a vehicle such as a bus, going down to a subway station) and tracking is no longer performed.

5 is a flowchart illustrating a method of tracking an object in a dynamic camera image in real time according to an embodiment of the present invention.

1 to 5 , moving image frames captured by the image capturing camera 100 implemented as a Pan-Tilt-Zoom (PTZ) camera capable of horizontal movement, vertical movement, and zoom adjustment while moving (1_frame to n_frame) is transmitted to the image analysis server 200 (S100).

The camera motion estimation module 230 of the image analysis server 200 receives the moving image frames 1_frame to n_frame transmitted through the image capturing camera 100 (S150), and is arranged at regular intervals in the image frame. A grid composed of grid points is created (S200).

Thereafter, the camera motion estimation module 230 converts the pixel values of the corresponding image frames into an achromatic scale, that is, a gray scale, rather than a color value ( S210 ).

Sequentially, the camera motion estimation module 230 calculates the gray scale value change amount in the current image frame (k_frame) compared to the previous image frame (eg, k-1_frame) for each of the grid points, and the calculated gray scale value Only when the amount of change is equal to or greater than a predetermined reference value, the corresponding grid point is selected as the feature point (S230).

Thereafter, the camera motion estimation module 230 estimates to which position in the next image frame (k+1_frame) the feature points extracted from the current image frame (k_frame) are moved through a predetermined optical flow method (S250).

According to an embodiment, it is possible to analyze the optical flow for the feature points extracted by the optical flow method, and it is possible to use the Pyramid Lucas Canae algorithm most suitable for real-time images because of its high accuracy compared to the required time of the algorithm.

Sequentially, the camera motion estimation module 230 uses the key point position information in the current image frame (k_frame) and the position information of the key points estimated for the next image frame (k+1_frame) in a key point pair (feature point in the current frame). and a motion vector of the feature point estimated in the next frame) is obtained (S270).

Next, the camera motion estimation module 230 calculates a two-dimensional histogram for all of the obtained motion vectors to determine the highest frequency motion vectors ( S290 ).

In this case, the motion vectors corresponding to the determined highest frequency become a motion vector of the background, that is, a motion vector related to the motion of the image capturing camera 100 .

Meanwhile, the camera motion compensation module 250 converts specific feature points in the previous image frame (k-1_frame) into specific positions of the current image frame (k_frame) through the motion vectors of the highest frequency determined by the camera motion estimation module 230 . A homography matrix is calculated for the purpose (S300).

In this case, the specific feature point means a feature point in which a motion vector of a pair of feature points is included in the determined highest frequency motion vectors, and calculating the homography matrix means estimating the motion of the imaging camera 100 . has the same meaning as

Sequentially, the camera motion compensation module 250 applies the inverse transformation of the calculated homograph matrix to pixels in the current image frame (k_frame) to remove the motion of the imaging camera 100 to compensate image frames (eg, 1_frame_am). to n_frame_am) are generated (S330).

The camera motion compensation module 250 transmits the compensation image frames 1_frame_am to n_frame_am generated by removing the motion of the imaging camera 100 as described above to the object detection module 270 (S350).

The object detection module 270 detects an object (eg, a person) to be tracked within the compensation image frames (1_frame_am to n_frame_am) transmitted from the camera motion compensation module 250 ( S400 ), and A feature vector representing the appearance is extracted (S430).

The object detection module 270 according to an embodiment of the present invention detects each key point (eg, eye, nose, mouth, shoulder, elbow, wrist, waist, knee, ankle, etc.) of a body part of a person. It is possible to detect a person more precisely by using a method of recognizing and estimating the entire area of the person in reverse (S400).

Thereafter, the object detection module 270 extracts a feature vector (appearance vector) obtained from pixel values for each key point of the detected object ( S430 ).

Sequentially, the object tracking module 290 uses a Kalman Filter to determine the current frame for the object (eg, the tracking object (Tr_obj)) that was being tracked until the previous image frame (eg, k-1_frame_am) as described above. A position predicted to exist in (eg, k_frame_am) is estimated ( S500 ).

Thereafter, the object tracking module 290 detects candidate objects (eg, Ca_obj1 to Ca_obj3) detected within a predetermined range adjacent to the estimated position from the object detection module 270 or candidate objects detected outside the predetermined range ( For example, whether Ca_obj4 to Ca_obj6) is matched with the object being tracked (Tr_obj) is determined (S550).

In this case, the matching refers to a case in which the object tracking module 290 determines that the candidate object Ca_obj and the tracking object Tr_obj are the same object during the comparison.

Since the method ( S550 ) in which the object tracking module 290 compares the candidate object and the tracking object and determines whether or not they match is described above, a redundant description will be omitted.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and variations are possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

10: tracking system
100: video recording camera
200: video analysis server
230: camera motion estimation module
250: camera motion compensation module
270: object detection module
290: object tracking module

Claims (10)

an image capturing camera capable of horizontal movement, vertical movement and rotation, and generating moving image frames captured while moving a specific area; and
Compensation image frames from which the motion of the imaging camera is removed are generated by inverse homography transformation of motion vectors according to the motion of the imaging camera with respect to the moving image frames generated from the imaging camera, and the generated compensation image frames A system for real-time tracking of an object in a dynamic camera image comprising a; an image analysis server that tracks an object detected according to a key point from a location similarity and a shape similarity. According to claim 1, wherein the image analysis server,
a camera motion estimation module for calculating motion vectors according to the motion of the imaging camera with respect to the moving image frames;
a camera motion compensation module for generating compensated image frames from which the motion of the image capturing camera is removed by inverse homography transforming the calculated motion vectors with respect to the moving image frames;
an object detection module that detects the object by recognizing the key point according to each part of the object from the compensation image frames; and
and an object tracking module for tracking the detected object according to the location similarity and appearance similarity. The method of claim 2, wherein the camera motion estimation module comprises:
The pixel values of the moving image frames are converted into gray scale, and a grid composed of grid points disposed at regular intervals in a previous image frame among the moving image frames is generated, and the previous image frame is compared with respect to each of the grid points. A grid point in which the gray scale value change in the current image frame is greater than or equal to a predetermined reference value is selected as a feature point,
The movement vectors of the feature point pair are obtained by estimating the predicted positions of the selected feature points in the next image frame through the pyramid Lucas Kanae algorithm, and the highest frequency movement vectors are obtained by calculating a two-dimensional histogram for the movement vectors of the feature point pair. A system for tracking an object in a dynamic camera image in real time, characterized in that it is calculated as motion vectors according to the motion of the imaging camera. The method of claim 2, wherein the motion compensation module comprises:
calculating a homography matrix for homography-converting feature points in a previous image frame among the moving image frames to a specific position of the current image frame based on the movement vectors according to the movement of the image capturing camera;
The inverse transformation of the calculated homography is applied to all pixels of the current image frame to generate compensated image frames obtained by removing the image capturing camera movement in the current image frame compared to the previous image frame. A system that tracks objects in real time. According to claim 2, wherein the object detection module,
A system for tracking an object in a dynamic camera image in real time, characterized in that extracting a feature vector representing the shape of the detected object according to the recognized key point. According to claim 2, wherein the object tracking module,
estimating a position predicted to exist in the current image frame with respect to the tracking object that was being tracked until the previous image frame among the objects detected by the object detection module using the Kalman filter,
The position similarity is determined by comparing the positions of the candidate objects detected in the current image frame by the object detection module with the positions of the tracking objects estimated using the Kalman filter,
determining the similarity in appearance by comparing the feature vector of the tracking object with the feature vector of the candidate objects,
The system for tracking an object in a dynamic camera image, characterized in that the tracking object is tracked by determining the sameness between the tracking object and the candidate object according to the determined position similarity and the shape similarity. The method of claim 6, wherein the object tracking module,
and determining the identity between the tracking object and the candidate object by assigning a weight according to a predetermined ratio to each of the determined degree of location similarity and the degree of appearance similarity,
When the candidate objects are detected within a predetermined range adjacent to the estimated location of the tracking object, a weight greater than the appearance similarity is given to the location similarity, and the candidate is outside a predetermined range adjacent to the estimated location of the tracking object. A system for tracking an object in a dynamic camera image in real time, characterized in that when objects are detected, a weight greater than that of the location similarity is given to the appearance similarity. transmitting the moving image frames captured by the image capturing camera moving to the camera motion estimation module;
estimating, by the camera motion estimation module, a motion vector of the image capturing camera from the transmitted moving image frames to estimate the motion of the image capturing camera;
generating, by the camera motion compensation module, compensating image frames obtained by removing the estimated motion of the image capturing camera from the moving image frames, and transmitting them to the object detection module;
detecting, by the object detection module, a tracking object in a corresponding compensation image frame among the transmitted compensation image frames, and extracting a feature vector of the detected tracking object;
estimating, by an object tracking module, an expected position of the tracking object in a compensation image frame after the corresponding compensation image frame by using a Kalman filter;
detecting, by the object detection module, at least one or more candidate objects in a compensation image frame subsequent to the corresponding compensation image frame, and extracting a feature vector of the detected candidate object; and
The object tracking module compares the position similarity between the candidate objects and the tracking object detected within or outside the predetermined range adjacent to the estimated expected position and the shape similarity according to the extracted feature vector to determine whether to match. A method of real-time tracking of an object in a dynamic camera image comprising the step of determining. The method of claim 8, wherein the step of estimating, by the camera motion estimation module, the motion of the image capturing camera,
converting pixel values of the transmitted moving image frames into gray scale;
generating a grid composed of grid points arranged at regular intervals with respect to the moving image frames;
selecting, for each of the grid points, a grid point in which a gray scale value change in a current image frame compared to a previous image frame is greater than or equal to a predetermined reference value as a feature point;
estimating the positions of the feature points with respect to a next image frame through a predetermined optical flow method;
obtaining motion vectors of a pair of feature points using the feature point selected in the current image frame and the feature point in a frame next to the estimated next image; and
and calculating a motion vector for the imaging camera by calculating a two-dimensional histogram with respect to the obtained motion vectors to determine motion vectors with the highest frequency. The method of claim 9, wherein the step of generating, by the camera motion compensation module, the compensation image frames and transmitting them to the object detection module, The camera motion compensation module generating the compensation image frames and transmitting the compensation image frames to the object detection module The steps to send to
calculating a homography matrix for converting feature points in the previous image frame to specific positions of the current image frame using the determined highest frequency motion vectors;
generating the compensated image frames from which the image capturing camera movement is removed by applying an inverse transformation of the calculated homograph matrix to pixels in the moving image frame; and
and transmitting the generated compensation image frames to the object detection module.

Images