KR102416825B1 - Apparatus and method for tracking object using skeleton analysis - Google Patents

Abstract

The present invention relates to an apparatus and method for tracking an object using skeleton analysis, and more particularly, by analyzing the joints of an object corresponding to a person in a frame of an image to generate skeleton information, and then based on the skeleton information for each object generated in an existing frame to detect the same object by comparing it with the prediction information for each object generated for the change of the joint point and scale of the object at the position predicted in the next frame, and update the tracking information for the same object to accurately track the same object It relates to an object tracking device and method using skeleton analysis that supports it. The present invention improves the reliability of object tracking because it is possible to accurately track the same object continuously without interruption even when it is difficult to track the object because a large number of objects are clustered and dispersed in the image when the object tracking method depends on the visual characteristics as in the prior art. heightening effect.

Description

Object tracking device and method using skeleton analysis {Apparatus and method for tracking object using skeleton analysis}

The present invention relates to an apparatus and method for tracking an object using skeleton analysis, and more particularly, by analyzing the joints of an object corresponding to a person in a frame of an image to generate skeleton information, and then based on the skeleton information for each object generated in an existing frame to detect the same object by comparing it with the prediction information for each object generated for the change of the joint point and scale of the object at the position predicted in the next frame, and update the tracking information for the same object to accurately track the same object It relates to an object tracking device and method using skeleton analysis that supports it.

Recently, various object tracking algorithms have been developed to identify an object corresponding to a person in an image and continuously track the same object. These algorithms are applied to various systems such as security systems and access control systems to track objects specified by the user While doing so, it is possible to perform a function of detecting an object on which a user-specified event has occurred and providing a notification to the user.

Existing algorithms for tracking objects use a combination of various algorithms such as differential image, SIFT (Scale Invariant Feature Transform), HOG (Histogram of Oriented Gradient), Haar feature, etc. A method of detecting and tracking an object having the same characteristics as the same object is generally used.

However, since most of these existing object tracking algorithms depend on the visual characteristics of the object based on the properties constituting the pixels included in the image, after a large number of people corresponding to the object to be tracked are gathered at the same location and overlapped with each other, In the case of dispersion, when a large number of people are wearing the same clothes, it is difficult to distinguish and track the object, and when the illuminance or quality of the image is changed due to a change in the external environment, it may fail to track the same object. Since it occurs frequently, there is a problem that the reliability of object tracking is lowered.

Korean Patent No. 10-2152318

The present invention identifies the joint of an object corresponding to a person through image analysis and obtains skeleton information while calculating the predicted information on the size of the object and the position of a specific joint at the expected position of the object based on the skeleton information, Reliability in object tracking by tracking the object in a way that judges the same object through comparison of the skeleton information obtained from the next image and the prediction information to accurately detect and track the same object without depending on the visual characteristics of the image The purpose is to support to increase the

In addition, an object of the present invention is to support to further increase the accuracy and reliability of the determination of the same object by additionally comparing the visual characteristics of the object determined as the same object in the previous image and the current image based on the skeleton information.

In the object tracking method using skeleton analysis according to an embodiment of the present invention, the object tracking device receives a frame for an image, and the frame is a learning model trained to generate skeleton information according to joint recognition of an object corresponding to a person. is applied to generate skeleton information including key points for each joint recognized for the object identified in the image and the scale of the object, and assigns a unique identifier to correspond to the identified object to determine the unique identifier and the A skeleton analysis step of generating tracking information including a ground ground location that is a ground ground location of the identified object corresponding to the identified object, and the object tracking device for each object identified in the frame through the skeleton analysis step A prediction step of applying the generated skeleton information to a Kalman filter to generate prediction information for each object, which is a reference point that is a key point corresponding to a neck in the next frame of the frame, and prediction information that is information predicted about the scale of the object; The tracking device compares the skeleton information of a specific object generated by performing the skeleton analysis step on the new frame with the prediction information for each object generated through the prediction step for the existing frame based on the reference point and scale, A matching step of determining whether an object of interest matching the specific object exists among one or more objects corresponding to a frame, and the object tracking device searches the specific object when the object of interest matching the specific object exists through the matching step An update step of updating the tracking information corresponding to the object of interest based on the tracking information generated through the skeleton analysis step may be included.

As an example related to the present invention, in the updating step, when the object tracking device matches the specific object with the object of interest, a unique identifier included in the tracking information corresponding to the specific object in the new frame is added to the object of interest Converts to a unique identifier identical to the unique identifier assigned to , and adds the ground grounding position of the specific object to the tracking information of the object of interest based on the tracking information of the specific object. It may be characterized in that the tracking information of the object of interest is updated to include information on the movement position by time.

As an example related to the present invention, the skeleton analysis step further comprises the step of including, in the tracking information, information about the bounding box generated by the object tracking apparatus setting a bounding box to the identified object, in the tracking information, In the updating step, the object tracking device adds information about the bounding box included in the tracking information of the specific object to the tracking information of the object of interest so that information on the location change of the bounding box of the object of interest over time is included. It may be characterized in that it further comprises the step of updating the tracking information of the object.

As an example related to the present invention, in the matching step, the reference point included in the skeleton information of the specific object generated by the object tracking device for the new frame and the reference point included in the prediction information for each object generated for the existing frame Points are compared for each object according to a preset first equation to calculate a position score for each object, and the scale included in the skeleton information of the specific object and the scale included in the prediction information for each object are in a preset second equation After calculating the scale score for each object by comparing each object according to each object, the final score is calculated by adding up the position score and the scale score for each object. It may be characterized in that it further comprises the step of identifying as the object of interest.

As an example related to the present invention, the first equation is

The Pscore is the location score, and the coordinates of the reference point included in the skeleton information of the specific object and the coordinates of the reference point included in the prediction information of the object generated for the existing frame are the Manhattan distance. It may be characterized in that it is applied to

As an example related to the present invention, the second equation is,

, the Sscore is the scale score, the minScale is the smaller of the scale included in the skeleton information of the specific object and the scale included in the prediction information of the object generated for the existing frame, and the maxScale is the specific It may be characterized as a larger value among a scale included in the object's skeleton information and a scale included in the object's prediction information generated for the existing frame.

As an example related to the present invention, the scale included in the skeleton information is

Based on the reference point, the r _limb is a first key point corresponding to the elbow, a second key point corresponding to the hip, a third key point corresponding to the knee, and a fourth key point corresponding to the ankle. It is a radius calculated on the basis of a pixel for each key point for each joint including a key point, and the C _limb is calculated by the learning model for each key point for each joint except for the reference point included in the skeleton information. It is reliability, and the R _limb may be characterized as a preset weight for each key point for each joint except for the reference point.

As an example related to the present invention, after the updating step, the method may further include an event detection step of generating an event when the updated tracking information corresponding to the object of interest satisfies a preset condition.

As an example related to the present invention, in the matching step, when the object tracking apparatus has a plurality of objects of interest matching the specific object, the boundary area of the specific object is determined based on the skeleton information corresponding to the specific object. Identifies and sets a mask area for the specific object, and sets a mask area for each of the plurality of objects of interest identified as matching the specific object in the existing frame in the same manner as in the setting method of the mask area for the specific object setting, and calculating a Gaussian score for the ratio of pixels belonging to a preset band among a plurality of Gaussian distributions calculated through SMOG-based Gaussian modeling for a plurality of pixels included in the mask region of the specific object, and After calculating a Gaussian score in the same manner as for calculating the Gaussian score for the mask region of the specific object for the mask region for each object of interest of The method may further include a visual comparison step of selecting an object having the smallest difference between the specific object and the Gaussian score as the object of interest matching the specific object.

The object tracking apparatus using skeleton analysis according to an embodiment of the present invention receives a frame for an image, and applies the frame to a learning model learned to generate skeleton information according to joint recognition of an object corresponding to a person. Generates skeleton information including a key point for each joint recognized for the object identified in the image and the scale of the object, and assigns a unique identifier to correspond to the identified object A skeleton analysis unit that generates tracking information including a ground ground position, which is a grounding position on the ground, corresponding to the identified object, and the skeleton information generated for each object identified in the frame through the skeleton analysis unit is applied to a Kalman filter In the next frame of the frame, a prediction unit generating prediction information for each object, which is a reference point, which is a key point corresponding to the neck, and prediction information, which is information predicted on the scale of an object, and a specific generated through the skeleton analysis unit for a new frame An object of interest matching the specific object exists among one or more objects corresponding to the existing frame by comparing the skeleton information of the object with the prediction information for each object generated through the prediction unit with respect to the existing frame based on the reference point and scale. If there is an object of interest that matches the specific object through a matching unit to determine whether or not to update the tracking information corresponding to the object of interest based on the tracking information generated through the skeleton analysis unit for the specific object It may include updates.

The object tracking apparatus according to an embodiment of the present invention generates skeleton information through joint and scale analysis for a specific object identified in a frame and tracking information for object tracking every time a frame constituting an image is received, while generating the skeleton information of the object. Based on the Kalman filter, the predicted movement position and the scale change of the object expected for the key point corresponding to the neck of the specific object in the next frame are predicted through the Kalman filter, and the specific object in the existing frame in the next frame It is possible to accurately track the same object by updating the tracking information of the same object with the tracking information of the specific object while detecting the object for which the skeleton information closest to the prediction information generated for the object is generated as the same object as the specific object, , through this, in the case of an object tracking method that relies on visual characteristics as in the past, it is possible to accurately track the same object continuously without interruption even when it is difficult to track an object because a large number of objects are clustered and dispersed in the image, thus increasing the reliability of object tracking. heightening effect.

In addition, the present invention further compares the visual characteristics of the object determined as the same object in the previous frame and the current frame based on the skeleton information to verify the result of the same object determination, thereby further increasing the accuracy and reliability of the same object determination It works.

1 is a block diagram of an object tracking apparatus using skeleton analysis according to an embodiment of the present invention.
2 to 6 are diagrams illustrating an operation of an object tracking apparatus using skeleton analysis according to an embodiment of the present invention.
7 is a flowchart of an object tracking method of an object tracking apparatus using skeleton analysis according to an embodiment of the present invention.

Hereinafter, a detailed embodiment of an object tracking apparatus (hereinafter referred to as an object tracking apparatus) using a skeleton analysis according to an embodiment of the present invention will be described with reference to the drawings.

1 is a block diagram of an object tracking apparatus 100 according to an embodiment of the present invention, and FIG. 2 is an exemplary operation diagram of the object tracking apparatus 100 .

First, as shown in FIG. 1 , the object tracking apparatus 100 may include a communication unit 110 , a storage unit 130 , and a control unit 120 .

The communication unit 110 may be connected to communicate with an external device through a communication network to receive an image of a space to be monitored.

In this case, the communication network described in the present invention may include a wired/wireless communication network, and as an example of such a wireless communication network, a wireless LAN (WLAN), a DLNA (Digital Living Network Alliance), a WiBro (Wireless Broadband: Wibro), WiMAX (World Interoperability for Microwave Access: Wimax), GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice- Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-A (LTE-A) Advanced), Wireless Mobile Broadband Service (WMBS), 5G mobile communication service, Bluetooth (Bluetooth), LoRa (Long Range), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct may be included. In addition, wired communication networks include wired LAN (Local Area Network), wired WAN (Wide Area Network), Power Line Communication (PLC), USB communication, Ethernet, serial communication, optical/coaxial A cable may be included.

In addition, the external device may be a camera or an image storage device for storing images (eg, a digital video recorder (DVR) or a network video recorder (NVR)).

In this case, the object tracking apparatus 100 may be configured as a component of the external device.

In addition, the control unit 120 may perform an overall control function of the object tracking device 100 , and the control unit 120 may include RAM, ROM, CPU, GPU, and a bus, and may include RAM, ROM, and CPU. , GPU, etc. may be connected to each other through a bus.

In addition, the object tracking apparatus 100 may be configured to further include a storage unit 130 , and the storage unit 130 stores various programs and data necessary for the operation of the control unit 120 described in the present invention. may be stored, or the image may be stored.

In this case, the storage unit 130 and the communication unit 110 may be included as components of the control unit 120 .

Meanwhile, as shown in FIGS. 1 and 2 , the control unit 120 controls the skeleton analysis unit 121 , the prediction unit 122 , the matching unit 123 , the update unit 124 , and the event detection unit 125 . may be included.

First, as shown in FIG. 3 , the skeleton analyzing unit 121 receives a frame constituting an image, and when receiving the frame, identifies an object corresponding to a person in the frame while identifying the identified object. A key point and object for each joint recognized for the object identified in the frame by applying the learning model preset to the skeleton analysis unit 121 after learning to generate skeleton information according to joint recognition is completed Skeleton information including the scale of may be generated.

In this case, the learning model may be composed of a deep learning algorithm, and the deep learning algorithm may be composed of one or more neural network models.

In addition, the neural network model (or neural network) described in the present invention may be composed of an input layer, one or more hidden layers, and an output layer, and the neural network model includes a Deep Neural Network (DNN). , Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), etc., can be applied to various types of neural networks.

That is, the learning model can identify an object corresponding to a person in the frame, identify a point corresponding to a joint with respect to the identified object, and set coordinates for a point corresponding to the joint as the key point. have. In this case, the coordinates may have an x-coordinate value and a y-coordinate value.

In addition, the key points for each joint may be connected to each other by a line between adjacent key points and included in the skeleton information.

In addition, the learning model may calculate the reliability (or joint reliability) of the coincidence probability with the actual joint (or the position of the actual joint) for each of one or more joints identified in the object corresponding to the person, and the skeleton The analysis unit 121 may match the reliability calculated by the learning model for each key point of each joint included in the skeleton information, and include the reliability for each joint in the skeleton information.

In addition, the learning model uses each of the one or more joints identified in the object as a key point corresponding to a neck, a key point corresponding to an elbow, a key point corresponding to a hip, and a knee. A key point for each joint may be generated so that a key point corresponding to , and a key point corresponding to an ankle are distinguished.

In this case, the key point corresponding to the neck may be an intermediate point of both shoulders identified in the object corresponding to the person.

As an example, the learning model includes label information labeled on the key point in the skeleton information to identify which joint of the neck, elbow, hip, knee and ankle is the key point for each key point for each joint. can

That is, the key point may be generated to be able to identify the key point for which joint.

In this case, the key point corresponding to the elbow, the key point corresponding to the hip, the key point corresponding to the knee, and the key point corresponding to the ankle may be generated in plurality.

In the above configuration, as for the key points for each joint, when a part of the object in the frame does not appear in the frame, some of the key points for each joint may be omitted.

Also, the skeleton analyzer 121 may generate information on the scale of the object based on key points for each joint calculated by the learning model.

For example, the skeleton analyzer 121 sets a key point corresponding to the neck as a reference point, and based on the reference point, the radius for each of the key points for each one or more joints identified in the frame except for the reference point. Calculated, weighted in advance in the skeleton analysis unit 121 for the identified one or more joints for each joint, and the reliability obtained in response to the joint by weighting the radius of the joint to obtain a weighted reflection value for each joint Then, the scale of the object can be calculated by averaging the weight reflection values for all joints.

In this way, the skeleton analysis unit 121 calculates a radius for each key point of one or more joints identified in the frame except for the reference point, and a weight and reliability corresponding to the joint in the radius calculated for each joint The scale may be calculated by weighted average of .

In this case, the radius may be a distance between the reference point and a key point of a joint other than the reference point.

As a detailed example of this, the skeleton analysis unit 121 is the first joint corresponding to the neck, the second joint corresponding to the elbow, and the third joint corresponding to the hip are identified in the object in the first joint. After setting the corresponding key point as a reference point, a preset weight corresponding to the second joint and a preset weight corresponding to the second joint and a radius between the key point corresponding to the second joint and the reference point based on the reference point A first weight reflection value is obtained by weighting a corresponding reliability (included in the skeleton information), and a radius between the key point corresponding to the third joint and the reference point based on the reference point corresponds to the third joint to obtain a second weight reflection value by weighting a preset weight and reliability corresponding to the third joint, and calculate the scale of the object by averaging the first weight reflection value and the second weight reflection value .

At this time, as shown in FIG. 4 , the skeleton analyzer 121 may include preset weighting information for each joint, and one of a plurality of weights preset in the setting information corresponding to each key point for each joint. The weight of the key point corresponding to the ankle may be set to the highest value.

The reason that the weight corresponding to the ankle is set to the highest value is that the key point corresponding to the ankle has the longest distance from the key point corresponding to the neck, which is the reference point, and thus has the lowest uncertainty and dispersion compared to key points in other joints. This is because the scale of the object can be reflected with the highest accuracy.

In addition, as the weights for each joint, the weight corresponding to the joint for the elbow is set to 0.273, the weight corresponding to the joint to the hip is set to 0.416, the weight corresponding to the joint to the knee is set to 0.713, Preferably, the weight corresponding to the joint for the ankle is set to 1.0 in the setting information.

In addition, the skeleton analyzer 121 may allocate a unique identifier for distinguishing (distinguish) from other objects corresponding to the object identified in the frame, and the identification on the unique identifier and the ground identified in the frame. Tracking information including a ground grounding position, which is a grounding location of the object, may be generated corresponding to the identified object.

In this case, the skeleton analyzer 121 may set a unique identifier included in the tracking information to the skeleton information corresponding to the tracking information.

In addition, the object identified corresponding to the frame described in the present invention may mean an object in which the skeleton information is generated.

In the above configuration, the learning model set in the skeleton analyzer 121 may calculate key points for other joints in addition to the joints, but the skeleton analyzer 121 is configured for each preset joint included in the skeleton information. Based on the label information as key points, a key point corresponding to a neck, a key point corresponding to an elbow, a key point corresponding to a hip, a key point corresponding to a knee, and an ankle Only key points corresponding to (ankle) are included, and a scale can be calculated based on these key points for each joint.

That is, the skeleton analyzer 121 may use only key points for a plurality of preset joints for object tracking.

Meanwhile, the prediction unit 122 works with the skeleton analyzer 121 to apply the skeleton information generated for each object identified in the frame through the skeleton analyzer 121 to a Kalman filter. Thus, in the next frame of the frame, the reference point, which is a key point corresponding to the neck, and prediction information, which is information predicted for the scale of the object, may be generated for each identified object.

For example, the prediction unit 122 corresponds to a reference point that is a key point corresponding to a neck of a specific object identified in a specific frame through the Kalman filter, and predicts corresponding to the specific object in the next frame of the specific frame. The predicted position (predicted movement position) of the reference point and the predicted value for the scale (scale change) of the specific object predicted in the next frame corresponding to the scale of the specific object included in the skeleton information calculated in the specific frame It is possible to generate prediction information including

In this case, the prediction unit 122 may use three Kalman filters, and apply the x-coordinate of the reference point to the first Kalman filter among the three Kalman filters to obtain the x-coordinate of the reference point predicted in the next frame. calculating, applying the y-coordinate of the reference point to a second Kalman filter among the three Kalman filters to calculate the y-coordinate of the reference point predicted in the next frame, and setting the scale (value) of the object to the three A prediction scale (value) of the object predicted in the next frame may be calculated by applying the third Kalman filter among the Kalman filters.

As described above, the skeleton analyzer 121 generates skeleton information and tracking information for each object included in the frame every time a frame is received, and the prediction unit 122 is the object identified by the skeleton analyzer 121. Correspondingly, prediction information may be generated.

In addition, the skeleton analyzer 121 or the prediction unit 122 matches the skeleton information generated corresponding to the specific object identified in the specific frame with the tracking information and the prediction information, and then the specific frame or the frame of the specific frame The identification information may be matched and stored in the storage unit 130 .

In this case, the skeleton analysis unit 121 or the prediction unit 122 may store the prediction information in the storage unit 130 by including the prediction information in the tracking information corresponding to the prediction information on a frame basis.

In addition, the skeleton analyzer 121 or the prediction unit 122 may provide the matching unit 123 with information about the skeleton generated corresponding to the object identified in the frame whenever the frame is received.

On the other hand, the matching unit 123 corresponds to the specific object identified in the new frame included in the image by the skeleton analysis unit 121 or the prediction unit 122, the skeleton generated through the skeleton analysis unit 121 information can be received.

In addition, the matching unit 123 compares the skeleton information of the specific object with the prediction information for each object generated through the prediction unit 122 with respect to the existing frame generated before the generation of the new frame and the reference point and scale. Based on the comparison, it may be determined whether an object of interest matching the specific object exists among one or more objects corresponding to the existing frame.

For example, the matching unit 123 stores the skeleton information of a specific object identified in the new frame and the prediction information for each object generated corresponding to the previous frame, which is an existing frame generated just before the generation of the new frame, in the storage unit ( 130) can be extracted.

In this case, the existing frame may be a frame generated before the generation of the previous frame.

In addition, the matching unit 123 compares the reference point included in the skeleton information corresponding to the specific object identified in the new frame and the scale of the specific object to the object predicted in the new frame for each object identified in the previous frame. It is possible to determine whether an object of interest, which is an object matching the specific object, exists among the one or more objects identified in the existing frame by comparing the prediction information for each object including the reference point and the scale of the object with the same properties.

Hereinafter, a detailed operation configuration of the matching unit 123 for identifying an object of interest matching the specific object will be described.

First, the matching unit 123 sets the reference point included in the skeleton information of the specific object generated for the new frame and the reference point included in the prediction information for each object generated for the existing frame to the preset Equation 1 below. Accordingly, by comparing for each object identified in the existing frame, it is possible to calculate a position comparison score according to the comparison of a reference point with the specific object for each object identified in the existing frame.

In this case, the Pscore is the position comparison score, the coordinates of the reference point included in the skeleton information of the specific object, and the coordinates of the reference point included in the prediction information of the object generated for the existing frame (identified in the existing frame) may be applied to the Manhattan distance.

As the position comparison score is lower, the difference between the position of the reference point predicted in the new frame for the first object identified in the existing frame corresponding to the scale comparison score and the reference point of the second object identified in the new frame is Since it means small, it may be determined that there is a high probability that the first object and the second object are the same object.

In addition, the matching unit 123 compares the scale included in the skeleton information of the specific object with the scale included in the prediction information for each object for each object identified in the existing frame according to Equation 2 below, which is set in advance. A scale comparison score for each object according to a scale comparison with the specific object for each object identified in the frame may be calculated.

At this time, the Sscore is the scale comparison score, and the minScale is the smaller of the scale included in the skeleton information of the specific object and the scale included in the prediction information of the object generated (identified in the existing frame) for the existing frame. value, and the maxScale may be a larger value among a scale included in the skeleton information of the specific object and a scale included in the prediction information.

The lower the scale comparison score, the smaller the difference between the scale predicted in the new frame for the first object identified in the existing frame corresponding to the scale comparison score and the scale of the second object identified in the new frame Therefore, it may be determined that the probability that the first object and the second object are the same object is high.

In the above configuration, a scale (or a scale value) included in the skeleton information may be calculated through Equation 3 below.

In this case, the r _limb includes a first key point corresponding to the elbow, a second key point corresponding to the hip, a third key point corresponding to the knee, and a fourth key point corresponding to the ankle, based on the reference point. It is a radius calculated on the basis of pixels for each key point for each joint including, and the C _limb is calculated by the learning model and for each key point for each joint except for the reference point included in the skeleton information. Reliability (joint reliability), and the R _limb may be a preset weight for each key point for each joint except for the reference point.

Here, when at least one of the first to fourth keypoints is not identified in the frame corresponding to the skeleton information, the corresponding keypoint may be excluded from Equation 3 above.

In addition, a preset weight for each key point for each joint may be included in the setting information.

Next, the matching unit 123 may calculate a final score by adding the position comparison score and scale comparison score obtained by comparing the prediction information of the object identified in the existing frame with the skeleton information of the specific object, and this By adding the position comparison score and scale comparison score obtained according to the comparison with the specific object for each one or more objects identified in the existing frame, a final score for each of the identified one or more objects may be calculated.

Also, the matching unit 123 may identify an object having the lowest final score and less than or equal to a preset reference value among one or more objects identified in the existing frame as the object of interest matching the specific object.

That is, the matching unit 123 determines the key point corresponding to the neck predicted according to the movement of the existing object in the new frame based on the skeleton information generated corresponding to the existing frame for each existing object identified in the existing frame. By comparing the scale with the reference point and the scale of the specific object, which are key points corresponding to the neck obtained for the specific object in the new frame, the reference point and scale of the specific object among the one or more existing objects identified in the existing frame are the most An existing object for which adjacent prediction information is generated may be identified as an object matching the specific object.

At this time, if the lowest final score exceeds a preset reference value (or first reference value), the matching unit 123 may determine the object corresponding to the lowest final score as an object that is not identical to the specific object, , it may be determined that the same object as the specific object is not detected in the existing frame.

Alternatively, the matching unit 123 may be configured to determine if the position comparison score among the objects identified in the existing frame is greater than a preset first set value (exceeds a first set value) greater than a first set value (first set value). The object corresponding to the location comparison score (which exceeds the set value) may be determined as an object that is not identical to (does not match) the specific object.

Alternatively, when the scale comparison score is greater than a preset second set value (exceeds a second set value), the matching unit 123 sets a scale larger than the second set value (exceeds the second set value). The object corresponding to the comparison score may be determined as an object that is not identical to (does not match) the specific object.

On the other hand, the update unit 124 works with the matching unit 123 to correspond to the specific object identified in the new frame, and the matching unit 123 detects an object of interest matching the specific object in the existing frame. In this case, tracking information of the specific object generated through the skeleton analysis unit 121 for the specific object may be extracted from the storage unit 130 .

Also, when the object of interest determined to match the specific object is detected, the update unit 124 converts the tracking information corresponding to the object of interest stored in the storage unit 130 to the tracking information generated in response to the specific object. can be updated (updated) based on

At this time, when the specific object and the object of interest coincide with each other according to the determination of the matching unit 123, the update unit 124 returns a unique identifier included in the tracking information corresponding to the specific object in the new frame. The unique identifier assigned to the object of interest (unique identifier included in the tracking information of the object of interest) and the same unique identifier are converted and matched, and the ground included in the tracking information of the specific object based on the tracking information of the specific object By adding the grounding position to the tracking information of the object of interest, the tracking information of the object of interest may be updated (or updated) to include information on the moving position of the object of interest by time, which is the same as the specific object.

In addition, when the update unit 124 updates the tracking information of the object of interest based on the tracking information of the specific object, the tracking information of the specific object is not stored in the storage unit 130 , but the storage unit 130 . ) can be deleted.

In addition, in the above configuration, the matching unit 123 does not have an object identified in one or more existing frames, so prediction information of an object corresponding to the existing frame does not exist, or corresponds to one or more different existing frames generated at different times If there is no object of interest matching the specific object identified in the new frame among the one or more identified objects, the specific object is determined as a new object, and the updater 124 interworks with the matching unit 123 . Accordingly, the tracking information of the specific object determined as a new object may be stored in the storage unit 130 while maintaining a unique identifier assigned to the tracking information of the specific object.

In this case, the matching unit 123 may compare the specific object with the existing object by using only a preset number of existing frames generated in the past based on the new frame corresponding to the specific object as comparison targets, and the comparison target is one If there is no object of interest matching the specific object in the above existing frame, the specific object may be determined as a new object.

According to the above-described configuration, as shown in FIG. 5 , the controller 120 of the object tracking apparatus 100 transmits tracking information of an object of interest initially identified in an existing frame in a plurality of frames received after the existing frame. Whenever an object matching the object of interest is detected, the tracking information of the object of interest may be updated as described above based on the detected object-specific tracking information generated corresponding to the plurality of frames, respectively.

Through this, the controller 120 can accurately track the same object detected in a plurality of frames, and generate tracking information including the location of the same object over time.

In addition, the tracking information updated for the object of interest includes the unique identifier of the object of interest, the current position where the object of interest touches the ground, the history of the ground contact position of the object of interest up to the most recently updated time point (history), and the most It may include a movement path according to one or more ground grounding positions added up to the time of the most recent update.

On the other hand, the event detection unit 125 compares the tracking information for each object stored in the storage unit 130 with a preset event occurrence condition to detect a specific object (or event occurrence object) corresponding to the tracking information satisfying the event occurrence condition. At the time, event information including a unique identifier of the specific object (event generating object) may be generated.

That is, the event detector 125 may generate an event when the updated tracking information corresponding to the object of interest satisfies a preset condition.

In addition, the event detection unit 125 may transmit the event information to a preset external terminal through the communication unit 110 when generating the event information or output the event information through an output device connected to the communication unit 110 . have.

Through the above-described configuration, the object tracking apparatus 100 according to an embodiment of the present invention receives skeleton information through joint and scale analysis for a specific object identified in the frame each time a frame constituting an image is received, and tracking information for object tracking Generates prediction information that predicts the movement position and the scale change of the object expected for the key point corresponding to the neck of the specific object in the next frame through the Kalman filter based on the skeleton information of the object while generating In a frame, while identifying an object in which the skeleton information closest to the prediction information generated for the specific object in the existing frame is generated as the same object as the specific object, the tracking information of the same object is updated with the tracking information of the specific object In this way, the same object can be accurately tracked, and through this, in the case of an object tracking method that relies on visual characteristics as in the past, even when it is difficult to track an object because a large number of objects are clustered and dispersed in the image, the same object is continuously and accurately tracked without interruption. This can increase the reliability of object tracking.

That is, the present invention corresponds to a specific object identified in a specific frame constituting an image with respect to a key point corresponding to the neck of the specific object, the predicted position predicted in the next frame of the specific frame, and the neck of the specific object After generating prediction information including the expected scale for the scale change predicted in the next frame with respect to the scale for the distribution pattern of the key point for each other joint of the specific object based on the key point, the prediction information and the prediction information in the next frame By detecting the object for which the closest skeleton information is generated as the same object as the specific object, as in the past, when tracking an object depending on the visual characteristics based on the information of individual pixels, the object with similar visual characteristics is a specific frame Even when it is difficult to distinguish which object is identified in an existing frame because there are multiple It can increase the reliability of object tracking.

On the other hand, in the above configuration, as shown in Fig. 5, the skeleton analyzer 121 sets a bounding box to the object identified in the frame and provides information on the bounding box generated by the identified It can be included in the tracking information corresponding to the object.

In addition, the update unit 124 adds information on the bounding box included in the tracking information of the specific object identified in the new frame to the tracking information of the object of interest, information on the location change of the bounding box of the object of interest over time The tracking information of the object of interest may be updated to include .

Meanwhile, the skeleton analyzer 121 may set an initial value for prediction reliability, which is a reliability value for the prediction information, in the tracking information of the object determined as a new object.

In addition, the matching unit 123 does not match all of the one or more objects identified in the new frame according to the comparison between the skeleton information for each object identified in the new frame and the prediction information for each object among the objects identified in the existing frame. A preset score may be subtracted from the prediction reliability included in the tracking information of the non-existent object.

In addition, the matching unit 123 may add a preset score to the prediction reliability included in the tracking information of the object corresponding to the existing frame determined to match any one of the one or more objects identified in the new frame. .

In addition, the update unit 124 returns tracking information of an object that does not match all of the one or more objects identified in the new frame among the objects identified in the existing frame based on the determination result of the matching unit 123 of the corresponding object. It can be updated based on the prediction information.

In this case, when the update unit 124 updates the tracking information based on the prediction information, the prediction information used to update the tracking information may be prediction information corresponding to the previous frame generated immediately before the new frame corresponding to the tracking information. .

In addition, the update unit 124 checks the tracking information for each object stored in the storage unit 130 and receives the tracking information in at least some or all of a preset number of subsequent frames received after the initial tracking information is generated for the existing object. If the same object as the existing object is not detected and the prediction reliability of the existing object is less than or equal to a preset reference reliability, the tracking information of the existing object may be deleted from the storage unit 130 .

As described above, in the present invention, after the skeleton information and tracking information are generated for a specific object in a specific frame, the specific object moves to a potential movement position of the specific object predicted for each of the preset frames after the specific frame. If the same object as the specific object is not continuously detected because it does not actually move, it is possible to exclude an object with low tracking accuracy by continuously reducing the reliability of the tracking of the specific object, thereby increasing the reliability of the tracking of the object. can

On the other hand, in addition to the above-described configuration, the object tracking apparatus 100 according to an embodiment of the present invention provides an object of a new frame and an object of an existing frame determined to be the same object through comparison using a key point and a scale corresponding to the neck of the object. A visual comparison can be performed to further increase the accuracy of whether the objects are the same between them, which will be described in detail.

The matching unit 123 identifies a boundary region of the specific object based on the skeleton information corresponding to the specific object when a plurality of objects of interest identical to the specific object identified in the new frame exist in the existing frame. Thus, a mask area for the specific object can be set.

In this case, the matching unit 123 may set the mask area as shown in FIG. 5 .

Also, the matching unit 123 sets a mask area for each of the plurality of objects of interest identified as matching the specific object of the new frame in the existing frame in the same manner as in the setting method of the mask area for the specific object. can

In addition, the matching unit 123 is configured to select a plurality of pixels included in a mask region of a specific object identified in the new frame in advance among a plurality of Gaussian distributions calculated through spatial mixture of Gaussians (SMOG)-based Gaussian modeling. A Gaussian score for the ratio of pixels belonging to the set band may be calculated.

In addition, the matching unit 123 calculates a Gaussian score for the plurality of object-specific mask regions in the same manner as for calculating the Gaussian score for the mask region of the specific object, and then a Gaussian corresponding to the specific object. score can be compared.

Also, the matching unit 123 may select an object having the smallest difference between the specific object and the Gaussian score among the plurality of objects of interest as the object of interest matching the specific object.

In this case, the Gaussian score may be calculated through Equation 4 below.

The SMOGscore may be a Gaussian score, and the pixel_match may be a ratio of pixels included in the plurality of preset bands among a plurality of pixels constituting the mask area.

In the above configuration, as shown in FIG. 6 , the matching unit 123 may generate eight Gaussian distributions by sampling a plurality of pixels included in the mask area based on the SMOG. Each Gaussian distribution is generated by sampling the mean and variance of the following four values.

1. R (Normalized Redness)

2. G (Normalized Greeness)

3. I (Intensity)

4. Y (Relative Normalized Y-Coordinate)

In this case, the Y value represents a spatial component. At initialization (the first frame in which skeleton information is first generated for a specific object), SMOG is initialized by dividing the mask area into 8 bands vertically.

Also, the matching unit 123 calculates the average and variance of the RGIY (the four values) for all pixels included in each of the eight bands through the SMOG.

In addition, as shown in Equation 5 below, the matching unit 123 compares the position comparison score with the specific object according to the above-described comparison with the specific object for each existing object identified in the existing frame with respect to the specific object of the new frame as shown in Equation 5 below. Calculating a sum score by adding the scale comparison score and the Gaussian score, and determining an existing object whose sum score is less than or equal to a preset reference value (or second reference value) and has the lowest sum score as the same object as the specific object. can do.

In this case, the score is a summed score, the Pscore _w is a position comparison score, the Sscore _w is a scale comparison score, and the SMOGscore _w is a Gaussian score.

As described above, the present invention verifies the result of determining the same object by additionally comparing the visual characteristics of the object determined to be the same object in the previous frame and the current frame based on the skeleton information, thereby providing accuracy and reliability for the determination of the same object. can be further increased.

7 is a flowchart of an object tracking method of the object tracking apparatus 100 according to an embodiment of the present invention.

As shown, the object tracking apparatus 100 receives a frame for an image (S1), and applies the frame to a learning model trained to generate skeleton information according to joint recognition of an object corresponding to a person to apply the image. Generates skeleton information including the recognized joint-specific key point and the scale of the object for the object identified in (S2), and assigns a unique identifier to the identified object to correspond to the identified object, the unique identifier and the identified A skeleton analysis step of generating tracking information including a ground ground location that is a ground ground location of an object corresponding to the identified object may be performed (S3).

In addition, the object tracking apparatus 100 applies the skeleton information generated for each object identified in the frame through the skeleton analysis step to a Kalman filter, a reference point that is a key point corresponding to the neck in the next frame of the frame, and A prediction step of generating prediction information, which is information predicted with respect to the scale of the object, for each object may be performed (S4).

In addition, the object tracking apparatus 100 compares the skeleton information of a specific object generated by performing the skeleton analysis step on the new frame, the prediction information for each object generated through the prediction step on the existing frame, and the reference point and scale A matching step of determining whether an object of interest matching the specific object exists among one or more objects corresponding to the existing frame may be performed by comparing based on ( S5 , S6 ).

In addition, if there is an object of interest matching the specific object through the matching step (S7), the object tracking apparatus 100 determines the interest based on the tracking information generated through the skeleton analysis step for the specific object An update step of updating the tracking information corresponding to the object may be performed (S8).

The various devices and components described herein may be implemented by hardware circuitry (eg, CMOS-based logic circuitry), firmware, software, or a combination thereof. For example, it may be implemented using transistors, logic gates, and electronic circuits in the form of various electrical structures.

Those of ordinary skill in the art to which the present invention pertains may modify and modify the above-described contents without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: object tracking device 110: communication unit
120: control unit 121: skeleton analysis unit
122: prediction unit 123: matching unit
124: update unit 125: event detection unit
130: storage

Claims (10)

The object tracking device receives a frame for an image, and applies the frame to a learning model trained to generate skeleton information according to joint recognition of an object corresponding to a person, and a key for each joint recognized for the object identified in the image Generates skeleton information including a point and a scale of an object, and assigns a unique identifier to correspond to the identified object. Tracking including the unique identifier and the grounding position that is the grounding position of the identified object on the ground a skeleton analysis step of generating information corresponding to the identified object;
The object tracking device applies the skeleton information generated for each object identified in the frame through the skeleton analysis step to a Kalman filter, a reference point that is a key point corresponding to the neck in the next frame of the frame, and the scale of the object a prediction step of generating prediction information, which is predicted information, for each object;
The object tracking device compares the skeleton information of a specific object generated by performing the skeleton analysis step for the new frame with the prediction information for each object generated through the prediction step for the existing frame based on the reference point and scale a matching step of determining whether an object of interest matching the specific object exists among one or more objects corresponding to the existing frame; and
When the object tracking device matches the specific object through the matching step, if the object of interest exists, tracking information corresponding to the object of interest based on the tracking information generated through the skeleton analysis step for the specific object update steps to update
An object tracking method using skeleton analysis, including a.
The method according to claim 1,
The update step is
When the object tracking device matches the specific object with the object of interest, the unique identifier included in the tracking information corresponding to the specific object in the new frame is converted into a unique identifier identical to the unique identifier assigned to the object of interest. The object of interest to match, and to include the information on the moving position of the same object of interest by time as the specific object by adding the ground grounding position of the specific object to the tracking information of the object of interest based on the tracking information of the specific object An object tracking method using skeleton analysis, characterized in that updating the tracking information of.
3. The method according to claim 2,
The skeleton analysis step is
Further comprising the step of including, by the object tracking device, information about the bounding box generated by setting a bounding box to the identified object in the tracking information,
The update step is
Tracking the object of interest so that the object tracking device adds information about a bounding box included in the tracking information of the specific object to the tracking information of the object of interest to include information on a change in the location of the bounding box of the object of interest over time Object tracking method using skeleton analysis, characterized in that it further comprises the step of updating the information.
The method according to claim 1,
The matching step is
The object tracking apparatus compares the reference point included in the skeleton information of the specific object generated for the new frame and the reference point included in the prediction information for each object generated for the existing frame for each object according to a preset first equation to calculate the position score for each object, and compare the scale included in the skeleton information of the specific object with the scale included in the prediction information for each object for each object according to a preset second equation to calculate the scale score for each object Computing the final score by adding the position score and the scale score for each object, and identifying an object having the lowest final score and a preset reference value or less among the objects identified in the existing frame as the object of interest Object tracking method using skeleton analysis characterized.
5. The method according to claim 4,
The first equation is

The Pscore is the location score, and the coordinates of the reference point included in the skeleton information of the specific object and the coordinates of the reference point included in the prediction information of the object generated for the existing frame are the Manhattan distance. Object tracking method using skeleton analysis, characterized in that applied to.
5. The method according to claim 4,
The second equation is,

, the Sscore is the scale score, the minScale is the smaller of the scale included in the skeleton information of the specific object and the scale included in the prediction information of the object generated for the existing frame, and the maxScale is the specific An object tracking method using skeleton analysis, characterized in that it is a larger value among a scale included in the object's skeleton information and a scale included in the object's prediction information generated for the existing frame.
The method according to claim 1,
The scale included in the skeleton information is

is calculated through
The r _limb includes a first key point corresponding to the elbow, a second key point corresponding to the hip, a third key point corresponding to the knee, and a fourth key point corresponding to the ankle, based on the reference point A radius calculated on the basis of a pixel for each key point for each joint, and the C _limb is the reliability of each key point for each joint except for the reference point calculated by the learning model and included in the skeleton information, the R The _limb is an object tracking method using skeleton analysis, characterized in that the weight is preset for each key point for each joint except for the reference point.
The method according to claim 1,
After the update step,
An event detection step of generating an event when the updated tracking information corresponding to the object of interest satisfies a preset condition
Object tracking method using skeleton analysis, characterized in that it further comprises.
The method according to claim 1,
The matching step is
If the object tracking apparatus has a plurality of objects of interest matching the specific object, identifying a boundary area of the specific object based on the skeleton information corresponding to the specific object and setting a mask area for the specific object, A mask area is set for each of the plurality of objects of interest identified as matching the specific object in the existing frame in the same manner as in the setting method of the mask area for the specific object, and a plurality of objects included in the mask area of the specific object are set. Calculates a Gaussian score for the ratio of pixels belonging to a preset band among a plurality of Gaussian distributions calculated through SMOG-based Gaussian modeling for pixels of After calculating the Gaussian score in the same manner as in the method of calculating the Gaussian score for the region, the Gaussian score is compared with the Gaussian score corresponding to the specific object, and the difference between the Gaussian score and the specific object among the plurality of objects of interest is the smallest Object tracking method using skeleton analysis, characterized in that it further comprises a visual comparison step of selecting an object as an object of interest matching the specific object.
Receives a frame for an image, and applies the frame to a learning model trained to generate skeleton information according to joint recognition of an object corresponding to a person. Generates skeleton information including a scale, assigns a unique identifier to correspond to the identified object, and identifies tracking information including the unique identifier and a ground ground position that is a grounding position on the ground of the identified object. a skeleton analysis unit that generates a corresponding object;
By applying the skeleton information generated for each object identified in the frame through the skeleton analysis unit to a Kalman filter, the reference point, which is a key point corresponding to the neck in the next frame of the frame, and prediction information that is information predicted about the scale of the object a prediction unit generating for each object;
One or more objects corresponding to the existing frame by comparing the skeleton information of a specific object generated through the skeleton analysis unit for a new frame with the prediction information for each object generated through the prediction unit for the existing frame based on the reference point and scale a matching unit that determines whether there is an object of interest matching the specific object among the objects; and
If there is an object of interest matching the specific object through the matching unit, an update unit that updates tracking information corresponding to the object of interest based on the tracking information generated through the skeleton analyzer for the specific object
An object tracking device using a skeleton analysis comprising a.

Images