KR101837027B1 - Device and method for tracking object using superpixel based on thermal image - Google Patents

Abstract

Disclosed are a method and apparatus for tracking an object using a super pixel based on a thermal image. The method for tracking an object using a super pixel based on a thermal image according to an embodiment of the present invention includes the steps of: collecting a thermal image for each frame; dividing the thermal image into a plurality of super pixels in a segment unit; giving a label to identify a segment corresponding to the object among segments of a reference frame; calculating a cost function between the segment of the following frame and the segment corresponding to the object; and giving the label to the segment corresponding to the object of the following frame based on the calculation value of the cost function. Accordingly, the present invention can accurately recognize and monitor the object.

Description

TECHNICAL FIELD [0001] The present invention relates to an apparatus and method for tracking an object using super-pixels based on thermal imaging,

The present invention relates to an apparatus and method for tracking an object using super-pixels based on thermal imaging. The present invention also relates to a monitoring system of an object comprising a cost axis based on thermal imaging.

Generally, in livestock farms that feed on livestock such as chickens, ducks, pigs, dogs, cows, horses and other livestock, it is necessary to timely change the biological changes such as physiological changes, disease occurrence, It is important to identify and take necessary measures.

On the other hand, a method has been disclosed in which bio-equipment is attached to each of the livestock in the prior art to grasp the living body changes of the livestock. The prior art has a problem in that the bio-equipment must be attached to each of a large number of livestock, so that an economic burden is imposed and the biometric data must be collectively collected and analyzed, so that it is impossible to immediately grasp the biometric data and immediately take measures therefor.

In addition, in order to grasp the living body change of the livestock through the living body change data, there is an inefficient problem that it is impossible to immediately grasp the living body change because the living body change data can be grasped by sending the living body changing data to the specialized institution and receiving the analyzed data .

In addition, it is necessary to accurately identify each object in order to accurately recognize and monitor the object including the major axis. However, it is not clearly distinguished for each object and merges, and the shape of each object is not divided, uniform, non-rigid) There is a problem that it is difficult to separate and track each object in the thermal image due to the characteristics of the livestock. It is necessary to improve the accuracy and reliability of object tracking by tracking the movement of the object per frame of the image.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 10-2014-0105626 (published on Sep. 02, 2014).

The present invention has been made to solve the above-mentioned problems of the prior art, and it is an object of the present invention to provide a method and apparatus for dividing a thermal image into a plurality of superpixels to generate a thermal image capable of tracking the recognition and movement of an object having a non- And an object tracking apparatus and method using an image-based super-pixel.

In addition, the present invention is directed to solving the above-described problems of the prior art, and it is an object of the present invention to label a segment of a superpixel and accurately label a superpixel segment of an object in a next frame based on a cost function between the superpixel segments And an object tracking apparatus and method using an super-pixel based thermal image capable of accurately monitoring an object.

It should be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a method of tracking an object using a super-pixel based on a thermal image, the method comprising: collecting a thermal image for each frame; A label identifying a segment corresponding to an object in the segment of the reference frame; computing a cost function between the segment of the next frame and the segment corresponding to the object; And labeling a segment corresponding to the object of the next frame based on the calculated value of the cost function.

According to an embodiment of the present invention, the calculating of the cost function includes calculating a color attribute between a segment of a next frame and a segment corresponding to an object of a reference frame, a center position of a segment of a next frame, Based on the positional attribute associated with the center position of < / RTI >

According to an embodiment of the present invention, labeling the segment of the next frame may impart the label to at least one segment having a minimum calculated value of the cost function.

According to one embodiment of the present application, at least one segment whose computed value of the cost function is the smallest may be determined based on Equation (1).

According to an embodiment of the present invention, labeling the segment of the next frame may include calculating the cost function of each segment of the next frame and each of the plurality of segments corresponding to the object of the reference frame, The label can be given to the segment of the next frame corresponding to the minimum value among the computed values for each segment corresponding to the object.

According to an embodiment of the present invention, an object tracking method further includes a step of converting a thermal image of the next frame into a binarized image, when the labeled segment of the segment of the next frame is one, Detecting a plurality of segments having similar characteristics to the labeled segments to determine a labeling object, and supplementing the detected labels with the plurality of segments.

According to an embodiment of the present invention, an object tracking method includes: calculating a distance between a plurality of segments to which the label of the next frame is assigned; and when the distance is equal to or greater than a preset distance, And removing the label.

According to one embodiment of the present invention, the step of dividing into super pixels includes a step of, based on at least one of density, boundary agreement, minimization of under-division, and uniformity in consideration of similar characteristics of each pixel of the thermal image, .

An object tracking apparatus using a super-pixel based on a thermographic image according to an embodiment of the present invention includes an image collecting unit for collecting thermal image for each frame, an image dividing unit for dividing the thermal image into a plurality of super- A segment control unit for providing a label identifying a segment corresponding to an object among the segment of the reference frame, and an operation unit for calculating a cost function between a segment of a next frame and the segment corresponding to the object, The label may be assigned to the segment corresponding to the object of the next frame based on the calculated value of the cost function.

According to an embodiment of the present invention, the arithmetic operation unit may calculate a color attribute between a segment of a next frame and a segment corresponding to an object of the reference frame, a color attribute between a center position of a segment of the next frame, and a center position of a segment corresponding to an object of the reference frame Lt; RTI ID = 0.0 > location < / RTI > attribute.

According to an embodiment of the present invention, the segment control unit may assign the label to at least one segment having a minimum calculated value of the cost function.

According to one embodiment of the present application, at least one segment with a minimum calculated value of the cost function may be determined based on Equation (2).

According to an embodiment of the present invention, the operation unit calculates the cost function of each of the segments of the next frame and each of the plurality of segments corresponding to the object of the reference frame, wherein the segment control unit It is possible to give the label to the segment of the next frame corresponding to the minimum value among the computed values for each segment.

According to an embodiment of the present invention, when there is one labeled segment in the segment of the next frame, the image control unit converts a thermal image of the next frame into a binary image, and the segment control unit It is possible to detect a plurality of segments having similar characteristics to the segment to which the label is given based on the image to determine a labeling target and to supplement the label to the detected plurality of segments.

According to an embodiment of the present invention, the operation unit calculates a distance between a plurality of segments to which the label is assigned in the next frame, and when the distance is equal to or larger than a preset distance, You can remove labels from small segments.

According to an embodiment of the present invention, the image controller divides the image into super pixels based on at least one characteristic of density, boundary agreement, minimization of under-division, and uniformity in consideration of similar characteristics of each pixel of the thermal image. can do.

According to an embodiment of the present invention, an object monitoring system includes a collection device that acquires image information of a barn, collects environmental information of the barn, and a behavior of a livestock based on image information and environment information received from the collection device Wherein the image information includes a thermal image of the livestock and the server is capable of monitoring livestock behavior based on the object tracking method described above.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-described task solution of the present invention, the thermal image is divided into a plurality of super pixels and the super-pixel segments of the object in the next frame are accurately labeled based on the cost function between the super pixel segments to accurately recognize the object Can be monitored.

According to an aspect of the present invention, there is provided a method for providing a label to a segment of a superpixel, labeling a segment of a next frame based on a cost function, The movement of the object having the shape can be accurately monitored.

1 is a diagram showing a schematic configuration of an object monitoring system according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of an object monitoring system according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating a configuration of an object tracking apparatus using a super-pixel based on a thermal image according to an embodiment of the present invention.
4 is a diagram illustrating an example of superpixel segmentation and labeling of thermographic images, according to one embodiment of the present invention.
FIG. 5 is a diagram illustrating an example in which an object is traced and a label is assigned in a next frame of a thermal image according to an exemplary embodiment of the present invention.
6 is a diagram illustrating an example of label supplementation of an object tracking method according to an embodiment of the present invention.
7 is a diagram illustrating an example of label correction of an object tracking method according to an embodiment of the present invention.
8 is a flowchart illustrating an object tracking method according to an embodiment of the present invention.
9 is a diagram illustrating a flow of label replenishment of an object tracking method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when a part is referred to as being "connected" to another part, it is not limited to a case where it is "directly connected" but also includes the case where it is "electrically connected" do.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

FIG. 1 is a block diagram showing a schematic configuration of an object monitoring system according to an embodiment of the present invention, and FIG. 2 is a block diagram showing a configuration of an object monitoring system according to an embodiment of the present invention. 1 and 2, the object monitoring system may include a collection device 100 and a server 200. [ The object monitoring system is for detecting an environment of livestock and housing and analyzing the environment to take necessary measures. Referring to FIG. 1, the information obtained in the collecting apparatus 100 is transmitted to the server 200 The server 200 can determine whether or not the animal is in an abnormal state. The server 200 may transmit various monitoring information including information on whether the animal is abnormal or not via the network 10 to the database 20 or the user terminal 30. [ For example, the object may include a large axis such as a cow, a pig, and the like.

The collecting apparatus 100 can acquire image information of a barn and collect environmental information of the barn. The collecting apparatus 100 may include a thermal imaging unit 110, a wide-angle imaging unit 120, and an environment sensing unit 130. First, the thermal imaging unit 110 can acquire a body temperature distribution image (thermal imaging image) of a livestock. Illustratively, the thermal imaging unit 110 may include a plurality of thermal imaging cameras. The thermal imaging unit 110 can acquire a temperature distribution image of each part of a livestock such as a head, a body, and a leg by capturing an infrared ray having a corresponding wavelength according to a specific temperature. That is, the thermal imaging unit 110 can display images of the captured livestock in different colors according to the temperature distribution of each part of the livestock. Further, the thermal imaging unit 110 may include a drive unit, and the angle of view may be changed by the drive unit.

The wide angle photographing unit 120 can acquire a wide angle image of the house. The wide-angle photographing unit 120 may include a plurality of wide-angle cameras, and the wide-angle photographs may include images and photographs of livestock that are housed in a house and a house. By using the wide angle camera, even if the wide angle photographing unit 120 is located at a low position of the housing, a large area of the house can be photographed. The wide angle photographing unit 120 may include a driving unit, and the photographing angle may be changed by the driving unit. Illustratively, the thermal imaging unit 110 and the wide-angle photographing unit 120 can receive a drive command from a user terminal 30 (to be described later), and the drive unit can be driven based on the drive command.

Further, according to one embodiment of the present invention, the thermal imaging unit 110 and the wide-angle imaging unit 120 may each include a communication unit, and may receive an identifier of each animal from the identifier communication module 150. [

The environment sensing unit 130 may sense the environment inside the housing. The above-mentioned environment may mean, for example, the atmospheric temperature inside the housing, the carbon dioxide concentration, and the like. 1, the environment sensing unit 130 may include at least one of a temperature / humidity measurement unit 131, a carbon dioxide measurement unit 132, a wind direction measurement unit 133, and a ventilation unit 134.

The information acquired by the environment sensing unit 130 may be transmitted to the gateway 135. The gateway 135 may collect the acquired information and transmit the environment information to the monitoring server 210. [ The environment information may include at least one of temperature and humidity of the housing, carbon dioxide concentration, wind direction, and wind velocity detected by the environment sensing unit 130.

The image information collected by the collecting apparatus 100 may include a body temperature distribution image (thermal image) and a wide angle image photographed by the thermal imaging unit 110 and the wide angle imaging unit 120. The image information and the environment information may be transmitted to the server 200 every predetermined period (for example, one second). Illustratively, the acquisition device 100 may transmit image and environmental information to the server 200 using RF communications or zigbee communications.

The server 200 can analyze the change of the housing environment, the living body changes of the livestock and the behavior pattern based on the image information and the environment information received from the collection device 100. [ In addition, the server 200 can determine an abnormal state of the livestock based on the analysis result.

Referring to FIG. 2, the server 200 may include a monitoring server 210, an image processing server 220, and a data server 230. The monitoring server 210 may sense a change in the housing environment based on the environmental information received from the collection device 100. [ Illustratively, the monitoring server 210 may determine that an anomaly has occurred if at least one of the temperature, humidity, carbon dioxide concentration, wind direction and wind speed of the housing exceeds a threshold value. At this time, the threshold value can be set corresponding to the temperature and humidity, the carbon dioxide concentration, the wind direction and the wind speed, respectively. The determination of the monitoring server 210 can be performed even if there is no direct change in the animal. For example, if fires in a house can occur and affect livestock, the occurrence of an anomaly can be determined through environmental changes in the house.

The image processing server 220 can analyze the biological changes or behavior patterns of the livestock based on the image information received from the collection device 100. [ By way of example, the biotransformation and behavioral pattern can include changes in physiology of the livestock, disease infections, injury, development and estrus.

The image processing server 220 may analyze the body temperature distribution image photographed by the thermal imaging unit 110 to determine whether the temperature distribution of a predetermined region of the livestock exceeds a predetermined first threshold range. The predetermined region may refer to a head, a trunk, a leg, or the like of a livestock. The first critical range may be, for example, the range of normal body temperature of the livestock. The image processing server 220 can determine that the animal is in an abnormal state when the body temperature of the predetermined area of the animal is greater than or less than the first threshold range.

That is, the image processing server 220 can analyze the biological changes of the livestock based on the body temperature distribution image. Illustratively, if the body temperature of the genital part of the livestock exceeds the first threshold range, the image processing server 220 may determine that the livestock is an estrus, Or falls below the first threshold range, it can be determined that an abnormality has occurred in the mother or the young.

In addition, the image processing server 220 may analyze the wide-angle image to determine whether the movement amount of the livestock is out of a predetermined second threshold range. The second threshold range may be set based on, for example, statistics or averages of the total distance the livestock has moved for a predetermined time (e.g., one day). The image processing server 220 can determine that the animal is in an abnormal state when the movement amount of the animal is greater than or less than the second threshold range. The image processing server 220 may analyze the photographed image and calculate the movement amount of the livestock according to a previously stored algorithm based on the size of the housing, the moving time of the livestock, and the like.

More specifically, the image processing server 220 can analyze the behavior pattern of livestock based on the moving amount of the livestock. Illustratively, if there is no movement of livestock and a movement amount less than the second critical range is observed, it can be determined that an abnormality has occurred in the behavior pattern due to disease or injury of the livestock.

In addition, according to one embodiment of the present invention, when the temperature distribution of a predetermined region of a livestock deviates from a predetermined first threshold range and the amount of livestock movement deviates from a second threshold range, It can be determined that it is in an abnormal state. The image processing server 220 can determine the abnormality state of the livestock referring to both the thermal image and the wide angle image, thereby further improving the accuracy of the determination.

Referring to FIGS. 1 and 2, the server 200 may include a data server 230. The data server 230 transmits the result of the abnormal state determination of the livestock to the network 10 together with at least a part of the analysis result of the change of the housing environment, the analysis result of the living body, To the database 20 and / or the user terminal 30 via the Internet. According to an embodiment, the data server 230 may transmit not only the database 20 and the user terminal 30 but also an apparatus provided in a livestock medical facility.

The network 10 refers to a connection structure between a node and a wireless network capable of exchanging information between nodes such as a terminal and a server. An example of such a network is a 3rd Generation Partnership Project (3GPP) network, a Long Term A Wide Area Network (WAN), a Personal Area Network (PAN), a Personal Area Network (LAN), a Personal Area Network (LAN) ), A Bluetooth network, a satellite broadcast network, an analog broadcast network, a Digital Multimedia Broadcasting (DMB) network, and the like.

The user terminal 30 is a device that communicates with the server 200 through the network 10 and may be a device such as a smartphone, a SmartPad, a tablet PC, a wearable device, System, a GSM (Global System for Mobile communication), a PDC (Personal Digital Cellular), a PHS (Personal Handyphone System), a PDA (Personal Digital Assistant), an IMT (International Mobile Telecommunication) 2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminals, desktop computers, and smart TVs.

The database 20 or the user terminal 30 may be connected to the server 200 (e.g., the data server 230) of the object monitoring system and may be configured to analyze the changes in the housing environment, Image information, environmental information, and the result of the abnormal state determination of the livestock. The database 20 or the user terminal 30 can access the server 200 at a desired time to check the monitoring information as described above. However, if the server 200 receives the abnormal state determination result from the server 200, .

Illustratively, the user terminal 30 may transmit a control signal to the server 200 to control the thermal imaging unit 110 or the wide-angle imaging unit 120. More specifically, when the user of the user terminal 30 wants to observe a specific part of the house (for example, a particular livestock) as the center, the user can connect to the server 200 and transmit the control signal. The photographing angle of the drive unit provided in the thermal imaging unit 110 and the wide angle photographing unit 120 can be changed based on the control signal. Accordingly, the user of the user terminal 30 can confirm a specific part of the barn to be observed in real time.

The object monitoring system may include a body temperature measurement module 140. The body temperature measurement module can be mounted at a predetermined position of the livestock, and the body temperature of the livestock can be measured. In addition, a plurality of body temperature measurement modules 140 may be provided, and body temperature may be measured by each part of the body of a livestock. The body temperature measurement module 140 may transmit information on the measured body temperature of the livestock to the data server 230. The body temperature measurement module 140 may transmit the measured body temperature to the data server 230 using RF communication or zigbee communication.

The data server 230 analyzes the body temperature information of the livestock received from the body temperature measurement module 140 and the body temperature distribution image acquired by the thermal imaging unit 110 and determines whether the difference in the temperature distribution of the livestock is within a predetermined error range And can transmit the result of the abnormal state determination of the animal to the database 20 and the user terminal 30. Illustratively, the data server 230 may compare body temperature and local temperature distribution for each portion of the livestock. That is, when the body temperature and the temperature distribution in the same region exceed a predetermined error range, it can be determined that an error has occurred in the body temperature measurement module 140 or the thermal image pickup unit 110, . By comparing the body temperature and the temperature distribution of such livestock, a reliable anomaly determination can be performed.

The object monitoring system may include a database 20 for storing information received from the server 200. The database 20 may store image information, environmental information, changes in housing environment, analysis results of biological changes and behavior patterns of livestock, and an anomalous state determination result of livestock in association with an identifier of a livestock.

Referring to FIG. 1, the identifier communication module 150 may transmit an identifier of a livestock to the thermal imaging unit 110 and the wide-angle photographing unit 120. Referring to FIG. The identifier communication module 150 can transmit the identifier of the livestock to the thermal imaging unit 110 and the wide angle imaging unit 120 using RF communication or zigbee communication. The thermal imaging unit 110 and the wide angle imaging unit 120 can share the identifier of the animal with the data server 230 together with the image information and the data server 230 can share the image information, , Analysis results of the biological changes and behavior patterns of the livestock, and results of the anomalous state determination of the livestock can be transmitted to the database 20 together with the identifiers. Illustratively, the database 20 may be provided with an object monitoring system and may be located remotely to receive and store the above-described information and identifiers from the data server 230 via the network 10. [

According to one embodiment of the present invention, the data server 230 receives the identifier of the livestock received from the identifier communication module 150 and the temperature image, the wide-angle image, and the image of the livestock received by the thermal imaging unit 110 and the wide- Each of the cattle can be identified based on the body temperature distribution image. When there are a plurality of livestock, the thermal imaging unit 110 and the wide-angle photographing unit 120 sense the identifier signal transmitted from the identifier communication module 150 of each livestock to determine the image information of each livestock Can be obtained. The thermal imaging unit 110 and the wide angle photographing unit 120 may be configured to provide identification information of a livestock that needs to be photographed by the user terminal 30 to the thermal imaging unit 110 and the wide angle imaging unit 120, (Such as signal strength determination) of the identifier signal transmitted from the identifier communication module 150 of the terminal 100 to acquire image information of a livestock that needs to be photographed.

Hereinafter, the thermal image-based object tracking apparatus 300 will be described. The thermal image-based object tracking apparatus 300 according to an embodiment of the present invention corresponds to the server 200 of the object monitoring system described above and may include at least one of the image processing server 220 and the data server 230 May be implemented as separate servers, devices or modules (units) that include the same or two server functions. However, the present invention is not limited thereto and may be provided as a separate hardware device or module communicatively connected to the user terminal 30 or the server 200.

FIG. 3 is a diagram illustrating a configuration of an object tracking apparatus using a super-pixel based on a thermal image according to an embodiment of the present invention.

3, the object tracking apparatus 300 may include an image collection unit 310, an image control unit 320, a segment control unit 330, and an operation unit 340. It should be noted that the object tracking apparatus 300 shown in FIG. 3 is only one embodiment of the present invention, and can be modified into various forms based on the components shown in FIG. 3. In the technical field to which the embodiments of the present invention belong, Those skilled in the art will understand the present invention. For example, the components and functions provided within the components may be combined into a smaller number of components or further separated into additional components.

The image collecting unit 310 may collect the thermal image for each frame. The thermal image may be an image photographed by the thermal imaging unit 110 described above. The image collecting unit 310 can collect the thermal image transmitted from the thermal image radiographing unit 110 to the server 200 by frame. The thermal image may comprise an infrared image for at least one object. For example, an object may include a major axis such as a cow, a pig, and the like.

According to one embodiment of the present invention, the image capturing unit 310 can acquire a thermal image captured by the thermal image capturing unit 110. The thermal image of the object may be transmitted to the image collection unit 310 every predetermined period (for example, 1 second). Illustratively, the image acquisition unit 310 may acquire thermographic images using RF communication or zigbee communication.

The image collecting unit 310 may acquire a thermal image photographed by applying a top-down view, which is a photographing method of looking up the object from top to bottom. The object tracking device 300 receives and utilizes a thermal image photographed by a top-down view method, thereby improving the effect of object recognition using the super-pixel described later, .

The image control unit 320 may divide the thermal image into a plurality of super pixels in a segment unit. For example, the image controller 320 may perform super pixel division based on a Simple Linear Iterative Clustering (SLIC) algorithm. A superpixel is a bundle of pixels (pixels) having similar characteristics in the original image (in this case, the thermal image can be an original image). The superpixel divides the original image into small uniform regions having similar characteristics, And can be grouped into a basic unit for processing an image. In addition, a segment means a divided unit, which means the same as a superpixel or corresponds to a superpixel. Each super-pixel may be given an index for identifying the super-pixel.

The image controller 320 may divide the thermal image into super pixels based on at least one characteristic of density, boundary agreement, minimization of under-division, and uniformity in consideration of similar characteristics of each pixel of the thermal image. For example, the similar characteristic may include a color vector (RGB, gray characteristic, etc.) of each pixel, a position value, and the like. Clustering can be a characteristic of how super-pixel shapes (appearance) resemble each other. In addition, the boundary match degree may be a characteristic indicating the degree of matching between the super pixel and the object boundary in the original image. If the boundaries between the super-pixel boundary and the original image do not match, it is not possible to detect the correct boundary when recognizing the object after the image segmentation from the super-pixel, which greatly affects the image processing process after the division. In addition, when the superpixel is implemented with a certain iterative algorithm, the pixels of the superpixel are not included in the last step of the algorithm. The minimization characteristic of the subdivision is a characteristic that minimizes such pixels, and the pixels not belonging to the superpixel can be assigned to the adjacent superpixel by post-processing. In addition, the uniformity may be a characteristic indicating the degree of similarity between pixels in the super pixel. The similarity between pixels can be calculated through a similarity calculation method that is commonly used in the field of image processing techniques, and thus a detailed description thereof will be omitted.

The segment control unit 330 may assign a label identifying the segment corresponding to the object among the segments of the reference frame. The object herein may refer to a target object to be tracked through the object tracking device. Further, the reference frame may be one of a plurality of frames in which the object is displayed, and may refer to a frame (e.g., a (t-1) th frame) before tracking the movement of the object. In addition, the object may include the animal described above, and the object in the following description will exemplarily describe tracking the movement of the cattle.

4 is a diagram illustrating an example of superpixel segmentation and labeling of thermographic images, according to one embodiment of the present invention.

Referring to FIG. 4, the image controller 320 may divide the frame of the thermal image collected by the image collection unit 310 into super-pixels of a segment unit. The segment control unit 330 may assign a label to a segment corresponding to an object to be tracked. 4A, the image control unit 320 assigns a target object x to be traced on a-1 frame (hereinafter referred to as a-1 frame) as a reference frame to a user input Referring to FIG. 4B, the image controller 320 may divide the thermal image of the a-1 frame into super-pixels of a plurality of segment units. At this time, the segment (1) can be understood as the uniform region described above. The segment control unit 330 may assign a label (for example, a red color) to segments 2 (3) and 4 (5) corresponding to the object x , So that the user can easily identify the object x to be tracked through the thermal image. The occlusion of the pixels of the thermal image can be frequently caused by the motion, the action, and the like of the object x. In addition, the aspect ratio of the object x may not be matched in the frame-by-frame thermal image as the object x moves. Also, a plurality of objects in the thermal image may have a color similar to that of the display The segment control unit 330 changes the color of surrounding objects in a segment unit of the super pixel so that the object x can be distinguished from neighboring objects (2), (3), (4), and (4) corresponding to x can be given a label.

FIG. 5 is a diagram illustrating an example in which an object is traced and a label is assigned in a next frame of a thermal image according to an exemplary embodiment of the present invention.

5 (a) is a diagram showing a label of an object (x) in the above-mentioned a-1 frame. 5B shows an example in which the thermal image of the a frame (hereinafter referred to as a frame) of the next frame by the image control unit 320 is divided into super pixels. In the present specification, the next frame means a frame of a thermal image in a state in which an object is moved. 5A and 5B, it is confirmed that the object x in the a-1 frame (FIG. 5A) is moved in the a frame (FIG. 5B). The object tracking apparatus 300 calculates a cost function between the thermal image of the a-1 frame and the thermal image of the a frame to give a label to the a frame corresponding to the moved object x, The movement (change) of the object (x) in a time series change can be tracked.

The operation unit 340 can calculate the cost function between the segment of the next frame and the segment corresponding to the object. More specifically, the operation unit 340 can calculate the cost function between the entire superpixel segment of the next frame and the superpixel segment corresponding to the object in the previous frame (reference frame). The cost function is a function considering the inter-segment space accessibility (distance attribute) and the segment appearance similarity (color attribute) between two frames, and the labeling (or label elimination) of the segment in the next frame through the cost function is more accurate . ≪ / RTI >

Specifically, the arithmetic operation unit 340 calculates the color attribute between the segment of the next frame (a frame) and the segment 2, 3, 4, 5 corresponding to the object of the reference frame (a-1 frame) The cost function can be calculated based on the center position of the segment and the positional attribute associated with the center position of the segment corresponding to the object of the reference frame. The color attributes may include, for example, RGB, gray values, and the like as color vector components in the thermal image according to the temperature. In addition, the location attribute may mean, for example, the distance between superpixel segments.

According to one embodiment of the present application, the calculation unit 340 calculates the number of segments 2, 3, 4, 5 corresponding to all the segments of the next frame (a frame) and the object of the reference frame (a- The calculation of the cost function can be repeated until the cost function calculation for each time is completed.

According to an embodiment of the present invention, the segment control unit 330 may label a segment having the smallest computed value of the cost function in the next frame. The jag control section 330 sets the super pixel segment whose calculation value of the cost function is minimum for each segment 2, 3, 4, 5 corresponding to the object of the reference frame (a-1 frame) (Matching) and giving a label. 5 (a) and 5 (b), the operation unit 340 includes a plurality of segments corresponding to all the segments of the next frame (a frame) and the object x of the reference frame (a-1) 2), (3), (4) and (5). Depending on the embodiment, not all of the segments of the next frame, but a segment (e.g., a portion excluding the background) corresponding to an object that is distinguished based on the color of the segment in the next frame and a segment A cost function with each of all the segments may be calculated.

5 (b), the segment control unit 330 calculates a cost function for each segment (2), (3), (4) and (5) of the object x in the previous frame, (7), (8) and (9) of the next frame having the smallest value among the values of the super-pixel segments 6 7) (8) Labeling can be given to (9). Referring to FIG. 5 (c), the segment control unit 330 may label the segment of the next frame having the minimum cost function with the segment corresponding to the object (x) in the previous frame. Illustratively, the segment control unit 330 may label the segment 6 of the next frame with the smallest value by computing the cost function with segment 2 of the previous frame, and segments 3, 4, 5 ) Segment and the cost function, respectively, so that the segments 7, 8, and 9 of the next frame having the minimum value can be also labeled.

According to one embodiment of the present invention, the segment control unit 330 can determine a segment having the smallest calculated value of the cost function based on Equation (1).

[Equation 1]

는 다음 프레임의 세그먼트와 기준 프레임의 객체에 대응하는 세그먼트 간의 비용함수이고, C는 프레임별 상기 세그먼트의 평균 색(라벨) 벡터이고, X는 프레임별 각 세그먼트의 중심 위치값이고, j(t)는 다음 프레임(t)의 전체 슈퍼픽셀 세그먼트이고, i(t)는 비용함수의 연산값이 최소인 다음 프레임의 슈퍼픽셀 세그먼트를 의미한다. 결과적으로 도 5(d)를 참조하면 비용함수 연산에 따른 최소값에 대응하는 다음 프레임에서의 세그먼트(6)(7)(8)(9)에 라벨이 부여되어 객체(x)가 추적된 다음 프레임의 열화상 영상을 확인할 수 있다.here,

Where C is a mean color (label) vector of the segment for each frame, X is the center position value of each segment for each frame, and j (t) is the center position value of each segment for each frame. Is the entire super pixel segment of the next frame t, and i (t) is the super pixel segment of the next frame in which the calculated value of the cost function is the smallest. As a result, referring to FIG. 5 (d), the segments 6, 7, 8, and 9 in the next frame corresponding to the minimum value according to the cost function calculation are labeled, Can be confirmed.

According to the example described above, the cost function is calculated with the segments 2, 3, 4 and 5 of the previous frame so that the number of segments 6, 7, 8 (9) However, in the tracking of the actual object (x), the object of the previous frame due to the change in shape or number of the segment at the time of super pixel division in the next frame, movement of the object, overlapping with another object, The number of segments corresponding to the minimum number of segments corresponding to the next frame may not coincide with the number of segments having the smallest cost function calculation value of the next frame. It is possible.

6 is a diagram illustrating an example of label supplementation of an object tracking method according to an embodiment of the present invention. Hereinafter, a description will be given of a method of supplementing the super-pixel segments of an object in a case where labels are not matched to the object in the next frame according to the object tracking method described above.

Illustratively, as shown in Fig. 6 (a), when there is one labeled segment among the segments corresponding to the object x in the next frame (a), the image control unit 320 determines that the next frame It is possible to convert the image image into the residual image as shown in Fig. 6 (b). Thereafter, the segment control unit 330 can determine a labeling target by detecting a plurality of segments having similar characteristics to the labeled segments based on the binarized image. Illustratively, the segment control 330 includes a plurality of segments having similarity to the labeled segment based on pixel link element analysis based on at least one of the following properties: tightness, border agreement, minimization of under-segmentation, and uniformity Can be detected. For example, the segment control unit 330 may perform a connect component analysis (CCA) algorithm to determine a plurality of segment regions having similar characteristics to the labeled segments. Next, the segment control unit 330 can identify the identifier of the super pixel corresponding to the determined plurality of segments. 6 (c), the segment control unit 330 may supplement the segment of the identified super pixel with a label if the detected plurality of segments are not labeled. In addition, the segment control unit 330 collects an identifier of a superpixel to which a label is superimposed and an identifier of a superpixel to which a label has been supplemented, and recognizes that all superpixel segments corresponding to the target object (x) have.

7 is a diagram illustrating an example of label correction of an object tracking method according to an embodiment of the present invention. According to the foregoing description, the cost function of the segment corresponding to the object of the previous frame and the segment of the next frame can be computed and the segment of the next frame corresponding to the minimum value can be labeled. In this manner, the labeling based on the minimum value of the cost function determines the segment to which the label is to be assigned through the operation value of the function. Therefore, if the calculated value of the cost function is the minimum value, The label may be misapplied for the reason that the minimum value of the cost function is calculated. As shown in FIG. 7 (a), in addition to the case where a label is assigned to the segment 6 corresponding to the object x to be tracked, the segment 7 of the other object y also has a minimum value A label may be given.

As described above, the object tracking apparatus 300 can correct a case where a label among the segments of the next frame is erroneously given. Specifically, the operation unit 340 can calculate a distance between the plurality of segments labeled with the next frame. The inter-segment distance D can be computed based on ∥X _m -X _n ∥. Where m, n is the label information of the two segments for which the separation distance is computed. The distance between the segments may be the distance between the center points of the super-pixil segment. 7A, the distance between the segment 6 of the object x to be tracked and the segment 7 of another object y can be calculated. Thereafter, the segment control unit 330 may remove the label of a relatively small segment when the segment distance is equal to or greater than a preset distance. Illustratively, the label of a superpixel segment having a relatively small size (area) of the compared two segments can be removed. Referring to FIG. 7 (b), it can be confirmed that the segment 7 of another object y is deleted and the segment 6 of the object x to be tracked remains in accordance with the calculated distance being equal to or greater than a preset distance .

8 is a flowchart illustrating an object tracking method according to an embodiment of the present invention.

The object tracking method shown in FIG. 8 may be performed by the object tracking apparatus 300 described with reference to FIGS. 1 to 7. FIG. Therefore, even if omitted below, the description of the object tracking apparatus 300 through FIGS. 1 to 7 can be similarly applied to FIG.

Referring to FIG. 8, in step S810, the image collection unit 310 may collect thermal image images for respective frames. In step S820, the image control unit 320 may divide the thermal image into a plurality of super pixels in a segment unit. In step S830, the segment control unit 330 may assign a label identifying the super pixel segment corresponding to the object among the segments of the reference frame.

In step S840, the operation unit 340 may calculate a cost function between a segment of the next frame and at least one segment corresponding to the object of the reference frame. The calculation unit S340 calculates the color attribute between the segment of the next frame (a frame) and the segment 2, 3, 4, 5 corresponding to the object of the reference frame (a-1 frame) The cost function can be calculated based on the position and the positional attribute associated with the center position of the segment corresponding to the object of the reference frame. The operation unit 340 can calculate the cost function between the segment 2 which is one of the segments corresponding to the object x in the reference frame and all the segments of the next frame. In addition, the operation unit 340 may repeat the above-described operation so that the cost function of each of all segments corresponding to the object x and all the segments of the next frame are calculated.

In step S850, the segment control unit 330 may label the segment corresponding to the object of the next frame whose cost function between the segments corresponding to the object (x) in the previous frame is the minimum value.

9 is a diagram illustrating a flow of label replenishment of an object tracking method according to an embodiment of the present invention.

In step S851, the number of the labeled segments of the super pixel segment of the next frame (a) can be confirmed. If there is one labeled segment, in step S852, the image control unit 320 can convert the thermal image of the next frame into the non-image.

In step S853, the segment control unit 330 can determine a labeling target by detecting a plurality of segments having similar characteristics to the segment to which the label is given based on the binarized image. Illustratively, the segment control 330 includes a plurality of segments having similarity to the labeled segment based on pixel link element analysis based on at least one of the following properties: tightness, border agreement, minimization of under-segmentation, and uniformity Can be detected.

In step S854, the segment control unit 330 confirms whether or not the label is given to the detected segment, and if the detected plural segments are not labeled, the segment can be supplemented with the label. In addition, it is possible to recognize that labels are assigned to all segments corresponding to the target object (x) by collecting labels previously assigned and replenished.

As described above, the object tracking apparatus 300 can recognize an object having a nonlinear shape more accurately by dividing a thermal image into a plurality of superpixels and performing object recognition in units of superpixel segments. Meanwhile, although an example has been described with respect to tracking of large animals, the object tracking device 300 and the object tracking method described above have a nonlinear shape, and it is necessary to recognize and monitor the division of a plurality of objects moving within a predetermined space It will be applicable to all fields.

The object tracking method according to one embodiment of the present invention can be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

10: Network 20: Database
30: user terminal 100: collecting device
110: Thermal image radiographing section 120: Wide angle radiographing section
130: Environmental sensing unit 131: Temperature and humidity measurement unit
132: carbon dioxide measuring unit 133: wind direction wind speed measuring unit
134: Ventilation unit 135: Gateway
140: body temperature measurement module 150: identifier communication module
200: server 210: monitoring server
220: image processing server 230: data server
300: Object tracking device 310:
320: image control unit 330: segment control unit
340:

Claims (18)

In an object tracking method using a super-pixel based on a thermal image,
Collecting a thermal image for each frame;
Dividing the thermal image into a plurality of super pixels in a segment unit;
Providing a label identifying a segment corresponding to the object to be tracked among the segments of the reference frame;
Calculating a cost function between a segment of a next frame and the segment corresponding to the object to be tracked; And
Labeling a segment corresponding to the object to be tracked of the next frame based on the calculated value of the cost function;
Lt; / RTI >
If no label is assigned to all the segments corresponding to the object to be tracked in the next frame and there is only one segment to which the label is assigned,
Converting a thermal image of the next frame into a binary image;
Performing a CCA on the binarized image to detect a plurality of segments associated with the tracking target object having similar characteristics to the one segment to which the label is attached to determine a labeling target; And
And supplementing the label to the detected plurality of segments,
When the label is assigned to a segment of another object other than the object to be tracked in the next frame,
Calculating a distance between a segment of the another object to which the label is attached in the next frame and a segment of the object to be tracked; And
And removing the label of the relatively small segment if the spacing distance is greater than or equal to a predetermined distance.
Object tracking method. The method according to claim 1,
Wherein the calculating the cost function comprises:
The color attribute between the segment of the next frame and the segment corresponding to the object of the reference frame, and the positional attribute associated with the center position of the segment of the next frame and the center position of the segment corresponding to the object of the reference frame. Tracking method. 3. The method of claim 2,
Wherein labeling the segment of the next frame comprises:
And assigning the label to a segment whose computed value of the cost function is the smallest. The method of claim 3,
A segment whose computed value of the cost function is minimum is determined based on Equation (1)
[Equation 1]

C is the average color vector of the frame-by-frame segment, X is the center position value of each segment per frame, j (t) is the entire segment of the next frame (t) Is a segment of the next frame. The method of claim 3,
Wherein labeling the segment of the next frame comprises:
Computing the cost function of each segment of the next frame and each of the plurality of segments corresponding to the object of the reference frame,
And gives the label to a segment of a next frame corresponding to a minimum value among the calculated values for each segment corresponding to the object. delete delete The method according to claim 1,
The step of dividing into superpixels comprises:
Is performed based on at least one of characteristics of density, boundary agreement, minimization of under-division, and uniformity in consideration of similar characteristics of each pixel of the thermal image. An object tracking apparatus using super-pixels based on thermal imaging,
An image collecting unit for collecting the thermal image for each frame;
An image control unit for dividing the thermal image into a plurality of super pixels in a segment unit;
A segment control unit that gives a label identifying a segment corresponding to the object to be tracked among the segments of the reference frame; And
A calculation unit for calculating a cost function between a segment of a next frame and the segment corresponding to the object to be tracked,
, ≪ / RTI &
The segment control unit,
Assigns the label to a segment corresponding to the tracking object of the next frame based on the computed value of the cost function,
If no label is assigned to all the segments corresponding to the object to be tracked in the next frame and there is only one segment to which the label is assigned,
The image control unit converts the thermal image of the next frame into a binary image,
The segment control unit performs a pixel linking element analysis (CCA) on the binarized image to detect a plurality of segments associated with the tracking target object having similar characteristics to the one segment to which the label is attached, And the label is supplemented to the detected plurality of segments,
When the label is assigned to a segment of another object other than the object to be tracked in the next frame,
Wherein the operation unit calculates a separation distance between a segment of the other object to which the label of the next frame is attached and a segment of the object to be tracked,
Wherein the segment control unit removes a label of a relatively small segment when the distance is equal to or greater than a preset distance. 10. The method of claim 9,
The operation unit,
The cost function is calculated based on the color attribute between the segment of the next frame and the segment corresponding to the object of the reference frame, and the positional attribute associated with the center position of the segment of the next frame and the center position of the segment corresponding to the object of the reference frame Object tracking device. 11. The method of claim 10,
The segment control unit,
And assigns the label to the segment whose computed value of the cost function is the smallest. 12. The method of claim 11,
A segment whose computed value of the cost function is the minimum is determined based on Equation (2)
&Quot; (2) "

C is the average color vector of the frame-by-frame segment, X is the center position value of each segment per frame, j (t) is the entire segment of the next frame (t) Is a segment of the next frame. 12. The method of claim 11,
Wherein the operation unit calculates the cost function of each segment of the next frame and each of a plurality of segments corresponding to the object of the reference frame,
Wherein the segment control section assigns the label to a segment of a next frame corresponding to a minimum value among the calculation values for each segment corresponding to the object. delete delete 10. The method of claim 9,
The image control unit includes:
The super-pixel is divided into super-pixels based on at least one of characteristics of density, boundary agreement, minimization of under-division, and uniformity in consideration of similar characteristics of each pixel of the thermal image. In an object monitoring system,
A collecting device for acquiring image information of a barn and collecting environmental information of the barn; And
A server for monitoring the behavior of livestock based on the image information and environment information received from the collection device,
, ≪ / RTI &
Wherein the image information comprises a thermal image of the livestock,
Wherein the server monitors the behavior of the livestock based on the object tracking method of any one of claims 1 to 5 and 8. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 1 to 5 and a computer.

Images