KR102584357B1 - Apparatus for identifying a livestock using a pattern, and system for classifying livestock behavior pattern based on images using the apparatus and method thereof - Google Patents

Abstract

The present invention relates to a livestock behavior pattern classification system based on images. The livestock behavior pattern classification system includes a database server storing individual cross-question image data sets of livestock animals managed within a farm; Extract object images from images, estimate the posture of the object from the extracted object image, extract location information of the semi-print area of the object image based on the estimated posture, and extract the semi-print image of the semi-print area and the semi-print for each object. An object identification device that compares image data sets to identify objects and provides unique identification information for the objects; and a behavior pattern classification device that extracts movement data for the identified entities using the images and classifies the behavior patterns for each entity using the movement data, wherein each individual of the livestock animals in the farm is provided using the images. Provides classification of behavior patterns.

Description

Apparatus for identifying a livestock using a pattern, and system for classifying livestock behavior pattern based on images the apparatus and method thereof}

The present invention relates to a system and method for classifying livestock behavior patterns based on images. More specifically, for livestock animals, such as dairy cows, where each individual has a unique cross-face image, the individual is identified using the half-face image of each individual. It relates to an image-based livestock behavior pattern classification system and method that identifies and extracts movement data of each identified entity based on the image, classifies and provides a behavior pattern for each entity.

Recently, interest in smart livestock farming to improve farm competitiveness and productivity is expanding to interest in optimizing production by developing technology through precision livestock farming and taking climate change and environmental pollution into account in a circular structure. The purpose of smart livestock farming is to maximize the productivity of the farm by incorporating ICT technology into livestock housing to properly maintain and manage the growing environment of livestock in the livestock barn remotely or automatically through a computer or mobile device. A smart livestock farm that is integrated with ICT technology is a farm that can monitor the livestock environment and remotely and automatically control real-time livestock farming monitoring equipment and control equipment, as well as information related to the breeding, disease, feeding, and management of livestock animals. A server with a business management program installed is provided.

Meanwhile, the smart livestock farming system can be applied to pig farms, poultry farms, dairy farms, etc., and applies ICT convergence technology to livestock facilities such as drinking machines, feeders, milking machines, etc. to monitor the status and automatically monitor their operations. Make it controllable.

In this way, in order to improve the productivity of livestock farms, it is necessary to install a smart livestock housing system, monitor and analyze the behavior of each individual animal in the livestock barn, and continuously manage the status of each individual animal in the livestock barn in real time. do. According to this need, various types of methods for monitoring and analyzing livestock behavior in livestock farms are being proposed.

Korean Patent No. 10-2396999 uses a pre-trained deep learning model to automatically recognize and detect livestock behavior in videos, and through this, perform short-term behavior analysis and long-term behavior analysis to automatically determine livestock behavior. We have proposed a technology for an “automatic livestock behavior recognition and monitoring system using deep learning” that enables real-time monitoring.

In order for these various smart livestock management systems or livestock behavior monitoring systems in the livestock barn to monitor each individual of livestock animals in real time, identification of each individual must be preceded.

As a method of individual identification of livestock animals in a livestock barn according to the prior art, a method of attaching a tag to each individual of livestock animals and a method of using images have been proposed. First, the tag mounting method is to equip all objects with wireless tags, then install tag recognizers in necessary locations within the farm, and the tag recognizer can recognize objects close to the recognition range. However, this method has a problem in that it cannot identify an object in a place where a tag recognizer is not installed. Meanwhile, in the method of identifying objects using images, even if a machine learning-based object identification model is used, not only is it difficult to accurately obtain images of the front of each individual livestock animal, but also the shapes of the body of the livestock animals are Since they are generally similar, there is an inherent problem that it is difficult to distinguish their characteristics.

In this way, even if images of all spaces in the farm are secured, if the objects included in each image cannot be properly identified, not only will it become difficult to accurately monitor the behavior or status of each individual, but also behavior analysis will become difficult. .

Meanwhile, the representative breed of dairy cattle, one of the livestock animals, is the Holstein breed, and it is widely raised on livestock farms in Korea. The color of this Holstein breed varies, but is mainly black and white, with black distributed over the upper part of the body. However, each Holstein cow has its own unique half-face image.

Therefore, the present invention seeks to propose a method for accurately identifying individuals using the half-face images of livestock animals and classifying the behavior patterns of each individual through such individual identification.

Korean Patent Publication No. 10-1866226 Korean Patent Publication No. 10-2021-0115204 Korean Patent Publication No. 10-2396999

The purpose of the present invention to solve the above-mentioned problems is to provide a half-face image-based object identification device configured to identify each individual in the movement image of livestock animals using the unique half-face image of each individual of the livestock animals. Do it as

In addition, the present invention uses the object identification device based on the above-mentioned half-moon image to identify each object from movement images of livestock animals, extracts motion data of each identified object, and stores the extracted motion data. Another purpose is to provide an image-based livestock behavior pattern classification system and method that can classify and provide behavior patterns of each individual using spatiotemporal correlations.

The livestock behavior pattern classification system based on images of livestock animals in a farm according to the first aspect of the present invention for achieving the above-described technical problem includes one or two or more imaging devices, and the imaging devices include: An image acquisition device that provides images of livestock animals acquired by; a database server that stores and manages individual cross-sectional image data sets of livestock animals managed within the farm and images provided from the image acquisition device; an object identification device that extracts an object image from the images provided from the image acquisition device and identifies the object included in the object image using a unique cross-print image of each individual of the livestock animals; a behavior pattern classification device that extracts motion data for the objects identified by the object identification device using the images and classifies behavior patterns for each object using the motion data; Equipped with a video, it classifies and provides behavioral patterns for each individual livestock animal on the farm.

In the livestock behavior pattern classification system based on images according to the above-described first aspect, the entity identification device includes: an image input module that receives images from the image acquisition device; an entity extraction module that extracts an entity image of a livestock animal from the input image through image processing; a pose estimation module that estimates the pose of an object from the extracted object image; a half-moon area extraction module that extracts half-moon area location information of the object image based on the estimated posture of the object; And an object recognition device that compares a half-print image according to the extracted half-mark area location information with a half-mark image data set for each object stored in the database server to identify the object in the object image, and provides unique identification information for the identified object. It is desirable to provide a module to identify objects included in each image and provide unique identification information for the objects.

In the livestock behavior pattern classification system based on the image according to the above-described first aspect, the posture estimation module extracts skeleton information of the object from the object image, and uses a machine learning-based posture estimation model to identify the skeleton. It is characterized by estimating the posture of an object corresponding to the information, and the posture estimation model is preferably a machine learning model learned and modeled using learning data consisting of skeleton information for each posture of livestock animals.

In the livestock behavior pattern classification system based on the image according to the above-described first aspect, the half-moon area extraction module includes skeleton information and posture estimation module extracted from the object image using a machine learning-based half-moon area extraction model. Characterized by extracting the half-face area location information of the object image corresponding to the posture information estimated by It is preferable that it is a machine learning model that has been learned and modeled using .

In the livestock behavior pattern classification system based on the image according to the first aspect described above, the half-face image data set stored in the database server is a half-face image data set for each individual of the pre-registered livestock animals in the farm, It includes half-print image data for a plurality of half-mark areas corresponding to unique identification information of livestock animal individuals, and the half-mark area preferably includes at least a left side, right side, upper side, and back half-mark area for the livestock animal individual. do.

In the livestock behavior pattern classification system based on images according to the above-described first aspect, the behavior pattern classification device extracts movement data for each object by analyzing the current frame and previous frame of images for which individual identification has been completed. extraction module; a motion estimation module that measures a motion estimate value for each object according to a preset standard using the motion data for each object; It is desirable to include a behavior classification module that determines the movement as a dynamic movement if the motion estimate value for each object exceeds a preset motion threshold, and determines it as a static movement if not, to classify the behavior pattern for each object.

In the livestock behavior pattern classification system based on the image according to the above-described first aspect, the movement data for each entity includes posture information of the entity based on skeleton information, location information of the entity in the image, and consecutive image frames. It is desirable to include time information.

An individual identification device for livestock animals based on a half-faced image according to a second aspect of the present invention includes an image input module that receives images acquired by at least one or two imaging devices; A database in which individual cross-sectional image data sets of livestock animals managed within the farm are stored and managed; an entity extraction module that extracts an entity image of a livestock animal from the input image through image processing; a pose estimation module that estimates the pose of an object from the extracted object image; a half-moon area extraction module that extracts half-moon area location information of the object image based on the estimated posture of the object; and an object recognition module that compares the extracted half-mark image according to the location information of the half-mark area with a half-mark image data set for each object stored in a database, identifies the object in the object image, and provides unique identification information for the identified object. Provided with a , the unique half-print image of each individual of livestock animals is used to identify the individual included in the image and provide unique identification information for the individual.

In the object identification device for livestock animals based on the half-face image according to the above-described second aspect, the pose estimation module extracts skeleton information of the object from the object image and uses a machine learning-based pose estimation model. It is characterized in that the posture of the object corresponding to the skeleton information is estimated, and the posture estimation model is preferably a machine learning model learned and modeled using learning data consisting of skeleton information for each posture of livestock animals.

In the object identification device for livestock animals based on the semi-print image according to the above-described second aspect, the semi-print area extraction module includes skeleton information extracted from the object image using a machine learning-based semi-print area extraction model, It is characterized in that it extracts the position information of the half-face area of the object image corresponding to the posture information estimated by the posture estimation module, and the half-face area extraction model includes the half-face area location information according to the skeleton information and posture information of the livestock animal. It is preferable that it is a machine learning model that has been learned and modeled using the training data.

In the object identification device for livestock animals based on the half-face image according to the above-described second aspect, the half-face image data set stored in the database is a half-face image data set for each individual of livestock animals pre-registered in the farm. and includes half-mark image data for a plurality of half-mark areas corresponding to unique identification information of livestock animal entities, wherein the half-mark area includes at least half-mark areas on the left side, right side, upper side, and back of the livestock animal entities. It is desirable to do so.

The livestock behavior pattern classification method based on images according to the third aspect of the present invention is a method of classifying livestock behavior patterns based on images of livestock animals within a farm using a computing system, comprising: (a) within the farm; Saving in advance a data set of images of individual images of managed livestock animals in a database; (b) providing images of livestock animals from one or more imaging devices; (c) extracting an object image from the provided images and identifying the object included in the object image using the unique half-face image of each individual of livestock animals; and (d) extracting movement data for the identified entities using the images and classifying behavior patterns for each entity using the movement data; Equipped with a video, it classifies and provides behavioral patterns for each individual livestock animal on the farm.

In the method for classifying livestock behavior patterns based on images according to the third aspect described above, step (c) includes: (c1) extracting an individual image of a livestock animal from the input image through image processing; (c2) estimating the posture of an object from the extracted object image; (c3) extracting location information of the half-moon area of the object image based on the estimated posture of the object; and (c4) comparing a half-print image according to the extracted half-mark area location information with a half-mark image data set for each object stored in the database to identify the entity in the object image and providing unique identification information for the identified entity. A step is provided to identify an object included in each image and provide unique identification information for the object.

In the image-based livestock behavior pattern classification method according to the third aspect described above, step (c2) extracts skeleton information of the object from the object image and uses a machine learning-based posture estimation model to It is characterized by estimating the posture of an object corresponding to skeleton information, and the posture estimation model is preferably a machine learning model learned and modeled using learning data consisting of skeleton information for each posture of livestock animals.

In the image-based livestock behavior pattern classification method according to the above-described third aspect, step (c3) includes skeleton information extracted from the object image and a posture estimation module using a machine learning-based half-moon region extraction model. Characterized by extracting the half-face area location information of the object image corresponding to the posture information estimated by It is preferable that it is a machine learning model that has been learned and modeled using .

In the method for classifying livestock behavior patterns based on images according to the third aspect described above, step (d) includes extracting movement data for each entity by analyzing the current frame and previous frame of images for which entity identification has been completed; measuring a motion estimate for each object according to a preset standard using the motion data for each object; If the motion estimate value for each object exceeds a preset motion threshold, determining it to be a dynamic movement; otherwise, determining it to be a static movement. It is desirable to classify the behavior pattern for each object.

In the method for classifying livestock behavior patterns based on images according to the third aspect described above, the movement data for each entity includes posture information of the entity based on skeleton information, location information of the entity in the image, and consecutive image frames. It is desirable to include time information.

The livestock behavior pattern classification system according to the present invention can identify individuals using movement images of livestock animals and classify and provide behavior patterns for each identified individual. Through the system according to the present invention, it is possible to provide information about the behavior patterns of each individual, as well as information about the behavior patterns of each individual in each space and time.

In addition, the livestock behavior pattern classification system according to the present invention can collect data based on images and identify objects using skeleton information and half-moon image information based on the collected images.

In addition, the livestock behavior pattern classification system according to the present invention classifies and determines behavior patterns through movement tracking images for each identified object, and the resulting information is transmitted to the web service server and mobile app service server to provide users or workers. can be provided. Therefore, workers involved in the behavior and diagnosis of livestock animals, such as ranch managers and veterinarians, can classify and analyze behavior of livestock and monitor them anywhere in real time.

1 is a diagram illustrating the overall structure of an image-based livestock behavior pattern classification system according to a preferred embodiment of the first aspect of the present invention.
Figure 2 is a schematic diagram illustrating the behavior of livestock animals, such as dairy cows, during feeding time in a barn for livestock animals, such as dairy cows, in order to explain the livestock behavior pattern classification system according to the present invention.
Figure 3 is an exemplary illustration of an individual image data set for livestock animals registered on a farm in the image-based livestock behavior pattern classification system according to a preferred embodiment of the first aspect of the present invention. .
Figure 4 is an image-based livestock behavior pattern classification system according to a preferred embodiment of the present invention, illustrating a skeleton data set for each posture according to a multi-view view for livestock animals in a standing posture. It is shown.
Figure 5 shows a process in which the behavior pattern classification device estimates the movement of each individual livestock animal based on spatiotemporal correlation in the image-based livestock behavior pattern classification system according to the preferred embodiment of the first aspect of the present invention. This is a schematic diagram showing an example.
Figure 6 is a flowchart sequentially explaining a method for classifying livestock behavior patterns based on images according to a second aspect of the present invention.
Figure 7 is a flowchart sequentially explaining a livestock individual identification method implemented by the object identification device based on the half-moon image according to the third aspect of the present invention.

Hereinafter, with reference to the attached drawings, an individual identification method for livestock animals based on half-moon images according to the present invention, and an image-based livestock behavior pattern classification system and method using the object identification method will be described in detail.

First, the livestock behavior pattern classification system based on images according to a preferred embodiment of the first aspect of the present invention will be described in detail.

1 is a diagram illustrating the overall structure of an image-based livestock behavior pattern classification system according to a preferred embodiment of the first aspect of the present invention. Referring to Figure 1, the livestock behavior pattern classification system 1 based on images according to the present invention includes an image acquisition device 10 having one or two or more imaging devices, and a database server in which data is stored and managed ( 20), an object identification device 30 that extracts object images from images provided from the image acquisition device and identifies objects, and classifies behavior patterns for each object using movement data for objects identified from the images. It is equipped with a pattern classification device 40, and is characterized in that it classifies and provides behavioral patterns for each individual livestock animal in the farm using images. Meanwhile, the livestock behavior pattern classification system according to the present invention uses information from the database server 20, which stores data provided from an individual identification device and/or a behavior pattern classification device, to classify and analyze behavior images of livestock animals in a farm. A central management server 150 configured to provide results to users may be further provided. The central management server 150 provides web services and app services to user terminals 160 and 170, enabling behavior classification and analysis monitoring for related workers anywhere.

Figure 2 is a schematic diagram illustrating the behavior of livestock animals, such as dairy cows, during feeding time in a barn 100 for livestock animals, such as dairy cows, in order to explain the livestock behavior pattern classification system according to the present invention. Referring to FIG. 2, in the livestock house 100, there are large livestock 101 that are crowded into the feed line at feeding time and are feeding, and large livestock that are moving by approaching late to find an empty space where feeding is possible after the start of feeding ( 102), large livestock feeding feed from an automatic feeder (103), large livestock ruminating while resting inside the barn outside the feeding compartment (104), large livestock active in drinking water (105), in the feeding compartment There may be large livestock (106) that are resting inside the barn and are actively ruminating or have a significantly low number of ruminations.

In the livestock behavior pattern classification system according to the present invention, the individual identification device and the behavior pattern classification device are an edge computer such as an edge computer 120 installed on a farm, taking into account data communication performance in the network or processing performance of the system. It may be implemented in a computing system, may be implemented in the central management server 150 connected to the edge computer through the network 130, or may be partially implemented in the edge computer 120 and partially implemented in the central management server 150.

The image acquisition device 10 includes one or two or more imaging devices, and the imaging devices may be configured as surveillance cameras or the like. The image acquisition device is installed at various angles in each space within the farm or barn 100 to observe the animals within the barn from various angles. The image acquisition device captures multi-view images of various spaces within the livestock barn and provides the captured images together with the captured time and space information.

The database server 20 stores and manages individual cross-question image data sets for livestock animals registered in the farm. The object-specific cross-print image data set includes cross-print image data for a plurality of cross-print areas acquired according to a multi-view view for each individual livestock animal. It is preferable that the cross-face area includes at least the left side, right side, upper side, and back side of the livestock animal. In the registration process for livestock animal management, external images of the left/right/top/back of each livestock animal are acquired using a 2D or 3D camera, and stored and managed on a database server.

Figure 3 illustrates an example image data set for each individual for livestock animals registered on a farm in the image-based livestock behavior pattern classification system according to a preferred embodiment of the present invention. Referring to Figure 3, the half-moon image data set for each object includes object identification information (ID), right half-moon data record r(i,j), upper half-moon data record u(i,j), and left half-moon data record l(i, j), and may be configured to include a back statement data record b(i,j). As shown in Figure 3, the half-moon areas (L, R) on the left and right sides of the object are 3 rows and 4 columns, the half-moon area (U) looking at the object from above is 4 rows and 2 columns, and the half-moon area (U) looking at the object from the back is 4 rows and 2 columns. Since the area (B) is composed of 3 rows and 2 columns, the half-text image data for each object can have an overall structure of 3*4*2.

Additionally, the database server may store and manage a skeleton-based posture estimation data set for livestock animals and a half-face area location information data set for each posture. In addition, the database server contains images provided by the image acquisition device, object identification information acquired by the object identification device for each image, motion images and motion data for each object, and spatiotemporal information for each object. stored and managed.

In the present invention, unlike the condition in which a conventional camera films livestock animals passing through a fixed frame, images acquired at a CCTV installation location that observes the livestock barn from various positions are used, and livestock animals move freely within the livestock barn. It is characterized in that the appearances are used as input images. Therefore, the input images used in the present invention can be obtained in various forms depending on the distance from the camera, the shooting angle, the difference in gaze between the livestock animals and the camera view, etc., even if the livestock animals are in the same posture.

Figure 4 illustrates an example of a skeleton data set for each posture according to a multi-view view of a livestock animal in a standing posture in the image-based livestock behavior pattern classification system according to a preferred embodiment of the present invention. Referring to Figure 4, for example, when a livestock animal is in a standing position, an image showing only the hind legs and buttocks (a), an image showing the buttocks and part of the side (b), and an image showing the entire left side (c) , various types of images can be obtained, such as an image showing the entire right side (d), an image showing both the side and the back (e), and an image showing only the back (f). Therefore, the present invention implements a machine learning-based posture estimation model to estimate the posture from various movement images of livestock animals obtained from a 2D or 3D camera.

The object identification device 30 includes an image input module 31 that receives an image from the image acquisition device, an object extraction module 32 that extracts an object image of a livestock animal from the input image, and the extracted object image. A posture estimation module 33 for estimating the posture of an object from the object, a half-print area extraction module 34 for extracting location information of the half-print area of the object image based on the estimated posture for the object, and a half-print area of the extracted half-print area It is provided with an object recognition module 35 that identifies the object based on the image and provides unique identification information for the object, identifies the object included in each image and provides unique identification information for the object.

The object extraction module 32 distinguishes a background area and a motion region from the input image through image processing for object identification, and extracts a motion region corresponding to the object image for the livestock animal.

The posture estimation module 33 extracts the skeleton information of the object from the object image and estimates the posture of the object corresponding to the skeleton information using a machine learning-based posture estimation model. The posture estimation model is a machine learning model that is modeled by training and testing using learning data consisting of skeleton information for each posture according to a multi-view view of livestock animals. The animal's posture can be estimated. The training data and test data constituting the learning data specifically include skeleton information according to each posture of the livestock animal and the camera shooting angle.

The system according to the present invention uses the posture information for each individual livestock animal in the farm obtained through the posture estimation module to set a skeleton-based posture estimation data set for livestock animals, and stores and manages it in a database server. . In addition, the system according to the present invention uses the above-described skeleton-based pose estimation dataset as learning data to train and test the machine learning-based pose estimation model, thereby improving the accuracy and reliability of the pose estimation model. It is desirable to be configured.

Through the posture estimation result by the posture estimation module, it is possible to estimate which part of the '3D object of the livestock animal' the images obtained through the camera view of the image acquisition device show, and the head and buttocks of the livestock animal. The projection transformation of the half-moon image can be calculated based on skeleton information such as position, orientation, and posture.

The semi-print area extraction module 34 uses a machine learning-based semi-print area extraction model to extract the semi-print area location information of the object image corresponding to the skeleton information extracted from the object image and the posture information estimated by the pose estimation module. Extract. The half-moon area extraction model is a machine learning model that is modeled by training and testing using learning data consisting of skeleton information of livestock animals and half-moon area location information data sets for each posture. Therefore, the half-moon area extraction module extracts half-moon area location information, which is a region of interest to be extracted for object identification, from posture-based skeleton information.

Meanwhile, the half-faced area extracted through the posture estimation model using reference information such as the head, buttocks, and joint information such as the central axis and legs of the livestock animal is a part of each region indicated by the '3D object of the livestock animal'. Since image distortion according to the camera's projection angle is extracted, it is desirable to perform image correction processing.

The object recognition module 35 compares and searches the corrected half-mark image according to the extracted half-mark area location information and the object-specific half-mark image data set stored in the database server using a template matching method to identify the entity in the object image. , provides unique identification information for the identified entity. At this time, if the template matching result shows one or more livestock animals, the object recognition module performs object identification by searching for the next-ranked region according to the importance of posture-based region extraction.

The behavior pattern classification device 40 includes a motion data extraction module 42, a motion estimation module 44, and a behavior classification module 46, and uses motion data for objects identified from images to classify each object. Classify behavioral patterns. The motion data extraction module 42 extracts motion data for each object by analyzing the current frame and previous frame of images for which object identification has been completed. The movement data for each object preferably includes posture information of the object based on skeleton information, position information of the object in the image, and time information of consecutive image frames. The motion estimation module 44 uses the motion data for each object to measure a motion estimate for each object according to a preset standard. The action classification module 46 determines the motion as a dynamic motion if the motion estimate value for each object exceeds a preset motion threshold, and otherwise determines it as a static motion.

Figure 5 illustrates a process in which the behavior pattern classification device estimates the movement of each individual livestock animal based on spatiotemporal correlation in the image-based livestock behavior pattern classification system according to a preferred embodiment of the first aspect of the present invention. This is a schematic diagram. Referring to Figure 5, when the current posture according to the skeleton information is 'standing', if there is a position movement and a change in the skeleton, the movement is estimated to be 'walking', and if there is a change in the skeleton without a position movement, the movement is It can be assumed to be ‘sitting’. Meanwhile, in a state where the current posture according to the skeleton information is 'sitting', if there is a change in the skeleton without a position movement, the movement can be estimated as 'standing', and if there is both a position movement and a change in the skeleton, the movement can be 'standing'. It can be assumed to be ‘walking’.

The central management server 150 provides the results of classifying and analyzing the behavior images of livestock animals in the farm through the user's computer terminal (160 in FIG. 2) and a mobile terminal with a mobile app installed (170 in FIG. 2). By being configured to allow relevant workers to classify and analyze behavior from anywhere.

Through the above-described configuration, the system according to the present invention identifies each individual of livestock animals from the image acquired from the camera view, and based on data on how certain individuals of livestock animals move and behave, sequentially Provides behavior classification information for each entity to enable behavior analysis from video frames.

Hereinafter, the image-based livestock behavior pattern classification method according to the second aspect of the present invention will be described in more detail. The image-based livestock behavior pattern classification method according to the second aspect of the present invention may be implemented by a computing system including a database server.

Figure 6 is a flowchart sequentially explaining the image-based livestock behavior pattern classification method according to the second aspect of the present invention. Referring to FIG. 6, the livestock behavior pattern classification method according to the present invention first uses an object identification device for livestock animals (step 220) to identify each object included in sequentially input images. Provides unique identification information for the entity (step 202).

Next, using the unique identification information for each object recognized from sequentially input images, the spatiotemporal correlation between the current frame and the previous frame of the images is analyzed, motion data for each object is extracted, and the extracted motion data is used. Thus, it is determined whether there is movement for each object (step 210). Here, the movement data for each object includes the object's posture information based on skeleton information, the position information of the object in the video, and the time information of consecutive video frames.

Next, it is determined whether the motion estimate for each object exceeds a preset motion threshold (step 211). Movements for each object include static movements, including sitting, standing, and resting, and dynamic movements, including walking. Therefore, if the estimated movement value for each individual livestock animal exceeds the threshold in the above-described determination step, the individual is classified as a 'dynamic behavior pattern' (step 212). If not, the entity is classified as a 'static behavior pattern' (step 214).

If the classified and determined movements are classified into dynamic behavior patterns, the movements of each individual, such as moving, stopping, standing up, or lying down, are converted into information. Meanwhile, if the classified and determined movements are classified into static behavior patterns, the movements of each individual, such as sitting and resting, standing and resting, etc., are converted into information. The movement information for each individual classified in this way becomes unit module information for behavioral analysis of livestock animals. In this way, once the classification of the behavior pattern for each object is determined, the behavior classification preprocessing task is completed.

The behavior classification device according to the present invention described above can identify each individual livestock animal from images acquired from image acquisition devices such as camera views, and uses such individual identification information to identify individual movements and behaviors of each livestock animal. can be identified, and behavioral classification information for each entity can be provided using successive video frames. As a result, it is possible to analyze individual behavior using the individual behavior classification information provided by the present invention.

Hereinafter, the configuration and operation of an individual identification device for livestock animals based on a half-question image according to the third aspect of the present invention will be described in more detail. The individual identification device for livestock animals based on a half-question image according to the third aspect of the present invention has the same configuration as the individual identification device 30 of the livestock behavior pattern classification system according to the first aspect of the present invention described above. . The object identification device for livestock animals based on half-face images according to the third aspect of the present invention uses the unique half-face images of each individual of livestock animals to identify an object included in the image and provide information about the corresponding object. It is characterized by providing unique identification information.

Hereinafter, an individual identification method using an individual identification device for livestock animals based on a half-moon image according to the third aspect of the present invention described above will be described in more detail.

Figure 7 is a flowchart sequentially explaining a livestock individual identification method implemented by the object identification device based on the half-moon image according to the third aspect of the present invention. Referring to FIG. 7, the object identification method according to the present invention separates the background area and the motion area from the input image through image processing for entity extraction, and separates the object image for the livestock animal from the background area (step 221). The object image may be a bounding box image including livestock animals or an image of livestock animals.

Next, skeleton information about the object is extracted from the object image (step 222).

Next, the posture of the object is estimated based on the skeleton information using a machine-learned posture estimation model (step 223). Here, the posture estimation model is a machine learning model that is tested and modeled after training with a skeleton data set for each posture according to a multi-view view of livestock animals as learning data. The pose estimation model is a machine learning model that is tested and modeled using skeleton data for livestock animals. The posture can be estimated.

The posture estimation result can estimate which part of the livestock animal the images obtained through the camera view of the image acquisition devices show, and is based on skeleton information such as the position, direction angle, and posture of the livestock animal's head and buttocks. Thus, the projection transformation of the half-moon image can be calculated. Therefore, based on the estimated posture, location information on the half-moon area of the livestock subject, which is the area of interest, is extracted (step 224).

The half-face area location information extracted through the posture estimation model using reference information such as the head, buttocks, and joint information such as the central axis and legs of the livestock animal is part of each region indicated by the '3D object of the livestock animal'. , Image distortion is extracted according to the projection angle of the camera, which is an image acquisition device. Therefore, in order to remove this image distortion, the semi-print image for the semi-print area is corrected (step 225).

Next, based on the corrected half-mark image and the half-mark area location information, the individual is identified by searching through template matching with the half-mark image data sets for each object registered in the farm (step 226). At this time, if the template matching result includes two or more entities, entity identification is performed by searching for the next ranked half-moon area according to the importance of posture-based half-moon area extraction.

Identified entity information provides real-time monitoring of each entity along with captured images of livestock animals to users through web services and app services.

Although the present invention has been described above with a focus on preferred embodiments, this is only an example and does not limit the present invention, and those skilled in the art will understand that it does not deviate from the essential characteristics of the present invention. It will be apparent that various modifications and applications not exemplified above are possible within the scope. In addition, these variations and differences in application should be construed as being included in the scope of the present invention as defined in the appended claims.

1: Livestock behavior pattern classification system
10: Image acquisition device
20: Database server
30: object identification device
31: video input module
32: Object extraction module
33: Posture estimation module
34: Half-question region extraction module
35: Object recognition module
40: Behavior pattern classification device
42: Movement data extraction module
44: Motion estimation module
46: Behavior classification module

Claims (17)

In the livestock behavior pattern classification system based on images of livestock animals on the farm,
An image acquisition device comprising one or more imaging devices and providing images of livestock animals acquired by the imaging devices;
a database server that stores and manages individual cross-sectional image data sets of livestock animals managed within the farm and images provided from the image acquisition device;
an object identification device that extracts an object image from the images provided from the image acquisition device and identifies the object included in the object image using a unique cross-print image of each individual of the livestock animals; and
a behavior pattern classification device that extracts motion data for the objects identified by the object identification device using the images and classifies behavior patterns for each object using the motion data;
Equipped with a video, it classifies and provides behavioral patterns for each individual livestock animal on the farm,
The object identification device,
an image input module that receives an image from the image acquisition device;
an entity extraction module that extracts an entity image of a livestock animal from the input image through image processing;
a posture estimation module that extracts skeleton information of an object from the extracted object image and estimates a posture of the object corresponding to the skeleton information using a machine learning-based posture estimation model;
a half-moon area extraction module that extracts half-moon area location information of the object image based on the estimated posture of the object; and
An object recognition module that compares a half-print image according to the extracted half-mark area location information with a half-mark image data set for each object stored in the database server to identify the object in the object image and provides unique identification information for the identified object. ;
A livestock behavior pattern classification system based on images, characterized in that it identifies the objects included in each image and provides unique identification information for the objects. delete The method of claim 1, wherein the posture estimation module:
An image-based livestock behavior pattern classification system characterized by being a machine learning model learned and modeled using learning data consisting of skeleton information for each posture of livestock animals. The method of claim 1, wherein the response region extraction module,
Using a machine learning-based half-moon area extraction model, the half-moon area location information of the object image corresponding to the skeleton information extracted from the object image and the posture information estimated by the posture estimation module is extracted,
The image-based livestock behavior pattern classification system is characterized in that the half-moon area extraction model is a machine learning model learned and modeled using learning data consisting of half-moon area location information according to the skeleton information and posture information of livestock animals. . The method of claim 1, wherein the half-face image data set stored in the database server is a half-face image data set for each of the pre-registered livestock animals in the farm,
Contains half-print image data for a plurality of half-mark areas corresponding to unique identification information of livestock animal entities,
An image-based livestock behavior pattern classification system, characterized in that the half-mark area includes at least the left side, right side, upper side, and back half-mark area for each livestock animal. The method of claim 1, wherein the behavior pattern classification device,
A motion data extraction module that extracts motion data for each object by analyzing the current frame and previous frame of images for which object identification has been completed;
a motion estimation module that measures a motion estimate value for each object according to a preset standard using the motion data for each object;
a behavior classification module that determines the motion as dynamic if the motion estimate for each object exceeds a preset motion threshold, and determines as static if not;
An image-based livestock behavior pattern classification system characterized by classifying individual behavior patterns. The method of claim 6, wherein the movement data for each object is:
An image-based livestock behavior pattern classification system that includes posture information of an object based on skeleton information, location information of an object in the image, and time information of consecutive image frames. an image input module that receives images acquired by at least one or more imaging devices;
A database in which individual cross-sectional image data sets of livestock animals managed within the farm are stored and managed;
an entity extraction module that extracts an entity image of a livestock animal from the input image through image processing;
a posture estimation module that extracts skeleton information of an object from the extracted object image and estimates a posture of the object corresponding to the skeleton information using a machine learning-based posture estimation model;
a half-moon area extraction module that extracts half-moon area location information of the object image based on the estimated posture of the object; and
an object recognition module that compares a cross-print image according to the extracted cross-print area location information with a set of object-specific cross-print image data stored in a database to identify the object in the object image and provides unique identification information for the identified object;
Equipped with, using the unique half-face image of each individual of livestock animals, identifying the object included in the image and providing unique identification information for the object, to livestock animals based on the half-face image. Object identification device for. The method of claim 8, wherein the posture estimation module:
An object identification device for livestock animals based on half-face images, characterized in that it is a machine learning model learned and modeled using learning data consisting of skeleton information for each posture of livestock animals. The method of claim 8, wherein the response region extraction module,
Using a machine learning-based half-moon area extraction model, the half-moon area location information of the object image corresponding to the skeleton information extracted from the object image and the posture information estimated by the posture estimation module is extracted,
The half-moon region extraction model is a machine learning model learned and modeled using learning data consisting of half-moon area location information according to the skeleton information and posture information of livestock animals. Object identification device. The method of claim 8, wherein the half-face image data set stored in the database is a half-face image data set for each individual of pre-registered livestock animals in the farm,
Contains half-print image data for a plurality of half-mark areas corresponding to unique identification information of livestock animal entities,
Wherein the half-mark area includes at least left, right, upper, and back half-mark areas of the livestock animal. In a method of classifying livestock behavior patterns based on images of livestock animals on a farm using a computing system,
(a) Saving in advance a data set of individual images of livestock animals managed on a farm in a database;
(b) providing images of livestock animals from one or more imaging devices;
(c) extracting an object image from the provided images and identifying the object included in the object image using the unique half-face image of each individual of livestock animals; and
(d) extracting motion data for the identified objects using the images and classifying behavior patterns for each object using the motion data;
Equipped with an image, it classifies and provides behavioral patterns for each individual livestock animal on the farm,
In step (c),
(c1) extracting an object image of a livestock animal from the provided image through image processing;
(c2) extracting skeleton information of the object from the extracted object image and estimating the posture of the object corresponding to the skeleton information using a machine learning-based posture estimation model;
(c3) extracting location information of the half-moon area of the object image based on the estimated posture of the object; and
(c4) comparing a half-print image according to the extracted half-mark area location information with a half-mark image data set for each object stored in the database to identify the entity in the object image and providing unique identification information for the identified entity. ;
A livestock behavior pattern classification method based on images, characterized in that it identifies the objects included in each image and provides unique identification information for the objects. delete The method of claim 12, wherein step (c2),
An image-based livestock behavior pattern classification method, characterized in that it is a machine learning model learned and modeled using learning data consisting of skeleton information for each posture of livestock animals. The method of claim 12, wherein step (c3),
Using a machine learning-based half-moon area extraction model, the half-moon area location information of the object image corresponding to the skeleton information extracted from the object image and the posture information estimated by the posture estimation module is extracted,
The image-based livestock behavior pattern classification method, wherein the half-moon area extraction model is a machine learning model learned and modeled using learning data consisting of half-moon area location information according to the skeleton information and posture information of livestock animals. . The method of claim 12, wherein step (d) is,
extracting motion data for each object by analyzing the current frame and previous frame of images for which object identification has been completed;
measuring a motion estimate for each object according to a preset standard using the motion data for each object;
If the motion estimate value for each object exceeds a preset motion threshold, determining it to be a dynamic movement; otherwise, determining it to be a static movement;
A method for classifying livestock behavior patterns based on video, characterized by classifying behavior patterns for each individual. The method of claim 16, wherein the movement data for each object is:
A livestock behavior pattern classification method based on video, characterized by including posture information of an object based on skeleton information, position information of an object in the video, and time information of consecutive video frames.