KR20210066710A - System and method for analyzing object tracking and object behavior using deep learning algorithms - Google Patents

Abstract

The present invention relates to a smart livestock management system using deep learning-based object tracking and behavior analysis and a method thereof. The smart livestock management system using deep learning-based object tracking and behavior analysis comprises: a brainwave detection module attached to the head of a pig object; a first photographing device for photographing a motion image of the pig object; a second photographing device that generates a thermal image of the pig object; a sound collecting device for collecting sound information; a monitoring server for predicting and determining whether the corresponding pig object has a disease and a type of the disease; and a manager terminal for receiving a disease outbreak notification message of the pig object from the monitoring server. Therefore, the smart livestock management system using deep learning-based object tracking and behavior analysis and the method thereof can predict whether a disease occurs in pigs.

Description

Smart livestock management system and method using deep learning-based object tracking and behavior analysis {System and method for analyzing object tracking and object behavior using deep learning algorithms}

The present invention relates to a smart livestock management system and method using deep learning-based object tracking and behavior analysis.

CCTV and related systems and devices have been built in a wide range of areas, both indoors and outdoors, to realize various purposes such as crime prevention, safety accident prevention, disaster monitoring, building and facility monitoring, identification of passengers using public transportation, parking, and traffic condition monitoring. It can be said that these devices or systems are gradually increasing due to various social needs.

These CCTVs and related systems are used in various ways, and CCTVs are usually installed at multiple locations in a specific area, and image data captured from these multiple CCTVs are output to the screen display means for control, and managers (controllers, Users, etc.) monitor a specific event while watching it, or create a DB of images from various places or locations and use them for ex post evidence.

In addition, a method of tracking an object of interest in a captured image using a video analysis (VS, Video Analysis) technique, etc. and monitoring the movement of the tracked object is also applied to the conventional CCTV or control system, and recently tracking A method of transmitting alarming information to a manager or the like when an abnormal behavior or behavior of an object is detected is also being used little by little.

Methods such as object detection or tracking include a method of detecting an object included in image data using an image difference method, an optical flow method, etc., and furthermore, a Mean-Shift method, a Kalman filter Methods for tracking an object using a Kalman filter or the like are used.

However, since these methods related to object detection and tracking are designed to be processed by a limited specific environment or feature parameters that can be detected and tracked by the method, performance is expressed to a certain extent in the limited environment specialized for each technique. However, in a situation or environment that is not suitable for these techniques, there is a problem that the efficiency of object detection and tracking is considerably lowered.

For this reason, conventional techniques and methods sensitively respond to changes in noise in the external environment, reducing the reliability of detection/tracking, and since it is difficult to continuously and accurately detect and track an object, it is essential that it cannot be universally applied to various environments. It can be said that there are limits.

In addition, even if a dynamic object is detected or tracked, the conventional method is at a level of simply expressing the result to a manager, etc., so it is still a human-dependent method in applying and subsequently utilizing the result. Therefore, it can be said that it is almost impossible to implement effective interfacing between the system and the person (administrator).

On the other hand, the present invention uses the above-described deep learning object detection and tracking technique and the volume-specific temperature change according to the sound and movement of pigs to analyze the movement path and behavior of pigs in the pig house, as well as the presence or absence of disease and the type of disease in pigs. We would like to present a predictable system and method.

Unexamined Patent Publication No. 10-2019-0058206

An object of the present invention is to provide a smart livestock management system and method using deep learning-based object tracking and behavior analysis that can solve conventional problems.

Smart livestock management system using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention for solving the above problem is attached to the head of a pig object in a pig house, and measures and transmits the EEG signal of the pig object an EEG detection module; a first imaging device for photographing a motion image of a pig object in the pig house; a second imaging device generating a thermal image of the pig object; a sound collecting device for collecting sound information including breathing and coughing sounds of the pig object; a monitoring server for predicting and judging the presence or absence of disease and the type of disease in the pig object based on the EEG signal, the motion image, the thermal image and the sound information; and a manager terminal receiving a disease outbreak notification message of a pig object from the monitoring server, wherein the monitoring server detects a sleep state, an abnormal seizure state, a feeling of depression and a feeling of depression of the pig object through the EEG variability diagram of the EEG signal detected in the pig object. After classifying the related EEG (electroencephalographic, EEG), the EEG analysis unit to analyze the pattern of EEG variability in comparison with the reference; When a target object (pig) is input from the manager terminal, a path tracking unit for tracking the movement path of the target object in reverse order based on the position signal of the EEG detection module located in the target object (pig); a behavior pattern and posture analysis unit that analyzes the behavior pattern and posture of the pig object based on a motion image for a change in movement of the targeted pig object; a body temperature analyzer analyzing the body temperature of the pig object based on the thermal image; an acoustic analysis unit for analyzing at least one of a breathing cycle of the pig object, a size and generation cycle of a cough sound, and a cry sound based on the acoustic information; And the presence or absence of disease is first determined based on the EEG variability pattern of the pig object, irregular posture changes of behavioral patterns, respiratory cycle, and cough sound, and whether or not disease occurs based on the body temperature and body temperature change of each part of the pig object selected first and a disease occurrence determination unit for secondary determination, wherein the behavioral posture and pattern analysis unit classifies the change of the pose pattern of the pig in the pig house into any one of Standing, Lying, and Sitting based on the motion image, and when the pig moves , the pose pattern change is analyzed as a distance versus time, the body temperature analysis unit analyzes the body temperature change for each part of pigs that are heated according to the pose pattern change, and the disease occurrence determination unit learns the deep learning algorithm when a disease occurs It is characterized in that the pig object is first selected based on the change of the time-series pose pattern, breathing cycle, and cough sound of the pigs.

A smart livestock management method using deep learning-based object tracking and behavior analysis according to another embodiment for solving the above problem comprises the steps of: measuring an EEG signal of a pig object in a pig house using an EEG detection module; When the target object (pig) is input from the manager terminal in the path tracking unit, tracking the movement path of the target object in reverse order based on the position signal of the EEG detection module located in the target object (pig); After classifying electroencephalographic (EEG) related to sleep state, abnormal seizure state, and depression of pigs through the EEG variability diagram of EEG signals detected in the pig object in the EEG analysis unit, the pattern of EEG variability is analyzed by comparing it with a reference. step; When the pig object moves based on the motion image for the change in the movement of the targeted pig object in the behavior pattern and posture analysis unit, the pose pattern change is analyzed as distance versus time, and then sample data of diseased pigs judging whether the behavior is abnormal by comparing with When the corresponding pig object is determined to be abnormal, collecting acoustic information including the breathing and coughing sounds of the pig object in the acoustic information collection unit; First determining the presence or absence of a disease based on the EEG variability pattern of the target object (pig), irregular posture change of the behavior pattern, breathing cycle, and cough sound in the vaginal occurrence determination unit; when the target object (pig) is determined to be disease-prone, collecting thermal image images for each body part of the pig object in a second imaging device; predicting the presence or absence of disease and the type of disease in the pig object based on the thermal image in the disease occurrence determination unit; and when it is predicted that the corresponding pig object has a disease, the notification unit generates a disease outbreak notification message and sends it to the manager terminal.

Using the smart livestock management system and method using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention, the type of object (pig) in the pig house, movement pattern, body area/weight change through a deep learning algorithm , pose pattern change can be easily determined, and furthermore, it has the advantage of being able to predict/prevent abnormality (disease) condition according to the movement pattern of the object (pig), body area/weight change, and pose pattern change at an early stage.

Therefore, the smart livestock management system and method using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention is based on the result of analyzing the pose pattern change of the targeted pig object as distance versus time, based on the corresponding pig Detects whether an object has behavioral abnormality, and in the case of a pig object detected as a behavioral abnormality, predicts the occurrence of primary disease by the pig object's brainwave mutation pattern, breathing and coughing sounds according to the change of the pose pattern, and then changes the pose pattern There is an advantage in that it is possible to predict whether or not a disease will occur in pigs by volume by predicting the occurrence of a secondary disease based on changes in body temperature for each quartile.

1 is a block diagram illustrating a smart livestock management system using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention.
2 is an exemplary diagram of a CNN architecture according to an embodiment of the present invention.
3 is an explanatory diagram of a faster R-CNN (Faster R-CNN) model applied in the present invention.
4 is an explanatory diagram of a fully convolution networks (FCN) model applied in the present invention.
5 is an exemplary diagram showing the extraction and display of movement paths and pose change patterns for each pig house/individual.
6 is a flowchart illustrating a smart livestock management method using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention.
7 is a block diagram illustrating a smart livestock management system using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention.
8 is an exemplary diagram illustrating a process of filtering breathing and coughing sounds, crying sounds, and urination sounds from ambient sounds in the acoustic information collected by the acoustic information collecting unit.
9 is an exemplary view showing the waveform of the EEG classified by the EEG analysis unit of FIG. 7 .
10 is a flowchart illustrating a smart livestock management method using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention.
11 illustrates an example computing environment in which one or more embodiments disclosed herein may be implemented.

Hereinafter, embodiments will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in the description of the embodiment, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and used by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Hereinafter, a smart livestock management system and method using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention and another embodiment will be described in more detail based on the accompanying drawings.

1 is a block diagram illustrating a smart livestock management system using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention, and FIG. 2 is an exemplary diagram of a CNN architecture according to an embodiment of the present invention, 3 is an explanatory diagram of a faster R-CNN (Faster R-CNN) model applied in the present invention, FIG. 4 is an explanatory diagram of an FCN (Fully convolution networks) model applied in the present invention, and FIG. 5 is a pig house/individual It is an example diagram showing the extracted movement path and pose change pattern of each star.

The smart livestock management system 10 using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention shown in FIG. 1 includes an image acquisition unit 100 and an artificial intelligence server 200 .

The image acquisition unit 200 acquires object images (face, body, legs, etc.) of objects (pigs) in a pig house from a camera. In one embodiment, it is possible to acquire an image of pigs in a pig house from a CCTV or IPTV camera, and it is also possible to additionally acquire an image of each pig part collected through a wired/wireless communication network.

Next, the artificial intelligence server 200 extracts a plurality of object (pig) image regions in the pig house image, and the part of the object (pig) within the extracted plurality of object (pig) image regions using a deep learning learning algorithm. After segmenting the image blocks for each part, the characteristics of the image blocks for each part are learned, and the type and movement pattern of the object (pig) in the pig house is based on the characteristics of the image block for each part. , a server that predicts an abnormal state of an object (pig) by determining a change in body area/weight, and a change in pose pattern.

More specifically, the artificial intelligence server 200 includes an object recognition unit 210 , a deep learning learning unit 220 , an object tracking unit 230 , an object entity analysis unit 240 , and an object pose pattern change analysis unit 250 . , a heatmap generating unit 260 , a statistical processing unit 270 , an abnormal state prediction unit 280 , and a weight change analysis unit 290 .

On the other hand, the object recognition unit 210, deep learning learning unit 220, object tracking unit 230, object entity analysis unit 240, object pose pattern change analysis unit 250, heatmap generation unit ( 260), the statistical processing unit 270, the abnormal state prediction unit 280, and the weight change analysis unit 290 may be configured in the form of a terminal and/or a server, and may communicate with each other through a network.

Here, the network network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a personal area network (PAN), a mobile communication network ( mobile radio communication network), Wibro (Wireless Broadband Internet), Mobile WiMAX, HSDPA (High Speed Downlink Packet Access), or all kinds of wired/wireless networks such as satellite communication networks.

The object recognition unit 210 extracts an object (pig) image region in the pig house image using a deep learning object recognition algorithm. Here, the deep learning object recognition algorithm includes a faster R-CNN based feature map (ResNet 101 backbone) and a Fully Convolutional Network (FCN) (see FIGS. 2 to 4).

As can be seen in Figure 2, the CNN model outputs the result for the input after processing through the layer stack including convolution (co-evolution), activation, pooling, regularization, and full concatenation. Compared to conventional approaches, CNNs have the advantages of requiring less image pre-processing and extracting features through learning, eliminating the need for expertise in the manual design of feature extractors. In addition, region-based regional neural network (R-CNN: Regions with CNN features) is to accurately identify main objects in one image by expressing them as a bounding box. It takes image data as input, displays an area with a box (Bounding box), and has a form of labeling (class label) for each object as output. That is, this is a method of recognizing an object by first suggesting area candidates in which an object is likely to be in the image and scoring them.

Referring to FIG. 3, Faster R-CNN includes a region proposal network (RPN) and a fast R-CNN detector. There are three main steps in the process of applying faster R-CNN to object detection.

First, a convolutional neural network is used for feature extraction, and a feature map is generated in the last layer. Second, RPN generates Regional Proposals in various aspect ratios and scales based on functional maps. And the region proposal generated in the third step feeds into a fast R-CNN detector for classification and bounding box regression.

More specifically, looking at the feature extraction process using CNN, the CNN layer of the Zeiler-Fergus network includes 5 convolutional layers and 3 max pooling layers.

The convolutional layer uses a filter that slides over the pixel array of the input image, and the dot product between the filter and the sub-array is computed. The value of the dot product and the bias value are added to obtain the convolution result for the subarray.

At this time, the initial weights of filters and biases are randomly assigned and can be continuously adjusted during training through a stochastic gradient descent (SGD) algorithm. The max pooling layer can reduce the spatial size of the input data by taking the max value from the sub-array of the input pixel array.

The ReLU function is used as an activation function, and element-wise activation can be applied with the max (0, x) function to reduce computational cost and improve accuracy. To avoid overfitting problems, dropout layers are designed to disconnect connections between neurons in two adjacent layers at a specific dropout rate.

The regularization layer is applied to help whiten the input image without affecting the learning rate, which can lead to high learning rate and fast convergence rate.

In addition, region proposal networks (RPNs) learn to generate object regions with various aspect ratios and ratios using anchors in faster R-CNNs (R-CNNs). A small sliding window is proposed to slide over the feature map and then projected onto a lower-dimensional feature (256 dimensions for the ZF network).

After convolution with two 1×1 filters and ReLU functions, the extracted features are fed to the bounding box regression layer and the classification layer, respectively. Non-maximal inhibition (NMS) can be used to merge proposals with high combinations (IoU). At this time, the proposals are ranked according to the object score, and only the top N ranked proposals are kept.

In addition, regional proposals generated through RPN are utilized as input regions of interest (RoI) of the high-speed R-CNN detector. For each RoI, the features of the convolutional layer are transformed into a fixed-length vector through the RoI pooling layer. Each fixed-length feature vector is fed as a sequence of fully connected layers, and the final feature vector is fed into a softmax layer to generate a regression layer that outputs the probability scores for the five classes and the relative coordinates of the bounding box.

Next, referring to FIG. 4, FCN (Fully Convolutional Network) changes the Fully-Connected Layer to the corresponding Convolution Layer, there is no difference from the CNN until the part that passes it, and is an algorithm that wants to make the output of the network different. It is not a classification score for a class, but an algorithm that attempts to generate a classification score map on a two-dimensional image.

Meanwhile, the object recognition unit 210 may perform one or more pre-processing of binarization, sessionization, and noise removal on an object (pig) image region.

In addition, using a Gaussian filter, a Laplacian filter, a Gaussian difference (Difference of Gaussian: DoG), and Canny edge detection, it is possible to transform or improve the pig house image.

In addition, noise reduction using a Gaussian filter, etc., a gradient operation for edge component detection is performed, and non-maximum suppression interpolating a broken edge line is performed (non-maximum value suppression). maximum suppression) and hysteresis thresholding for binarizing the edge map may be performed.

Next, the deep learning learning unit 220 extracts and divides an image block (patch) for each part of the object (pig) within the image region of a plurality of objects (pig) extracted using the deep learning learning algorithm, and then , after learning the characteristics of the image patch for each part, and performing separation learning to separate adjacent objects adjacent in the image block for each part, the type of the object (pig) (egg / sow), It generates learning data of movement pattern and pose pattern change.

The deep learning learning unit 220 extracts and divides the image block for each part, performs batch normalization, and verifies and learns the characteristics of the object for each part of the object.

For reference, batch normalization refers to the operation of normalizing the activation value or output value of the activation function. It is the task of normalizing the distribution of data (batch) in each layer of the neural network. As a method of adding a kind of noise, it is normalized for each batch, so the variance and value of the mean for the entire data may be different.

During normalization in each hidden layer, the input distribution becomes constant, and as a result, even if the learning rate is set large, the learning speed is increased as a result.

In other words, batch normalization refers to normalizing a mini-batch during training as a unit, and means to normalize the distribution so that the mean is 0 and the variance is 1.

As an example of batch normalization, first, the average and variance of a mini-batch used as an input are calculated, and normalization is performed so that the mean is 0 and the variance is 1 for the activation value/output value of the hidden layer. As a result, the data distribution becomes less skewed, and enlargement scale and shift transformation are performed at each batch normalization step.

On the other hand, deep learning is a technology used to cluster or classify objects or data. It is a technology that inputs a large amount of data into a computer and classifies similar ones.

At this time, many machine learning algorithms have already appeared on how to classify data. Deep learning is an artificial intelligence learning method proposed to overcome the limitations of artificial neural networks.

The deep learning learning unit 220 disclosed herein employs a deep learning learning algorithm, and the deep learning learning algorithm is a Deep Belief Network, an Autoencoder, a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Deep Q-Network, etc. may include, and the deep learning learning algorithm listed in the present invention is only an example, and is not limited thereto.

Next, the object tracking unit 230 tracks the movement path pattern of at least one or more objects (pigs) detected in the pig place image based on the learning data. The object tracker 230 may use a deep learning object tracking algorithm such as Meanshift, camshift, kalman tracker, GOTURN, and TLD.

Next, the object analysis unit 240 analyzes the type of at least one or more objects (pigs) detected in the pig place image and the number of individuals per pig place based on the learning data.

Next, the object pose pattern change analysis unit 250 analyzes the pose change pattern of the object (pig) in at least one object (pig) image area detected in the pig house image based on the learning data.

The object pose pattern change analysis unit 250 may use svm, Resnet, Alexnet, VGGNet, MobileNet, and the like, but is not limited thereto.

Here, the pose change pattern is a change in posture of at least one of Standing (standing state), Lying (lying state), and Sitting (sitting state).

Next, the weight change analysis unit 290 is the body area of the object (pig) in the image area of at least one or more objects (pigs) detected in the pig place image based on the learning data, and the change in body weight according to the body area analyze and judge

The weight change analysis unit 290 may generate a predetermined correlation expression between body area and actual weight, and may provide important data for determining standardization for each seed and pig shipping time by calculating an approximation of the weight through this. .

The heatmap generating unit 260 is a heat map for each type of object (pig) (pig / sow) in the at least one object (pig) image area detected in the pig place image based on the learning data and the movement path pattern. create

The statistical processing unit 270 analyzes the pose change pattern over time of the object based on the analysis data of the object entity analysis unit, the object tracking unit, the object behavior analysis unit, and the heatmap generation unit. create

The abnormal state prediction unit 280 predicts an abnormal state (disease) of the object (pig) based on the behavior change statistical data and the result of the weight change analysis unit.

The abnormal state (disease) is swine reproductive and respiratory syndrome (PRRS), swine influenza (SIV), swine epidemic diarrhea (PED), Augeckis disease, Brachispira infection, swine fever (CSF), critosporidiosis, foot-and-mouth disease (FMD), Hepatitis E virus (HEV), Lawsonia intracellularis, Mycoplasma pneumonia (M.hyo), Parvovirus (PPV), Pasteurella disease, Rotavirus, Salmonellosis, Brachyspira hyodysenterie , swine blisters, infectious gastroenteritis, and adenomatosis may be at least one or more.

For reference, AFS is caused by African swine fever virus (ASFV), which infects domestic pigs, warthogs and wild boars, and this disease is transmitted when a healthy animal comes in direct contact with a diseased animal, or an infected prey. It can be a disease that can be transmitted through indirect contact through, as well as through biological agents (such as soft mites).

Signs of AFS can occur in hyperacute, acute, subacute and chronic forms, and mortality rates vary from 0 to 100%. This depends on the virulence of the virus that infects pigs, and the characteristic that can be identified as an acute disease is a short incubation period of 3 to 7 days, followed by high fever (up to 42°C) and death within 5 to 10 days. The most consistent clinical signs are loss of appetite, depression and recumbent behavior and, among other signs, bleeding from the skin of the ears, abdomen, and legs or shortness of breath, vomiting, bleeding from the nose or rectum, sometimes accompanied by diarrhea . Also, weight loss, joint swelling and respiratory problems. Cases of this type of disease are extremely rare in a group outbreak.

Aujeszky's disease is a contagious viral disease caused by a herpes virus called Pseudorabies virus (PRV). The most common form is acute febrile syndrome, affecting mainly pigs (the main viral pathogen). However, other animal species are also susceptible to the disease. Clinical signs depend on the age and physiological development of the infected animal. For example, in the case of piglets, it causes neurological diseases (swivel movements, seizures) and soon leads to death, and in the case of growing animals, it mainly causes respiratory and gastrointestinal problems, delaying growth.

Swine fever (CSF) is the second most serious disease after aphthous fever among infectious diseases of pigs and wild boars, and poses a serious threat to pig husbandry and has a great socioeconomic impact. This disease is caused by enveloped RNA belonging to the genus Pestivirus of the Flavivirus family. CSF cannot be transmitted to humans, and symptoms appear in various ways depending on the toxicity of the infecting virus and the growth stage of the animal.

Signs are that the superacute form can result in virtually asymptomatic death within 48 hours, but the more common acute form of the disease develops symptoms of high fever (up to 42°C) as the first stage. Recognizable, diseased animals at this stage become lethargic, stop feeding and show symptoms of conjunctivitis with mucus discharge from the eyes, gastrointestinal and respiratory problems, blood imbalances, and neurological disorders.

6 is a flowchart illustrating a smart livestock management method using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention.

As shown in FIG. 6 , the smart livestock management method ( S700 ) using deep learning-based object tracking and behavior analysis according to an embodiment of the present invention includes a plurality of livestock shots taken from different angles in real time. Acquire a pig house image (S710).

Thereafter, the artificial intelligence server 200 extracts a plurality of object (pig) image regions in the pig house image, and each part of the object (pig) within the extracted plurality of object (pig) image regions using a deep learning learning algorithm After dividing the image block (patch), the characteristics of the image block (patch) for each part are learned, and the type of object (pig) in the pig house, the movement pattern, based on the characteristics of the image block (pach) for each part, By determining the change in the pose pattern, the abnormal state of the object (pig) is predicted ( S720 ).

The process S720 may include the following process.

First, the object recognition unit 210 uses a deep learning object recognition algorithm to extract an object (pig) image region in the pig house image (S721).

Here, the deep learning object recognition algorithm includes a faster R-CNN based feature map (ResNet 101 backbone) and a Fully Convolutional Network (FCN).

The step S721 includes performing one or more preprocessing of binarization, sessionization, and noise reduction on the object (pig) image region, and includes a Gaussian filter, a Laplacian filter, and a Gaussian difference. Gaussian: DoG) and Canny edge detection may be used to transform or improve a pig house image.

In addition, noise reduction using a Gaussian filter, etc., a gradient operation for edge component detection is performed, and non-maximum suppression interpolating a broken edge line is performed (non-maximum value suppression). maximum suppression) or a process of performing hysteresis thresholding for binarizing the edge map.

Next, when the S721 process is completed, the image block (patch) for each part of the object (pig) is divided within the plurality of object (pig) image regions extracted by the deep learning learning unit 220 using the deep learning learning algorithm. Then, after learning the characteristics of the image patch for each part, and performing separation learning to separate adjacent objects in the image block for each part, the type of the object (pig) (egg / sow) ), movement patterns, and pose pattern changes are generated as learning data.

Thereafter, the object analysis unit 240 analyzes (S722) the type of at least one or more objects (pigs) detected in the pig place image and the number of individuals per pig place based on the learning data, and in the object tracking unit 230 A movement path pattern of at least one object (pig) detected in the pig place image is tracked based on the learning data (S723).

At the same time, the pose change pattern of the object (pig) is analyzed in the image area of at least one object (pig) detected in the pig place image based on the learning data (S723). Here, the pose change pattern is a posture change of at least one of Standing, Lying, and Sitting.

Next, the body area of the object (pig) in the image region of at least one object (pig) detected in the pig place image based on the learning data by the weight change analysis unit 290 and the change in body weight according to the body area is analyzed and determined (S724), and the type of object (pig) in the at least one object (pig) image area detected in the pig place image based on the learning data and the movement path pattern in the heatmap generator 260 ( A heat map for each piglet/sow) is generated (S725).

After that, the statistical processing unit 270 analyzes the pose change pattern of the object (pig) over time based on the type of at least one object (pig), the number of individuals per pig house, the movement path pattern, the pose change pattern, and the heat map. Behavior change statistical data is generated (S726), and the abnormal state prediction unit 280 predicts an abnormal state (disease) of the object (pig) based on the behavior change statistical data and the weight change ( S727 ).

The abnormal state (disease) is African swine fever (ASF), swine reproductive and respiratory syndrome (PRRS), swine influenza (SIV), swine epidemic diarrhea (PED), Augekis disease, Brachispira infection, swine fever (CSF), Critosporidiosis, foot-and-mouth disease (FMD), hepatitis E virus (HEV), Lawsonia intracellularis, mycoplasma pneumonia (M.hyo), parvovirus (PPV), pasteurella disease, rotavirus, salmonellosis , Brachyspira hyodysenterie, swine blisters, infectious gastroenteritis, may be at least one or more of adenomatosis.

For reference, AFS is caused by African swine fever virus (ASFV), which infects domestic pigs, warthogs and wild boars, and this disease is transmitted when a healthy animal comes in direct contact with a diseased animal, or an infected prey. It can be a disease that can be transmitted through indirect contact through, as well as through biological agents (such as soft mites).

Signs of AFS can occur in hyperacute, acute, subacute and chronic forms, and mortality rates vary from 0 to 100%. This depends on the virulence of the virus that infects pigs, and the characteristic that can be identified as an acute disease is a short incubation period of 3 to 7 days, followed by high fever (up to 42°C) and death within 5 to 10 days. The most consistent clinical signs are loss of appetite, depression and recumbent behavior and, among other signs, bleeding from the skin of the ears, abdomen, and legs or shortness of breath, vomiting, bleeding from the nose or rectum, sometimes accompanied by diarrhea . Also, weight loss, joint swelling and respiratory problems. Cases of this type of disease are extremely rare in a group outbreak.

Aujeszky's disease is a contagious viral disease caused by a herpes virus called Pseudorabies virus (PRV). The most common form is acute febrile syndrome, affecting mainly pigs (the main viral pathogen). However, other animal species are also susceptible to the disease. Clinical signs depend on the age and physiological development of the infected animal. For example, in the case of piglets, it causes neurological diseases (swivel movements, seizures) and soon leads to death, and in the case of growing animals, it mainly causes respiratory and gastrointestinal problems, delaying growth.

Swine fever (CSF) is the second most serious disease after aphthous fever among infectious diseases of pigs and wild boars, and poses a serious threat to pig husbandry and has a great socioeconomic impact. This disease is caused by enveloped RNA belonging to the genus Pestivirus of the Flavivirus family. CSF cannot be transmitted to humans, and symptoms appear in various ways depending on the toxicity of the infecting virus and the growth stage of the animal.

Signs are that the superacute form can result in virtually asymptomatic death within 48 hours, but the more common acute form of the disease develops symptoms of high fever (up to 42°C) as the first stage. Recognizable, diseased animals at this stage become lethargic, stop feeding and show symptoms of conjunctivitis with mucus discharge from the eyes, gastrointestinal and respiratory problems, blood imbalances, and neurological disorders.

Using the deep learning-based object tracking and behavior analysis system and method according to an embodiment of the present invention, the type of object (pig) in the pig house, movement pattern, body area/weight change, and pose pattern change through a deep learning algorithm It can be easily determined, and furthermore, there is an advantage that abnormal (disease) conditions can be predicted/prevented at an early stage according to the movement pattern, body area/weight change, and pose pattern change of the object (pig).

7 is a block diagram of a smart livestock management system using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention.

As shown in Figure 7, the smart livestock management system 500 using deep learning-based object tracking and behavior analysis in another embodiment of the present invention is an EEG detection module 501, a first imaging device 510, the first 2 includes an imaging device 520 , a sound collecting device 530 , a monitoring server 540 , and a manager terminal 550 .

The EEG detection module 501 may be attached to each pig object in the pig house, and may be configured to detect the EEG signal of the pig object. The EEG detection module 501 transmits the EEG signal detected from the module ID, location information, and the pig object to the monitoring server 540 to be described later through a unique communication channel.

The EEG detection module 501 operates based on characteristics corresponding to a band pass filter of 0.5 Hz to 100 Hz, a sampling rate of 500 Hz and an impedance of 10 kΩ or less, and silver-silver chloride (Ag-AgCl) patch type. Electrodes can be used to measure EEG signals of pig objects.

The first imaging device 510 captures a motion image of a pig object in the pig house.

The second imaging device 520 generates a thermal image of the pig object captured by the first imaging device 510 .

The sound collecting device 530 tracks the pig object photographed by the first imaging device 510 and the second imaging device 520, and sounds generated from the pig object, for example, breathing and coughing sounds, cries, Acoustic information including urination sound is collected.

The monitoring server 540 analyzes the abnormal behavioral response of the pig object based on the EEG signal, the motion image, the thermal image and the sound information, and predicts the presence or absence of disease and the type of disease according to the abnormal behavior response, If the pig object is predicted to have a disease, a notification message is generated and provided.

More specifically, the monitoring server 540 includes a path tracking unit 541 , a behavior pattern and posture analysis unit 542 , a body temperature analysis unit 543 , an acoustic analysis unit 544 , a disease occurrence determination unit 545 , It includes a notification unit 546 , a behavior pattern learning unit 547 , an EEG signal analysis unit 548 , and an analysis parameter setting unit 549 .

The EEG analysis unit 548 classifies the EEG (electroencephalographic, EEG) related to the sleep state, the abnormal seizure state, and the depression of the pigs through the EEG variability diagram of the EEG signal detected in the pig object, and then compares the EEG variability with the reference. Analyze the pattern.

For reference, the EEG analysis unit 548 inputs the EEG signal of the pig object into a preamplifier (Model 1700, AM Systems Inc., USA) and an amplifier (amplifier: CyberAmp 380, Axon Instruments Inc., USA), After amplification by a factor of 10,000 and filtered with a 0.5 Hz high-pass filter and a 60 Hz low-pass filter, the signals were collected on a data acquisition board at a sampling rate of 200 Hz. : DigiData 1200A, Axon Instruments Inc., USA, or DAQ Pad, National Instruments Inc., USA) were used to classify sleep state, seizure state, and electroencephalographic (EEG) related to depression.

In addition, the EEG analysis unit 548 analyzes the pattern of the EEG variability by using a known EEG spectrum analysis (Lee M-G, et al., Experimental neurology, 2011; 20).

Next, when the target object is input from the manager terminal, the path tracking unit 541 tracks the movement path of the target object in the reverse order based on the position signal of the EEG detection module 501 located in the target object (pig).

Next, the behavior pattern and posture analysis unit 542 analyzes the behavior pattern and posture of the pig object based on a motion image for a change in movement in the movement path of the targeted pig object.

That is, the change of the pose pattern generated in the process of moving from the first point to the second point is classified into any one of Standing, Lying, and Sitting, and when the pig moves, the change of the pose pattern is analyzed as distance versus time.

Next, the body temperature analyzer 543 analyzes the body temperature of the pig object based on the thermal image. In more detail, changes in body temperature for each part of pigs that are heated according to changes in pose patterns are analyzed.

Next, the acoustic analysis unit 544 analyzes at least one of a breathing cycle of the pig object, a size and generation cycle of a cough sound, and a cry sound based on acoustic information generated according to a change in the pose pattern of the targeted pig object. do.

Next, the disease occurrence determination unit 545 first determines the presence or absence of a disease based on the EEG pattern of the pig object, the irregular posture change of the behavior pattern, the breathing cycle, and the cough sound, Based on body temperature and body temperature changes, secondary judgment is made on the presence or absence of disease

More specifically, the disease occurrence determination unit 545 determines the occurrence of disease based on the EEG pattern of the pigs who have learned the deep learning learning algorithm, the irregular change of the time-series pose pattern, the change of the respiratory cycle, and the cough sound. Suspected pig objects are first screened.

Next, the notification unit 546 predicts that the second determined pig object has a disease, and generates and provides a notification message. The notification message may provide the type of expected disease and sample data of pigs having the disease.

Next, the behavior pattern learning unit 547 learns by registering information including the EEG mutation pattern, behavior pattern, posture change, breathing cycle, and cough sound of the pig object predicted to be diseased as learning data. The learned data is used as sample data.

Next, the analysis parameter setting unit 548 interworks with the manager terminal to be described later, and the path tracking unit 541 , the behavior pattern and posture analysis unit 542 , the body temperature analysis unit 543 , and the sound analysis input from the manager terminal. It may be a configuration for setting and changing parameters of each unit as parameters of the unit 544 .

Next, the manager terminal 550 may be configured to receive and display the prediction result value of the monitoring server 540 , and receive a disease outbreak notification message of the pig object from the monitoring server 540 .

8 is a flowchart illustrating a smart livestock management method using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention.

The smart livestock management method (S800) using deep learning-based object tracking and behavior analysis according to another embodiment of the present invention measures the EEG signal of a pig object in a pig house using an EEG detection module (S801), and then tracks the path When the target object (pig) is input from the manager terminal in the unit, the movement path of the target object is tracked in the reverse order based on the position signal of the EEG detection module located in the target object (pig) (S802).

Thereafter, the behavior pattern and posture analysis unit 542 analyzes the behavior pattern and posture of the pig object based on a motion image for a change in movement in the movement path of the targeted pig object ( S810 ). That is, the change in the pose pattern that occurs in the process of moving from the first point to the second point is classified into any one of Standing, Lying, and Sitting, and when the pig moves, the change in the pose pattern is analyzed as distance versus time, and then the disease It is determined whether the behavior is abnormal through comparison with the sample data of the infected pigs (S820).

Thereafter, when it is determined that the pig object is abnormal in the process S820, acoustic information including the breathing and coughing sounds of the pig object is collected (S820). Thereafter, the disease occurrence determination unit first determines the presence or absence of a disease based on the EEG pattern of the target object (pig), the irregular posture change of the behavior pattern, the breathing cycle, and the cough sound (S830). Here, the S830 process is a pig object suspected of having a disease based on the EEG variability pattern of pigs who have learned the deep learning learning algorithm, irregular changes in time-series pose patterns, breathing cycle, and changes in cough sound. It may be a process of selecting tea.

If it is determined that the pig object is disease-prone in step S830, the second imaging device 520 collects thermal image images for each body part of the pig object (S840). In this case, the collected thermal image may be a thermal image for each body part according to a change in a pose pattern. Thereafter, the disease occurrence determination unit predicts the presence or absence of disease and the type of disease in the pig object based on the thermal image (S850). The process S850 may include analyzing the change in body temperature for each part that is heated according to the change in the pose pattern, and then comparing the analyzed result value with the sample data of the pig object suffering from the disease.

Thereafter, when it is predicted that the corresponding pig object has a disease in the process S850, the notification unit generates a disease occurrence notification message and sends it to the manager terminal (S860).

Therefore, in the deep learning-based object tracking and behavior analysis method according to another embodiment of the present invention, the behavior of the pig object based on the result of analyzing the EEG variability pattern and pose pattern change of the targeted pig object as distance versus time Detects abnormalities and, in the case of a pig object detected as a behavioral abnormality, predicts the occurrence of a primary disease by changing the EEG pattern of the pig object and breathing and coughing sounds according to the change in the pose pattern, and then, according to the change in the pose pattern By predicting the occurrence of a secondary disease based on the body temperature change by body quartile, there is an advantage in that it is possible to predict whether or not a disease occurs in pigs by volume.

11 illustrates an example computing environment in which one or more embodiments disclosed herein may be implemented, and is an illustration of a system 1000 including a computing device 1100 configured to implement one or more embodiments described above. shows

For example, computing device 1100 may be a personal computer, server computer, handheld or laptop device, mobile device (mobile phone, PDA, media player, etc.), multiprocessor system, consumer electronics, minicomputer, mainframe computer, distributed computing environments including any of the aforementioned systems or devices, and the like. The computing device 1100 may include at least one processing unit 1110 and a memory 1120 . Here, the processing unit 1110 may include, for example, a central processing unit (CPU), a graphic processing unit (GPU), a microprocessor, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Arrays (FPGA), etc. and may have a plurality of cores. The memory 1120 may be a volatile memory (eg, RAM, etc.), a non-volatile memory (eg, ROM, flash memory, etc.), or a combination thereof.

Additionally, computing device 1100 may include additional storage 1130 . Storage 1130 includes, but is not limited to, magnetic storage, optical storage, and the like. The storage 1130 may store computer readable instructions for implementing one or more embodiments disclosed herein, and other computer readable instructions for implementing an operating system, an application program, and the like. Computer readable instructions stored in storage 1130 may be loaded into memory 1120 for execution by processing unit 1110 .

Computing device 1100 may also include input device(s) 1140 and output device(s) 1150 . Here, the input device(s) 1140 may include, for example, a keyboard, mouse, pen, voice input device, touch input device, infrared camera, video input device, or any other input device, or the like. Further, the output device(s) 1150 may include, for example, one or more displays, speakers, printers, or any other output device, or the like. Also, the computing device 1100 may use an input device or an output device included in another computing device as the input device(s) 1140 or the output device(s) 1150 .

Computing device 1100 may also include communication connection(s) 1160 that enable computing device 1100 to communicate with another device (eg, computing device 1300 ). Here, communication connection(s) 1160 may be a modem, network interface card (NIC), integrated network interface, radio frequency transmitter/receiver, infrared port, USB connection, or other for connecting computing device 1100 to another computing device. It may include interfaces. Also, the communication connection(s) 1160 may include a wired connection or a wireless connection.

Each component of the aforementioned computing device 1100 may be connected by various interconnections such as a bus (eg, peripheral component interconnection (PCI), USB, firmware (IEEE 1394), optical bus structure, etc.) and may be interconnected by a network 1200 .

As used herein, terms such as “component,” “part,” and the like generally refer to a computer-related entity that is hardware, a combination of hardware and software, software, or software in execution.

For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. For example, both an application running on a controller and a controller may be a component. One or more components may reside within a process and/or thread of execution, and components may be localized on one computer or distributed between two or more computers.

The present invention is not limited by the above-described embodiments and the accompanying drawings. It is intended to limit the scope of rights by the appended claims, and it is to those of ordinary skill in the art that various types of substitutions, modifications and changes can be made without departing from the technical spirit of the present invention described in the claims. it will be self-evident

first embodiment
10: Smart livestock management system using deep learning-based object tracking and behavior analysis.
100: image acquisition unit
200: artificial intelligence server
210: object recognition unit
220: deep learning learning unit
230: object tracking unit
240: object entity analysis unit
250: object pose pattern change analysis unit
260: Heatmap generation unit
270: statistics processing unit
280: abnormal state prediction unit
290: weight change analysis unit
second embodiment
500: Smart livestock management system using deep learning-based object tracking and behavior analysis.
501: EEG detection module
510: first imaging device
520: second imaging device
530: sound collection device
540: monitoring server
541: route tracking unit
542: behavior pattern and posture analysis unit
543: body temperature analysis unit
544: acoustic analysis unit
545: disease occurrence determination unit
546: notification unit
547: behavior pattern learning unit
548: analysis parameter setting unit
549: EEG signal analysis unit
550: manager terminal

Claims (2)

an EEG detection module attached to the head of a pig object in a pig house to measure and transmit an EEG signal of the pig object;
a first imaging device for photographing a motion image of a pig object in the pig house;
a second imaging device generating a thermal image of the pig object;
a sound collecting device for collecting sound information including breathing and coughing sounds of the pig object;
a monitoring server for predicting and judging the presence or absence of disease and the type of disease in the pig object based on the EEG signal, the motion image, the thermal image, and the sound information; and
and a manager terminal receiving a disease outbreak notification message of a pig object from the monitoring server,
The monitoring server
After classifying the EEG (electroencephalographic, EEG) related to the sleep state, abnormal seizure state, and depression of the pig object through the EEG variability diagram of the EEG signal detected in the pig object, EEG analysis to analyze the pattern of EEG variability in comparison with the reference part;
When a target object (pig) is input from the manager terminal, a path tracking unit for tracking the movement path of the target object in reverse order based on the position signal of the EEG detection module located in the target object (pig);
a behavior pattern and posture analysis unit that analyzes the behavior pattern and posture of the pig object based on a motion image for a change in movement of the targeted pig object;
a body temperature analyzer analyzing the body temperature of the pig object based on the thermal image;
an acoustic analysis unit for analyzing at least one of a breathing cycle of the pig object, a size and generation cycle of a cough sound, and a cry sound based on the acoustic information; and
The presence or absence of disease is primarily determined based on the EEG variability pattern of the pig object, irregular posture changes in behavioral patterns, breathing cycle, and cough sound, and disease occurrence is determined based on the body temperature and body temperature change of each part of the pig object selected first Including a disease occurrence determination unit for secondary judgment,
The behavioral posture and pattern analysis unit
Based on the motion image, the change of the pose pattern of the pig in the pig house is classified into any one of Standing, Lying, and Sitting, and when the pig moves, the change of the pose pattern is analyzed as a distance versus time,
The body temperature analysis unit analyzes changes in body temperature for each part of pigs that are heated according to the change in the pose pattern,
The disease occurrence determination unit
Deep learning-based object tracking and behavior analysis, characterized in that the pig object is first selected based on the change in the pose pattern, breathing cycle, and cough sound of diseased pigs who have learned the deep learning learning algorithm Smart livestock management system using.
Measuring the EEG signal of the pig object in the pig house using the EEG detection module;
When the target object (pig) is input from the manager terminal in the path tracking unit, tracking the movement path of the target object in the reverse order based on the position signal of the EEG detection module located in the target object (pig);
The EEG analysis unit classifies the EEG (electroencephalographic, EEG) related to the sleep state, abnormal seizure state, and depression of pigs through the EEG variability diagram of the EEG signal detected in the pig object, and then compares the EEG pattern with the reference to analyze the EEG pattern. step;
When the pig object moves based on the motion image for the change in the movement of the targeted pig object in the behavior pattern and posture analysis unit, the pose pattern change is analyzed as distance versus time, and then sample data of diseased pigs judging whether the behavior is abnormal by comparing with
When the corresponding pig object is determined to be abnormal, collecting acoustic information including the breathing and coughing sounds of the pig object in the acoustic information collection unit;
first determining the presence or absence of a disease in the vaginal occurrence determination unit based on the EEG pattern of the target object (pig), the irregular posture change of the behavior pattern, the respiratory cycle, and the cough sound;
When it is determined that the target object (pig) is disease-prone, collecting thermal image images for each body part of the pig object in a second imaging device
predicting the presence or absence of disease and the type of disease in the pig object based on the thermal image in the disease occurrence determination unit; and
Smart livestock management method using deep learning-based object tracking and behavior analysis, comprising the step of generating a disease outbreak notification message from the notification unit and sending it to a manager terminal when the pig object is predicted to have a disease.

Images