CN114358163A - A method and system for monitoring feed intake based on twin network and depth data - Google Patents

Abstract

The invention discloses a twin network and depth data-based feed intake monitoring method and system, which comprises the following steps: acquiring images before eating and images after eating of the cows for a plurality of times; inputting the before-eating image and the after-eating image into a twin network, respectively mapping the twin network and the after-eating image to the same vector space through a feature extraction network to obtain a before-eating multi-dimensional feature vector and a after-eating multi-dimensional feature vector, and tiling the two multi-dimensional feature vectors; making a difference between the two tiled multidimensional feature vectors to obtain a new feature vector; and calculating the new characteristic vector through one-time full connection to obtain the feed intake. According to the method, the feed intake of the dairy cows in a single time can be predicted without preprocessing the feed pile images before and after ingestion, the influence of illumination is small, the difference of the prediction performance under different illumination conditions is small, and the stability and the accuracy are improved. In addition, the method can be directly combined with other methods based on computer vision to realize the complete non-contact monitoring of the single feed intake of the individual cows.

Description

A method and system for monitoring feed intake based on twin network and depth data

technical field

The invention belongs to the field of agriculture and livestock breeding systems, and in particular relates to a method and system for monitoring feed intake based on twin network and depth data.

Background technique

Feed intake is one of the main factors affecting the growth and development and lactation performance of dairy cows, and it is also an important indicator to reflect the individual health status of dairy cows. Therefore, feed intake monitoring is of great significance for the refined feeding of dairy cows.

At present, the main methods for monitoring individual feed intake of dairy cows include a method based on the combination of RFID and a feeding trough with a weighing sensor, a feed intake estimation method based on wearable devices, and a near-infrared spectroscopy analysis method. The method based on RFID combined with the feeding trough with load cell has high precision and accuracy, but it requires high cost, frequent cleaning and maintenance, and is less used in pastures; wearable device-based The feed intake estimation method collects the feeding behavior parameters of dairy cows through devices such as smart collars, foot rings and bridles to construct a feed intake assessment prediction model. Near-infrared spectroscopy mainly uses equipment such as near-infrared spectrometers to analyze cow dung to determine feed components, digestibility, etc., and then estimate cow feed intake through relevant calculations. Although this method is stress-free and relatively expensive. Low, but it is difficult to obtain cow manure, it is difficult to be accurate to individual cows, and the estimation accuracy is low. Therefore, for the monitoring of individual feed intake of dairy cows, more research is needed to obtain a feed intake monitoring method with lower cost, higher accuracy and more practicability.

In recent years, with the continuous development of optical imaging technology, computer vision is also used to monitor the feed intake of dairy cows, which does not require expensive equipment and avoids the stress response caused by wearing. For example, a 3D camera is used to measure the feed volume, and the relationship between the volume and the feed weight is obtained through linear and quadratic least squares t-test regression analysis. The error of the system within 22.68kg is 0.5kg. This technology proves that the computer vision method It is feasible, but its simulation accuracy needs to be improved; and using multiple high-pixel RGB cameras to take multiple photos from various angles to carry out 3D reconstruction of the monitored feed pile in a specific area, through the shape and volume of the feed pile Changes are used to predict feed intake, and the simulation accuracy is high. Under laboratory conditions, the error in the calculated mass is 0.483kg for feed piles under 7kg, and the estimation error under cowshed conditions is also less than 0.5kg, but the main limitation of this method is that in real scenarios, the During the feeding process, it is difficult for the feed piles to be concentrated in the marked area, and the marks used to determine the feed range are also easily contaminated; A new tensor is generated to train a convolutional neural network model to monitor the feed intake of individual dairy cows. The results show that the absolute average error of the system model for the prediction of feed intake is 0.127kg, and the mean square error is 0.034kg2, but Due to the influence of factors such as lighting, even if it is the same object and the same color, the RGB values are different in the two photos at different times. It is difficult to obtain meaningful results by subtracting the three RGB channels. On the one hand, two color images may lose some effective information in the subtraction, and a large amount of data needs to be preprocessed before training and using the model. These technologies demonstrate the potential of using computer vision technology to measure feed intake, feed volume and feed weight, but there are still some problems with the existing technologies, mainly including the need to improve the accuracy, cumbersome data processing process, and difficult to work stably in complex environments And other issues.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a feed intake monitoring method and system based on twin network and depth data, in order to solve the problems of high cost, insufficient precision, and cumbersome data processing process existing in traditional feed intake monitoring methods.

On the one hand, the present invention provides a method for monitoring feed intake based on twin network and depth data, including:

Collect pre-feeding images and post-feeding images of several feeding cows;

Input the image before eating and the image after eating into the twin network, map them to the same vector space through the feature extraction network respectively, obtain the multidimensional feature vector of the image before eating and the multidimensional feature vector of the image after eating, and equalize the two multidimensional feature vectors. shop;

Making a difference between the tiled multidimensional feature vector before eating and the tiled multidimensional feature vector after eating to obtain a new feature vector;

The new feature vector is calculated through a full connection to obtain the feed intake.

Optionally, the process of collecting pre-feeding images and post-feeding images of several feeding times of the cow includes:

The pre-eating image and the post-eating image under different lighting conditions are collected by a depth camera, the image types of the pre-eating image and the post-eating image are depth images, and the different lighting conditions include low light, strong light , indoor light, indoor low light and no light.

Optionally, the process of collecting pre-feeding images and post-feeding images of several feeding times of the dairy cow also includes:

After the ear tag of the cow is sensed through the RFID sensor at the beginning of feeding, the cow ID and feed weight before eating are collected and recorded;

After the feeding is completed, when the post-feeding image is collected, the weight of the post-feeding feed is simultaneously collected and the feeding time is obtained.

Optionally, before inputting the image before eating and the image after eating into the twin network, the method further includes:

Data enhancement processing is performed on the image, and the data enhancement processing includes vertical flipping, horizontal flipping, and vertical-horizontal flipping.

Optionally, in the twin network, in the process of mapping the image before eating and the image after eating to the same vector space by feature extraction network, the feature extraction network adopts the residual network ResNet101 structure, The residual network uses skip connections.

Optionally, the process of feature extraction through the residual network ResNet101 structure includes:

First perform convolution processing through the first convolution layer, and then calculate the residual through the following four residual layers; each of the residual layers includes several residual blocks, and each of the residual blocks includes three residual blocks A convolution layer, and the convolution kernel sizes of the three convolution layers are 1x1, 3x3, and 1x1, respectively.

On the other hand, the present invention provides a feed intake monitoring system based on twin network and depth data, including:

The acquisition module is used to collect the depth data of several feeding times of dairy cows;

a database module for storing the depth data;

The processing module is used for inputting the depth data collected by the collection module into the twin network for processing to obtain the feed intake.

Optionally, the acquisition module includes a depth camera, an RFID sensor and a weight sensor;

The depth camera is used to collect images before eating and after eating under different lighting conditions, and the image types of the images before eating and after eating are depth images, and the different lighting conditions include low light, strong light, Indoor light, indoor low light and no light;

The RFID sensor is used to collect cow ID and feeding time data after sensing the appearance of the ear tag;

The weight sensor is used to collect feed weights before and after eating, respectively.

Optionally, the processing module includes a feature extraction module and a feed intake calculation module;

The feature extraction module is used to perform two-way feature extraction on the depth data to obtain two-way features;

The feed intake calculation module is used to make a difference between the two-way features, and obtain the feed intake through full connection calculation.

Optionally, the feature extraction module adopts a two-way residual network ResNet101 structure, the residual network adopts a skip connection, and the residual network ResNet101 structure includes one convolutional layer and four residual layers, each of the residual The difference layer is composed of several residual blocks, and each residual block includes 3 convolution layers, and the size of the convolution kernel is 1x1, 3x3 and 1x1 respectively.

The technical effect of the present invention is:

The present invention proposes a method for predicting feed intake of dairy cows based on depth data and twin network. The depth images of two feed piles before and after feeding by dairy cows are mapped to the same vector space through a feature extraction network shared by two weights and then subtracted. , and the obtained feature vector is sent to the feed intake calculation layer for calculation, so as to realize the prediction of the single feed intake of dairy cows. The method can realize the prediction of single feed intake of dairy cows without preprocessing the images of feed piles before and after feeding, and is less affected by illumination, and the prediction performance has little difference under different illumination conditions, which is more efficient than the prior art. Stability and accuracy. In addition, this method can be directly combined with other computer vision-based methods to achieve a completely non-contact monitoring of individual cows' LI.

Description of drawings

The accompanying drawings constituting a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a schematic structural diagram of a feed intake monitoring model in Embodiment 1 of the present invention;

Fig. 2 is the data acquisition flow chart in the first embodiment of the present invention;

FIG. 3 is a schematic structural diagram of a feature extraction network in Embodiment 1 of the present invention.

Detailed ways

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

Example 1

As shown in Figure 1, the present embodiment provides a method for monitoring feed intake of dairy cows based on twin network and deep data processing, including:

1) Input the two images before and after feeding into the twin network, and map them to the same vector space through the feature extraction network respectively;

2) tile the obtained two multi-dimensional feature vectors, find their differences, and generate new feature vectors;

3) Calculate the feed intake from the new feature vector through a full connection.

Data collection process:

The invention mainly consists of a depth camera, a feeding trough, a weighing sensor, a data transmission control terminal, an RFID radio frequency identification sensor, a computer and a set of software systems developed by C++.

The image data was collected with ORBBEC Astra Mini depth camera, its working distance is 0.4-2m, the field of view is H58.4°, V is 45.5°, the working temperature is 10°C-40°C, the accuracy per meter is 3mm, the depth processing chip MX400, the camera is placed Above the feed trough, the distance from the camera to the ground is 97cm; the image capture specification is 480*640 pixels.

The experimental data were obtained by simulation experiments in the Precision Feeding Technology and Equipment Laboratory of Northeast Agricultural University, artificially simulating the feeding scene of dairy cows in a semi-open cowshed environment, collecting different lighting conditions (low light, strong light, indoor light, indoor Images of feed piles before and after feeding in low light and no light), using the feed as a fully mixed TMR diet. As shown in Figure 2, the data collection process is: after the RFID sensor senses the ear tag, the system is activated, the depth camera captures the feed image in the feed trough (before simulating feeding), and the RFID sensor starts to collect and record the cow ID. , feeding time and other data, the weight sensor records the weight of the feed pile before eating, stores it in the local database, simulates that the ear tag leaves the feeding area after the feeding is finished, the weighing sensor collects the weight of the feed pile after eating, calculates the feed intake and calculates Store it in the database, and at the same time, the depth camera shoots the RGB and depth images in the feed trough again (after simulating feeding) and transmits them to the computer for saving.

Considering the influence of light conditions on the image acquisition of the depth camera, a total of five different light levels between strong light and no light were collected for the feed pile images. During the experiment, a total of 483 groups of depth images and RGB images of feed piles of 0-31 kg were collected. The model established by the present invention uses depth data, and RGB data is used in the comparison model.

Training the model requires a large amount of sample data, and the data diversity also determines the accuracy and generalization ability of the model. Therefore, the present invention enhances the test data. After data enhancement, it includes: vertical flip, horizontal flip, vertical and horizontal flip, a total of 1932 sets of depth and color images are obtained.

Create a sample set based on the collected data:

In order to determine the feeding amount of dairy cows through images, it is necessary to determine the image difference between the feed piles before and after eating, and the twin network needs to take two depth images as input, so it is necessary to combine the depth images obtained in the experiment in pairs to generate a new data set . In this experiment, the sample data were combined in pairs to generate a total of 24150 groups of combined data with weight difference between [0,8200]g, and the combined data was used as input data, and the weight difference between the two images of feed piles in the combination was calculated. The value (that is, the feed intake) is used as a label to establish a data set, which is divided into a training set and a test set according to 8:2.

The feed intake monitoring model based on the twin network proposed by the present invention is mainly composed of two parts: a feature extraction module and a feed intake calculation module. Each branch adopts the same network structure and shares weights, which not only reduces the amount of parameters of the model , which also ensures the consistency of the mapping space. The specific calculation process is as follows:

1) Input the two images before and after feeding into the twin network, and map them to the same vector space through the feature extraction network respectively;

2) tile the obtained two multi-dimensional feature vectors, find their differences, and generate new feature vectors;

3) Calculate the feed intake from the new feature vector through a full connection.

The feature extraction network uses the ResNet101 structure, and the residual network uses skip connections, that is, when the input of the neural network is x and the function map to be fitted (ie, the output) is H(x), when x and F(x) When the dimensions are the same, the original function map H(x) is calculated as:

H(x)=F(x)+x (1)

When the dimensions of x and F(x) are different, the calculation method of the original function map H(x) is:

H(x)=F(x)+W(x) (2)

Where W(x) represents the convolution operation, which is used to adjust the dimension of x.

The introduction of the residual network improves the correlation between input and output, thereby ensuring good convergence in the deep network, and can effectively avoid the problem of gradient disappearance or gradient explosion. The internal structure of the network is shown in Figure 3. The structure is mainly composed of a single It consists of convolution layer and 4 residual layers, each residual layer is composed of several residual blocks, each residual block includes 3 convolution layers, and the size of convolution kernel is 1x1, 3x3, 1x1 respectively.

The loss function selects the MSE-LOSS function, and the calculation method of the loss value I _loss is shown in formula (3):

为预测值，y_i为实际值，n为训练集样本数量。in

is the predicted value, y _i is the actual value, and n is the number of samples in the training set.

Embodiment 2

In the process of training the model, the present invention uses the stochastic gradient descent method, sets the batch-size size to 32, and the weight decay to 0.1. Whenever the loss value of the model on the validation set decreases, the model is saved, and finally the lowest loss is selected. 's model. The iterative termination condition of algorithm training is that the training reaches 500 epochs. Since the twin network is not easy to converge during training, the present invention first trains the feature extraction network during training, and then fixes its weight to train the feed intake calculation layer.

The model evaluation method uses mean absolute error (MAE) and root mean square relative error (RMSE) to measure the model evaluation performance. The formula is as follows:

为采食量实测值，

为预测值，

为平均值。in

is the measured value of feed intake,

is the predicted value,

is the average value.

The present invention takes 0.05, 0.1, and 0.5 as the learning rates respectively. Among them, the loss value decreases slowly as the iteration batch increases. When the learning rate is large, the loss value decreases faster, but it is not easy to converge. In contrast, training with a small learning rate can make the loss value drops even lower, so the present invention sets the algorithm learning rate to 0.05.

In order to determine the appropriate network depth, the present invention compares the recognition performance of the Siamese network model under four feature extraction network layers. , when the network depth is increased from 50 layers to 101 layers, the MAE and RMSE are only reduced by 0.14% and 0.1% respectively. It can be seen that further deepening the network has little effect on the model performance improvement. Therefore, the present invention sets the feature extraction network layer The number is 101.

Table 1

Among them, the number of network layers here refers to the number of convolutional layers + fully connected layers, not the number of all layers in the network

The present invention tries three ways to find the difference of the extracted feature vectors. The first way is to directly merge two 512-dimensional vectors into a 1024-dimensional vector, the second way is to subtract the two vectors, and the third way To divide two vectors. The comparison results are shown in Table 2. It can be seen that the subtraction effect is the best.

Table 2

In order to determine the influence of lighting conditions on the performance of the model, the models were tested under five conditions: weak light, strong light, indoor light, indoor weak light, and no light. The results are shown in Table 2. Small, the prediction performance is not much different under the five lights, the RMSE is the smallest under no-light conditions, and the MAE is the smallest under strong light.

table 3

In order to determine the prediction performance difference between the feed intake prediction model based on the twin network proposed by the present invention and other models, when the feature extraction network structure is ResNet101, other two feed intake calculation models are trained by other methods. A model calculates the weight of a single feed pile image before and after feeding, and then subtracts it to obtain the current feed intake (hereinafter referred to as the feed intake prediction model based on weight subtraction, WSNet). The second method subtracts the negative values of the feed pile images before and after feeding on the depth channel to obtain a new tensor. This tensor is used as the input variable, and the feed intake is used as the output variable to train a residual network. model to calculate feed intake (hereinafter referred to as the feed intake prediction model based on image subtraction, ISNet, it is worth mentioning that the prediction accuracy of this method is higher than that of the model using RGB-D four-channel subtraction data as training data, which proves that Subtraction of RGB data does not play an active role in model calculation). The comparative analysis of the prediction performance of the three models is shown in Table 4.

Table 4

Through the comparative analysis of MAE and RMSE, it is found that the single feed intake prediction model based on weight subtraction has the largest error and the worst stability, the single feed intake prediction model based on image subtraction is second, and the twin network model is the best. In the 4860 sets of test data, the maximum error of the single feed intake prediction model based on weight subtraction is 1359.23g, and the maximum error of the twin network model is 507.46g; the MAE of the twin network is compared with the ResNet model based on weight subtraction and the image subtraction based model. The Resnet model drops by 49.4% and 7.5%, and the RMSE drops by 51.9% and 4.2%, respectively. It can be seen that the twin network model has higher accuracy and stronger stability than the other two models, and can better calculate the feed intake of dairy cows. This is because in the twin network model, the input image can be filtered after feature extraction. Some errors and redundant information in the original data are removed, and the effective information is extracted and then the difference is calculated, which makes the information used in the calculation of feed intake more accurate and improves the prediction performance of the model.

The present invention designs a set of data acquisition method and system, which takes the collected 24150 groups of feed pile depth images before and after feeding as the data source, optimizes the training of the constructed feed intake monitoring model based on the twin network, and determines the learning of the network algorithm. The ratio is 0.05, the number of layers of the feature extraction network is set to 101, and the model can achieve better performance under the method of subtracting the difference of the feature vector. In the range of 0-8200g, the mean absolute error MAE for predicting the feed intake of dairy cows is 100.6g, and the root mean square error RMSE is 128.02g, which is superior to the prior art. The effectiveness of the feed intake monitoring model for image feature extraction, high-dimensional image spatial difference calculation and feed intake quantification before and after feeding was illustrated.

Under different lighting conditions, the prediction performance of the model is not much different. The RMSE is the smallest under no-illumination conditions, and the MAE is the smallest under strong light. It can be seen that using the depth image as the data source can effectively avoid the influence of illumination on the model simulation accuracy. Compared with The existing technology is more stable and accurate, and at the same time verifies the feasibility of using a depth camera to monitor feed intake in a semi-open cattle farm.

The constructed monitoring model based on twin network and in-depth data can more accurately reflect the changes of dairy cattle feed intake. Combining this method with other computer vision methods can realize completely non-contact individual dairy cattle feed intake monitoring. In future applications, the fusion of depth data and RGB data should be considered to provide more effective information, and to further realize the identification and classification of cow feeding behavior, activity area and other information.

The above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited to this. Substitutions should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (10)

1. A food intake monitoring method based on a twin network and depth data is characterized by comprising the following steps:

acquiring images before eating and images after eating of the cows for a plurality of times;

inputting the before-eating image and the after-eating image into a twin network, respectively mapping the twin network to the same vector space through a feature extraction network to obtain a before-eating image multi-dimensional feature vector and a after-eating image multi-dimensional feature vector, and tiling the two multi-dimensional feature vectors;

making a difference between the tiled before-eating multi-dimensional feature vector and the tiled after-eating multi-dimensional feature vector to obtain a new feature vector;

and calculating the new characteristic vector through one-time full connection to obtain the feed intake.

2. The method of claim 1, wherein the capturing of the pre-feeding image and the post-feeding image of the cow's food taken several times comprises:

the method comprises the steps of acquiring images before food intake and images after food intake under different illumination conditions through a depth camera, wherein the images before food intake and the images after food intake are depth images, and the different illumination conditions comprise weak light, strong light, indoor weak light and no illumination.

3. The method of claim 1, wherein the process of acquiring the before-feeding image and the after-feeding image of the cow feeding for a plurality of times further comprises:

when the ingestion starts, after the ear tag of the cow is induced by the RFID inductor, the ID of the cow and the weight of the feed before eating are collected and recorded;

and after the ingestion is finished, when the after-ingestion image is acquired, the weight of the after-ingestion feed is synchronously acquired and the ingestion duration is obtained.

4. The method of claim 3, wherein before inputting the pre-prandial image and the post-prandial image into the twin network, the method further comprises:

and carrying out data enhancement processing on the image, wherein the data enhancement processing comprises vertical turning, horizontal turning and vertical and horizontal turning.

5. The method according to claim 1, wherein in the twin network, in the process of respectively mapping the before-eating image and the after-eating image to the same vector space through a feature extraction network, the feature extraction network adopts a residual error network ResNet101 structure, and the residual error network adopts jump connection.

6. The method of claim 5, wherein the performing feature extraction through the residual network ResNet101 structure comprises:

performing convolution processing on the first layer of convolution layer, and calculating residual errors through the four subsequent residual error layers; each residual layer comprises a plurality of residual blocks, each residual block comprises three convolutional layers, and the sizes of convolution kernels of the three convolutional layers are 1x1, 3x3 and 1x1 respectively.

7. A food consumption monitoring system based on twin network and depth data is characterized by comprising:

the acquisition module is used for acquiring depth data of the cows after eating for a plurality of times;

the database module is used for storing the depth data;

and the processing module is used for inputting the depth data acquired by the acquisition module into the twin network for processing to obtain the feed intake.

8. The system of claim 7, wherein the acquisition module comprises a depth camera, an RFID sensor, and a weight sensor;

the depth camera is used for acquiring images before food intake and images after food intake under different illumination conditions, wherein the image types of the images before food intake and the images after food intake are depth images, and the different illumination conditions comprise weak light, strong light, indoor weak light and no illumination;

the RFID sensor is used for sensing the appearance of the ear tag and then acquiring the ID and ingestion duration data of the dairy cow;

the weight sensor is used for respectively collecting the weight of the feed before and after eating.

9. The system of claim 7, wherein the processing module comprises a feature extraction module and a feed intake calculation module;

the feature extraction module is used for performing two-path feature extraction on the depth data to obtain two paths of features;

and the feed intake calculation module is used for making a difference between the two paths of characteristics and obtaining the feed intake through full-connection calculation.

10. The system according to claim 9, wherein the feature extraction module employs a two-way residual network ResNet101 structure, the residual network employs a skip connection, the residual network ResNet101 structure includes one convolutional layer and four residual layers, each residual layer is composed of a number of residual blocks, each residual block includes 3 convolutional layers, and the convolutional core sizes are 1x1, 3x3 and 1x1, respectively.

Images