JP7260922B2 - Learning data generation device, learning device, behavior analysis device, behavior analysis device, program, and recording medium - Google Patents

Description

The present invention relates to a learning data generation device, a learning device, a behavior analysis device, a behavior analysis device, a program, and a recording medium.

In order to develop a labor-saving herd management technology that corresponds to the scale of dairy farming, there is a need for a method that can automatically monitor the behavior of each individual in the herd. As a technology for monitoring individual cows in a herd, a technology for monitoring by attaching a wearable sensor such as an accelerometer to each individual has been put into practical use. As an example of such technology, a collar-type sensor is attached to the cow to grasp the amount of activity of the cow, and the time of activity, rumination and rest is calculated. Techniques for detecting slaughtered cattle are known.

However, with such a technique, the larger the cattle herd, the greater the introduction cost and the greater the labor involved in managing the equipment. The cost of introducing and operating such technology is, for example, several tens of thousands of yen per collar-type sensor for grasping the amount of activity of a cow, and several thousand yen per year for usage fees.

In contrast, technology that monitors cows based on video analysis has the advantage of reducing costs and facilitating management because there is no need to attach sensors to cows. Techniques described in Non-Patent Documents 1 and 2 are examples of techniques for monitoring cattle based on image analysis. Non-Patent Document 1 discloses that hoof disease can be detected by analyzing a cow's gait from a depth camera image of the cow. In addition, Non-Patent Document 2 describes detecting a change in posture as a sign of parturition from a photographed image of a cow.

In addition, examples of techniques for identifying and tracking individuals targeting humans include techniques described in Patent Literatures 1 and 2. Patent Literature 1 describes a configuration that detects a person moving within a specific area, acquires a still image of the person, determines whether the person is a resident or a non-resident, and follows the behavior in chronological order. Further, Patent Literature 2 describes a configuration for detecting a moving object in an image and extracting a feature amount for identifying a person.

JP 2017-224249 JP 2019-169843

Sho Sunagawa et al., Information Processing Society of Japan Research Report, Vol. 2017-CVIM-206, No.2(2017) Yusuke Okimoto et al., Proceedings of the JSAI Annual Conference, 32nd Annual Conference (2018)

However, the techniques described in Non-Patent Documents 1 and 2 assume the monitoring of a single individual, and therefore are not suitable for analyzing the behavior of a plurality of individual cows.

On the other hand, the techniques described in Patent Documents 1 and 2 are for monitoring a plurality of individuals, and it is described that a machine-learned model is used for individual identification, but learning data for machine learning No mention is made of the collection of In order to develop an individual identification technology that can be used universally for various cattle herds with different individual composition and imaging conditions, an individual identification system using a trained model that can be easily constructed for each herd is required. be done.

One aspect of the present invention was made to solve the above-described problems, and the purpose thereof is to realize a highly versatile video-based behavior analysis system that can be applied to various populations. .

In order to solve the above problems, the present invention includes the following aspects.
1) An image acquisition unit that acquires an image including an individual, a moving object detection unit that detects a moving object in the acquired image, and a body region including the entire body of the individual in the detected moving object is framed. an extraction unit for extracting a body region image of the body region to which the frame is attached; and a generation unit for generating training data for a discrimination model that identifies an individual from the extracted body region image It has
2) The frame enclosing unit inputs image data of the detected moving object as input data to a discriminant model subjected to machine learning so as to extract a body region including the entire body of the individual from an image including the individual, and outputs the result. The learning data generation device according to 1), which gives a frame to the body region to be processed.
3) The frame enclosing unit inputs the image data of the detected moving object as input data to a generative model that performs machine learning so as to generate an image representing the entire body of the individual from an image representing a part of the body of the individual. The learning data generating device according to 1) or 2), wherein the image representing the entire body of the individual output by the above is set as a body region, and the body region is surrounded by a frame.
4) The learning data generating device according to any one of 1) to 3), wherein the individual is a livestock individual raised in multiple animals.
5) The learning data generation device according to any one of 1) to 4), wherein the generation unit classifies and accumulates the body region images for each individual to generate the learning data.
6) A program for causing a computer to function as the learning data generation device according to any one of 1) to 5), comprising: an image acquisition step of acquiring an image containing an individual; a moving body detection step for detecting; a frame surrounding step for giving a frame surrounding a body region including the entire body of the individual among the detected moving bodies; and extracting a body region image of the body region given the frame surrounding. A program characterized by executing an extraction step and a generation step of generating training data for a discrimination model for identifying an individual from the extracted body region image.
7) using an image acquisition unit that acquires an image containing an individual and learning data generated by the learning data generation device according to any one of 1) to 5) to identify the individual contained in the image; an identification unit that outputs individual identification information of an individual contained in the image by inputting the data of the acquired image as input data into an identification model that identifies an individual, which is constructed by machine learning; 1. A behavior analysis apparatus, comprising: an identification image acquisition unit that acquires time-series images of an individual that has been identified; and an analysis unit that analyzes the behavior of the individual by referring to the acquired images.
8) A program for causing a computer to function as the behavior analysis device according to 7), comprising an image acquisition step of acquiring an image including an individual, and the learning data generation device according to any one of 1) to 5). Inputting the obtained image data as input data to an identification model for identifying an individual, which is constructed by machine learning so as to identify the individual contained in the image using the learning data generated by an identification step of outputting individual identification information of an individual contained in the image; an identification image obtaining step of obtaining time-series images of the identified individual; and referring to the obtained image to determine the behavior of the individual. A program characterized by performing an analysis step of analyzing.
9) a learning data acquisition unit that acquires learning data generated by the learning data generation device according to any one of 1) to 5); and an individual included in an image using the acquired learning data. and a learning unit that builds a discrimination model that discriminates an individual by performing machine learning so as to discriminate.
10) A program for causing a computer to function as the learning device according to 9), which acquires learning data generated by the learning data generation device according to any one of 1) to 5). and a learning step of building an identification model for identifying individuals by performing machine learning so as to identify individuals contained in images using the acquired learning data. program.
11) An image acquisition unit that acquires an image including an individual, and by inputting the data of the acquired image into the discrimination model constructed by the learning device according to 9) as input data, the image included in the image and an identification unit that outputs individual identification information of an individual.
12) A program for causing a computer to function as the individual identification device according to 11), comprising: an image acquisition step of acquiring an image containing an individual; and an identification step of outputting individual identification information of an individual contained in the image by inputting data of the obtained image as input data.
A program for causing a computer to function as each unit included in the individual identification device.
13) A recording medium recording the program according to 6), 8), 10) or 12).
14) An image acquisition step of acquiring an image including the individual, a moving object detection step of detecting a moving object in the acquired image, and a body region including the entire body of the individual in the detected moving object is framed. an extraction step of extracting a body region image of the body region to which the frame is attached; and a generation step of generating learning data for a discrimination model that identifies an individual from the extracted body region image A method for generating data for learning, including
15) An image acquisition step of acquiring an image containing an individual, and the learning data generated by the learning data generation method according to 14) are used to perform machine learning so as to identify the individual contained in the image. an identification step of outputting individual identification information of the individual contained in the image by inputting the acquired image data as input data to the identification model for identifying the individual, and a time series of the identified individual. A behavior analysis method, comprising: an identification image acquisition step of acquiring an image; and an analysis step of analyzing the behavior of the individual with reference to the acquired image.
16) The learning data generation device includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the acquired image, and a body region that includes the entire body of the detected individual. A body region specifying unit, an extracting unit for extracting a body region image of the specified body region, and a generating unit for generating learning data for a discrimination model for identifying an individual from the extracted body region image. .
17) In the learning data generation device according to 16), the detection unit uses an image including an individual as input data, and the detection model machine-learned to detect the individual in the image is added to the image acquisition unit. detects the individual output by inputting the acquired image as input data.
18) In the learning data generation device according to 16) or 17), the body region identifying unit identifies a body region including the entire body of the individual detected by the detecting unit by image segmentation.
19) In the learning data generation device according to any one of 16) to 18), the body region identifying unit detects a skeleton of the individual from an image representing a part of the body of the individual and estimates the posture. A body region representing the entire body of the individual is identified by referring to the output posture of the individual by inputting the detected image data of the individual as input data to the posture estimation model machine-learned in (1).
20) In the learning data generation device according to any one of 16) to 19), the generation unit classifies and accumulates body region images for each same individual to generate the learning data.
21) The behavioral analysis device includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the acquired image, an identification unit that identifies the detected individual, and the identified individual. a position specifying unit that specifies a position of a body part; an information acquisition unit that acquires time-series information representing a change in the specified position within a predetermined time period; and a behavioral analysis unit that analyzes behavioral patterns.
22) In the behavioral analysis device according to 21), the position specifying unit uses an image including an individual as input data, and uses machine learning to specify the position of the body part of the individual in the image. The position output by inputting the image acquired by the image acquisition unit into the model as input data is specified.
23) In the behavioral analysis device described in 21) or 22), the position specifying unit acquires the coordinates of the position of the body part of the individual.
24) In the behavioral analysis device according to any one of 21) to 23), the behavioral analysis unit uses time-series information as input data and performs machine learning to estimate a behavioral pattern corresponding to the time-series information. The behavior pattern outputted by inputting the time-series information acquired by the information acquiring unit as input data into the analysis model thus obtained is acquired as an analysis result.
25) In the behavioral analysis device according to any one of 21) to 24), the position specifying unit detects a skeleton of the individual from an image representing a part of the body of the individual and estimates the posture of the individual. The position is identified by referring to the posture of the individual output by inputting the detected image data of the individual to the learned posture estimation model as input data.

According to one aspect of the present invention, it is possible to realize a highly versatile video-based behavior analysis system that can be applied to various populations.

1 is a block diagram showing a main configuration of a learning data generation device according to one aspect of the present invention; FIG. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. It is a figure explaining the data generation process for learning of the data generation apparatus for learning which concerns on one aspect of this invention. 4 is a flowchart showing learning data generation processing by a learning data generation device; 1 is a block diagram showing a configuration of a main part of a learning device according to one aspect of the present invention; FIG. 4 is a flowchart showing learning processing by a learning device; 1 is a block diagram showing a configuration of a main part of an individual identification device according to one aspect of the present invention; FIG. It is a flow chart which shows individual identification processing by an individual identification device. 1 is a block diagram showing a configuration of main parts of a behavior analysis device according to one aspect of the present invention; FIG. It is a flowchart which shows the action-analysis process by a action-analysis apparatus. It is a figure explaining an example of individual identification used with the data generation device for learning concerning one mode of the present invention. FIG. 5 is a diagram illustrating an example of an object detection machine learning model used for detecting an individual in an image in the learning data generating device according to one aspect of the present invention; FIG. 5 is a diagram illustrating an example of a skeleton detection machine learning model used for detecting an individual in an image in the learning data generation device according to one aspect of the present invention; FIG. 5 is a diagram illustrating an example of a form of specifying a body region in the learning data generation device according to one aspect of the present invention; FIG. 5 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from images representing a part of the body in the learning data generation device according to one aspect of the present invention; 1 is a block diagram showing a configuration of main parts of a behavioral analysis system according to one aspect of the present invention; FIG. FIG. 4 is a diagram illustrating body parts of an individual identified by the behavioral analysis device; FIG. 4 is a diagram illustrating behavior patterns of individuals analyzed by the behavior analysis device; 4 is a flowchart showing behavioral analysis processing by the behavioral analysis device;

[Learning data generator]
A learning data generation device according to an aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a moving object detection unit that detects a moving object in the acquired image, and the individual in the detected moving object. an extracting unit for extracting a body region image of the framed body region; and an individual identified from the extracted body region image. a generation unit that generates learning data for the individual identification model.

FIG. 1 is a block diagram showing the main configuration of a learning data generation device according to one aspect of the present invention. The learning data generation device 10 includes an image acquisition unit 11 , a moving object detection unit 12 , a frame enclosing unit 13 , an extraction unit 14 , a learning data generation unit (generation unit) 15 , and a learning data storage unit 16 . The learning data generation device 10 generates learning data used for constructing an individual identification model by machine learning.

(individual)
An individual identification model obtained by machine-learning the learning data generated by the learning data generation device 10 is a machine-learned model for identifying an individual in a photographed image of a plurality of individuals. Individuals to be identified by the individual identification model can be humans, animals, other moving objects, etc., but are preferably individuals managed in groups within a closed space. Individuals to be identified by the individual identification model are more preferably multi-headed livestock individuals, more preferably cattle individuals, still more preferably dairy cow individuals, and most preferably Holstein dairy cow individuals. is. In the following, an individual Holstein dairy cow will be described as an example of an individual to be identified, but the present invention is not limited to this. Alternatively, by using a simple marker or the like, it can be applied in the same manner as Holstein dairy cow individuals. For example, the present invention can be applied to Nishikigoi, pigs, cats and dogs. As an example of a configuration using a simple marker, as shown in FIG. 16, there is a configuration in which a marker such as a numerical value is attached to the collar. FIG. 16 is a diagram illustrating an example of individual identification used by the learning data generating device according to one aspect of the present invention. By using such a simple marker, it is possible to easily identify even species that are difficult to identify by patterns, and the present invention can be applied to individuals of various species.

As shown in FIG. 2, the multi-headed individual may be an individual that is kept free-range indoors, or an individual that is tied to a chain or the like and kept in a tie. . FIG. 2 is a diagram illustrating learning data generation processing of the learning data generation device according to one aspect of the present invention. In addition, an individual that is reared with multiple animals may be an individual that is reared outdoors in a controlled area. In this embodiment, a case where the individual is a bovine individual will be described as an example.

(Image acquisition unit 11)
The image acquisition unit 11 acquires an image including an individual bovine. The image acquisition unit 11 acquires an image including the individual from the camera 20 that has captured the individual bovine through an input/output unit (not shown). The image acquisition unit 11 acquires an image including the bovine individual from the camera 20 by wireless communication or wired communication. The image acquisition unit 11 may acquire an image each time the camera 20 captures an image of the individual bovine, or may acquire an image at predetermined time intervals.

The camera 20 may photograph the bovine individual from above, or may photograph it obliquely from above. In the learning data generation device 10, not only an image photographed from above but also an image photographed obliquely from above can be suitably used. As an example, when a cow is raised indoors, the camera 20 may be installed on the ceiling or on the wall. Further, when the cow is raised outdoors, the camera 20 may be fixed to a pole, fence, or the like.

An image containing an individual bovine is an image obtained by capturing an individual bovine, and may be a visible light image, an infrared image captured by a thermo camera or the like, or obtained by a three-dimensional laser sensor or the like. It may also be a three-dimensional image. Night images can be obtained without illumination by using infrared images or three-dimensional images obtained by laser sensors. Also, the image containing the bovine individual may be a still image or a moving image. The image acquisition unit 11 preferably acquires images including the individual bovine from the plurality of cameras 20, and more preferably acquires images of the individual bovine captured from different directions by the plurality of cameras 20, respectively. The image acquisition unit 11 sends the acquired image containing the individual cow to the moving object detection unit 12 .

(Moving body detection unit 12)
The moving body detection unit 12 detects a moving body in the image including the acquired bovine individual. The moving object detection unit 12 detects the movement of an object in the image containing the individual cow, and detects the moving object as a moving object. The moving object detection unit 12 performs, for example, inter-frame difference processing for comparing a plurality of image frames captured at different times, background difference processing for comparing a captured image with a background image acquired in advance, and the like. Detect moving objects in images. The moving body detection unit 12 acquires the position information of the detected moving body in the image and sends it to the enclosing unit 13 .

(Frame Surrounding Part 13)
The framing unit 13 gives a framing to a body region including the whole body of the individual cow among the detected moving bodies. When the detected moving object is a bovine individual, the framing unit 13 gives a frame to the body region of the bovine individual. The framing unit 13 specifies the body region of the bovine individual in the image of the partial region corresponding to the acquired position information of the moving body, and adds a frame to it.

If the image of the partial region corresponding to the positional information of the moving object is framed without specifying the body region of the individual cow, as shown in FIG. A frame may be given only to the part. In some cases, a moving object other than a cow is given a frame.

On the other hand, the framing unit 13 specifies the body region of the bovine individual in the image of the partial region corresponding to the position information of the moving body and adds a frame to it. A border is added to exist inside the frame. FIG. 4 is a diagram illustrating learning data generation processing of the learning data generation device according to one aspect of the present invention.

The body region of the bovine individual in the framed portion 13 can be specified, for example, by a method using a trained model that has undergone machine learning so as to specify the body region of the bovine individual, or by a method such as feature point matching or pattern matching. can. That is, the framing unit 13 inputs the image data of the detected moving object as input data to a discriminant model subjected to machine learning so as to extract a body region including the entire body of the bovine individual from an image including the bovine individual. Adds a bounding box to the body region output by . Such a machine-learned discriminant model, for example, accumulates images of the entire body of thousands to tens of thousands of cattle individuals, using learning data as shown in FIG. It is obtained by applying known learning algorithms such as random forest, k-means method, convolutional neural network. FIG. 5 is a diagram illustrating learning data generation processing of the learning data generation device according to one aspect of the present invention.

In addition, the frame enclosing part 13 indicates that only part of the individual cow is included in the captured image, individual cows overlap each other, and part of the individual cow is hidden behind an obstacle such as a pillar or wall. In such cases, a body region image is generated from an image representing a part of the bovine individual, and a frame is added to the generated body region image. That is, the framing unit 13 inputs the image data of the detected moving object as input data to a generation model that performs machine learning so as to generate an image representing the entire body of the individual from an image representing a part of the body of the individual. The image representing the entire body of the individual output by doing is set as the body region, and the body region is given a frame.

Known generative models such as variational autoencoders (VAE) and generative adversarial networks (GAN) can be used as such generative models. For example, when a variational autoencoder is used, as shown in FIG. 6, from image data of a part of the body of an individual cow, numerical values representing features of the image data are obtained by convolution of the encoder. Then, by deconvolving the characteristic values, an image of the whole body of the individual cow is reconstructed. FIG. 6 is a diagram illustrating learning data generation processing of the learning data generation device according to one aspect of the present invention. For example, as shown in FIG. 6, machine-learned generation can be performed by applying a known generation model using learning data that accumulates images of the entire body of thousands to tens of thousands of cows. a model is obtained. In this way, by using a generative model that has been trained to restore a body region image of the entire body from an image of a part of the body of an individual bovine, an image of a part of the body of an individual bovine can be used as input image data. , can reconstruct an image of the whole body of the bovine individual. In the example shown in FIG. 6, the data output from the generative model may be only the frame surrounding the entire body, and patterns, colors, and the like may not be output.

7 and 8 are diagrams for explaining the learning data generation process of the learning data generation device according to one aspect of the present invention. As shown in FIG. 7, inputting image data representing a part of the body of an individual cow into a machine-learned generative model yields a numerical value Z representing the feature of the image data through decoder convolution. The numerical value Z is a numerical value representing the characteristics of the input image, for example, a numerical value representing information regarding the scale of the image and the area of the entire body to which the area included in the image corresponds. Then, by deconvolving the obtained numerical value Z to restore it, a body region image representing the body region of the bovine individual is generated and output. In addition, as shown in FIG. 7, inputting image data representing a part of the body of a bovine individual that includes features not included in the training data to this machine-learned generative model yields a body region of the bovine individual. An image is generated and output. In this way, even when unknown bovine image data is input to the generative model, data in which the entire body of the bovine individual is framed is output. In this way, even when image data representing a part of the body of a cow containing features not included in the training data is input to the generative model, the output is data that encloses the entire body of the cow. . 8, even if the scale of the image representing the part of the body of the individual cow is larger than that in FIG. A body region image of the individual is output.

In this way, when the image of the moving object is only a part of the individual cow, the enclosing unit 13 performs machine learning so as to generate an image representing the entire body of the individual from an image representing a part of the body of the individual. By using the generative model, the body region of the bovine individual is generated and given a frame.

(Extraction unit 14)
The extracting unit 14 extracts the body region image with the surrounding frame. The extraction unit 14 removes the image outside the frame from the image captured by the camera 20, and cuts out the body region image inside the frame. The extraction unit 14 sends the extracted body region image to the learning data generation unit 15 .

(Learning data generator 15)
The learning data generation unit 15 generates learning data for an individual identification model for identifying an individual from the extracted body region image. The training data can be unlabeled, unsupervised data, and includes, for example, hundreds to tens of thousands of body region image data. The learning data generation unit 15 can classify and accumulate body region images for each individual and generate learning data. When a predetermined number of body region images are accumulated, or when an instruction to generate learning data from the user is received, the learning data generation unit 15 generates learning data using the accumulated body region images. may be generated.

Note that the classification of the body region images for each identical individual by the learning data generation unit 15 is performed, for example, by referring to information such as the degree of similarity of the body region images and the positional relationship between the images before and after in the time series. can be done. Such classification of body region images for each same individual is a classification and estimation model that has been machine-learned so as to perform estimation based on the similarity of the body region image from the body region image and estimation based on the positional relationship with the previous and next images in the time series. may be performed using

The learning data generating unit 15 generates learning data corresponding to the individual group identified by the individual identification model. For example, an individual identification model machine-learned from learning data is a cow individual number. 1 to bovine individual No. When identifying a population X containing up to 10 bovine individuals, learning data including body region image data of the ten bovine individuals contained in the population X is generated. The learning data generation unit 15 stores the generated learning data in the learning data storage unit 16 in association with information on the corresponding population.

(Learning data storage unit 16)
The learning data storage unit 16 stores the generated learning data. Further, the learning data storage unit 16 may store data such as position information in the image of the image acquired by the image acquisition unit 11 and the moving object detected by the moving object detection unit 12 . Further, the learning data storage unit 16 may store a program indicating a series of processes necessary for the operation performed by the learning data generation device 10 .

(Learning data generation processing)
An example of the learning data generation process by the learning data generation device 10 will be described with reference to FIG. 9 . FIG. 9 is a flowchart showing learning data generation processing by the learning data generation device 10 of FIG.

First, in step S91, the image acquiring unit 11 acquires an image including a bovine individual. Note that the learning data generation process by the learning data generation device 10 may be started by receiving input of captured image data from the camera 20 .

Next, the moving body detection unit 12 detects a moving body in the image including the acquired bovine individual (step S92). Subsequently, in step S93, the enclosing unit 13 encloses the body region of the individual cow among the detected moving bodies. Next, in step S94, the extracting unit 14 extracts the body region image with the surrounding frame. Then, the learning data generator 15 generates learning data from the body region image (step S94). The learning data generation unit 15 stores the generated learning data in the learning data storage unit 16, and ends the learning data generation process.

In this way, the learning data generation device 10 extracts and accumulates body region images of bovine individuals whose moving bodies have been detected, thereby automatically generating learning data used for constructing an individual identification model by machine learning. . This makes it easy to generate learning data for constructing an individual identification model. Therefore, it is possible to easily generate learning data for each individual group with respect to various individual groups having different individual configurations and imaging conditions. In addition, the learning data generated by the learning data generation device 10 includes body region images of individual cows in various orientations. Therefore, individual identification using an individual identification model learned using such learning data enables individual identification regardless of the orientation of the individual cow in the image to be identified.

Conventionally, machine learning for image recognition has a problem that it takes time and effort to prepare data for learning. According to the learning data generation device 10, learning data used for machine learning of an individual identification model can be automatically accumulated. As a result, for example, it is possible to easily construct an individual identification model corresponding to each dairy farm site with different individual compositions of cattle populations and different photographing conditions, and it is possible to realize general-purpose use of the individual identification model.

[Learning device]
A learning device according to an embodiment of the present invention includes a learning data acquisition unit that acquires learning data generated by a learning data generation device according to an embodiment of the present invention; and a learning unit for constructing an individual identification model for identifying an individual by performing machine learning so as to identify the individual contained in the image.

FIG. 10 is a block diagram showing a main configuration of a learning device according to one aspect of the present invention. The learning device 100 includes a learning data acquisition unit 101 , a learning unit 102 , and a learning model storage unit 103 . The learning device 100 constructs an individual identification model for identifying individuals.

(Learning data acquisition unit 101)
The learning data acquisition unit 101 acquires the learning data stored in the learning data storage unit 16 of the learning data generation device 10 via an input/output unit (not shown). The learning data acquisition unit 101 acquires learning data corresponding to a group of individuals identified by an individual identification model machine-learned in the learning unit 102 . For example, if the individual identification model is a bovine individual no. 1 to bovine individual No. In the case of a trained model that identifies bovine individuals in a population X including up to ten bovine individuals, the learning data acquisition unit 101 acquires learning data corresponding to the population X. The learning data acquisition unit 101 sends the acquired learning data to the learning unit 102 .

(Learning unit 102)
The learning unit 102 constructs an individual identification model for identifying individuals using the acquired learning data. The learning unit 102 machine-learns an individual identification model by applying known learning algorithms such as a support vector machine, a random forest, a k-means method, and a convolutional neural network, using the acquired learning data.

The machine-learned individual identification model constructed in the learning unit 102 is a learned model that outputs identification information for identifying an individual by inputting image data including an individual as input data. For example, when the individual identification model identifies a bovine individual in the population X, the individual identification model identifies any bovine individual in the population X (bovine individual No. 1, cow Individual No. 2, ... or bovine individual No. 10) is output. The learning unit 102 stores the machine-learned individual identification model in the learning model storage unit 103 .

(Learning model storage unit 103)
The learning model storage unit 103 stores machine-learned individual identification models. Further, the learning data acquired by the learning data acquisition unit 101 may be stored in the learning model storage unit 103 . Also, the learning model storage unit 103 may store a program indicating a series of processes necessary for the operation performed by the learning device 100 .

(learning process)
An example of the learning process by the learning device 100 will be described with reference to FIG. FIG. 11 is a flow chart showing learning processing by the learning device 100 of FIG.

First, in step S<b>111 , the learning data acquisition unit 101 acquires learning data from the learning data generation device 10 . Next, the learning unit 102 machine-learns an individual identification model for identifying an individual using the acquired learning data (step S112). Then, the learning unit 102 stores the machine-learned individual identification model in the learning model storage unit 103 (step S113), and ends the learning process.

In this way, the learning device 100 constructs an individual identification model by machine learning using learning data generated for each individual population. It is possible to easily construct an individual identification model corresponding to each site. As a result, general-purpose use of the individual identification model can be realized.

[Individual identification device]
An individual identification device according to an embodiment of the present invention includes an image acquisition unit that acquires an image including an individual, and an individual identification model constructed by a learning device according to an embodiment of the present invention, which includes an image of the acquired image. an identification unit that outputs individual identification information of an individual contained in the image by inputting data as input data.

FIG. 12 is a block diagram showing the main configuration of an individual identification device according to one aspect of the present invention. The individual identification device 120 includes an image acquisition section 121 , an identification section 122 and an identification image accumulation section 123 . The individual identification device identifies individuals in the population using the individual identification model.

(individual)
Individuals to be identified by the individual identification device 120 can be humans, animals, other mobile objects, etc., but are preferably individuals managed in groups within a closed space. Individuals to be identified by the individual identification model are more preferably multi-headed livestock individuals, more preferably cattle individuals, still more preferably dairy cow individuals, and most preferably Holstein dairy cow individuals. is. As shown in FIG. 2, the multi-headed individual may be an individual that is kept free-range indoors, or an individual that is tied to a chain or the like and kept in a tie. . In addition, an individual that is reared with multiple animals may be an individual that is reared outdoors in a controlled area. In this embodiment, a case where the individual is a bovine individual will be described as an example.

(Image acquisition unit 121)
The image acquisition unit 121 acquires an image including an individual bovine. The image acquisition unit 121 acquires an image including an individual from the camera 20 that has captured the image of the individual bovine through an input/output unit (not shown). The image acquisition unit 121 acquires an image including the bovine individual from the camera 20 by wireless communication or wired communication. The image acquisition unit 121 may acquire an image each time the camera 20 captures an image of an individual bovine, or may acquire an image at predetermined time intervals.

The camera 20 may photograph the bovine individual from above, or may photograph it obliquely from above. In the individual identification device 120, it is possible to suitably use not only an image of an individual bovine photographed from above, but also an image photographed obliquely from above. As an example, when a cow is raised indoors, the camera 20 may be installed on the ceiling or on the wall. Further, when the cow is raised outdoors, the camera 20 may be fixed to a pole, fence, or the like.

An image containing an individual bovine is an image obtained by capturing an individual bovine, and may be a visible light image, an infrared image captured by a thermo camera or the like, or obtained by a three-dimensional laser sensor or the like. It may also be a three-dimensional image. Night images can be obtained without illumination by using infrared images or three-dimensional images obtained by laser sensors. Also, the image containing the bovine individual may be a still image or a moving image. The image acquisition unit 121 sends the acquired image containing the bovine individual to the identification unit 122 .

(Identification unit 122)
The identification unit 122 inputs data of the acquired image as input data to the individual identification model constructed by the learning device 100, and outputs individual identification information of the cow individual in the image. The identification unit 122 acquires the individual identification model stored in the learning model storage unit 103 of the learning device 100 via an input/output unit (not shown). The individual identification model is a machine-learned model corresponding to the individual group identified by the individual identification device 120 . For example, the individual group to be identified by the individual identification device 120 is cow individual No. 1 to bovine individual No. In the case of a population X containing up to 10 10 cattle individuals, the individual identification model is a machine-learned model corresponding to the population X.

The identification unit 122 associates individual identification information indicating which bovine individual contained in the acquired image with the image data and stores them in the identification image storage unit 123 . In addition, as shown in FIG. 4, the identification unit 122 may attach the individual identification information to the image data and display it on the display unit (not shown). For example, as shown in FIG. 4, the identifying unit 122 assigns a bovine individual No. to each bovine individual in the image to identify the bovine individual. is displayed.

The image identified by the identification unit 122 may be an image obtained by extracting a body region image including the entire body of the bovine from the image acquisition unit 121 . That is, the individual identification device 120 includes a frame enclosing unit that encloses the body region of the bovine individual in the image acquired by the image acquisition unit 121, and an extraction unit that extracts the body region image to which the frame encloses. You may have more. Further, a moving object detection unit that detects a moving object in the image acquired by the image acquisition unit 121 may be further provided. Then, the identifying unit 122 inputs the body region image extracted by the extracting unit as input data to the individual identification model, thereby outputting the individual identification information of the bovine individual in the image. The image identified by the identifying unit 122 may be the image itself obtained by the image obtaining unit 121 from the camera 20, for example, when the individual cow in the image is alone.

In this way, the individual identification device 120 does not use the image itself acquired from the camera 20 as an image to be identified by the individual identification device 120, but an image that has been subjected to the same processing as the learning data generation processing by the learning data generation device 10. A region image may also be used. As a result, the image that is the target of individual identification does not include portions unnecessary for individual identification, and more accurate individual identification is possible.

(Identification image accumulation unit 123)
The identification image accumulating unit 123 stores images that have undergone individual identification. Further, the identification image storage unit 123 may store a program indicating a series of processes required for the operation performed by the individual identification device 120 .

(Individual identification processing)
An example of individual identification processing by the individual identification device 120 will be described with reference to FIG. 13 . FIG. 13 is a flowchart showing individual identification processing by the individual identification device 120 of FIG.

First, in step S131, the image acquiring unit 121 acquires an image including a bovine individual. It should be noted that the individual identification processing by the individual identification device 120 may be started by receiving input of captured image data from the camera 20 .

Next, the identification unit 122 reads a trained individual identification model from the learning device 100 (step S132). Subsequently, in step S133, the identification unit 122 inputs image data to the read individual identification model. Then, in step S134, the identification unit 122 associates the individual identification information of the bovine individual output by the individual identification model with the image, stores them in the identification image storage unit 123, and terminates the individual identification process.

In this way, the individual identification device 120 identifies individuals using an individual identification model corresponding to each individual group. It is possible. Further, by using an individual identification model trained by the learning device 100, individual identification is possible regardless of the orientation of the individual cow in the image to be identified. According to the individual identification device 120, general-purpose use of the individual identification model is possible.

[Behavior analysis device]
A behavior analysis device according to an embodiment of the present invention uses an image acquisition unit that acquires an image including an individual, and learning data generated by the learning data generation device to identify the individual included in the image. an identification unit that outputs individual identification information of an individual contained in the image by inputting the data of the acquired image as input data into an identification model that identifies an individual, which is constructed by machine learning; It comprises an identification image acquisition unit that acquires time-series images of the identified individual, and an analysis unit that analyzes the behavior of the individual by referring to the acquired images. That is, the behavior analysis device includes an identification image acquisition unit that acquires time-series images of an individual identified by the individual identification device according to an embodiment of the present invention; and an analysis unit for analyzing.

FIG. 14 is a block diagram showing the main configuration of a behavior analysis device according to one aspect of the present invention. The behavior analysis device 140 includes an identification image acquisition unit 141 , an analysis unit 142 and an analysis result storage unit 143 . A behavior analysis device tracks and analyzes the behavior of the identified individual.

(Identification image acquisition unit 141)
The identification image acquisition unit 141 acquires the image of the individual cow accumulated in the identification image accumulation unit 123 of the individual identification device 120 via an input/output unit (not shown). The identification image acquisition unit 141 acquires time-series images of the individual cow identified by the individual identification device 120 . The identification image acquisition unit 141 sends the acquired time-series images of the bovine individual to the analysis unit 142 .

(Analysis unit 142)
The analysis unit 142 analyzes the behavior of the bovine individual by referring to the time-series images of the bovine individual. A time-series image of a bovine individual allows the behavior of the bovine individual to be tracked. The analysis unit 142 analyzes the behavior of the bovine within a predetermined period of time by tracking the behavior of the bovine based on the time-series images of the bovine. The analysis unit 142 analyzes, for example, the amount of behavior, the range of behavior, and the pattern of behavior of the individual cow within a predetermined period of time by tracking the behavior of the individual bovine based on the time-series images of the individual bovine. The analysis unit 142 stores the analysis result of the bovine individual in the analysis result storage unit 143 .

The analysis unit 142 may predict the physical condition, estrous state, disease, etc. of the bovine by analyzing behavioral patterns such as the time the bovine is sitting, standing, and moving around. In addition, the analysis unit 142 may manage the rearing of individual cows by analyzing behavior patterns such as the number and time of drinking water and the number and time of eating food.

The analysis unit 142 obtains the positional coordinates in the image of the individual cow from the image of the individual cow, and converts the positional coordinates into the spatial coordinates of the space in which the individual cow exists, thereby determining the range of behavior of the individual cow in the space, Behavioral patterns and the like may be tracked and analyzed. Conversion from positional coordinates to spatial coordinates is carried out by, for example, using information on reference points whose spatial coordinates on the image are known to convert the acquired positional coordinates into spatial coordinates of the space in which the bovine individual exists. can be done. If the image of the bovine individual referred to by the analysis unit 142 is a body region image including the entire body of the bovine individual, the positional coordinates of the feet of the bovine individual can be obtained. can be tracked. The position coordinates or spatial coordinates of the feet of the bovine individual may be displayed on a display unit (not shown) together with the body region image of the bovine individual as shown in FIG.

(behavior analysis processing)
An example of behavior analysis processing by the behavior analysis device 140 will be described with reference to FIG. 15 . FIG. 15 is a flowchart showing action analysis processing by the action analysis device 140 of FIG.

First, in step S151, the identification image acquisition unit 141 acquires time-series image data of an individual bovine. Next, the analysis unit 142 analyzes the behavior of the bovine by tracking the behavior of the bovine using the time-series images of the bovine (step S152). The analysis unit 142 stores the analysis result in the analysis result storage unit 143 and ends the behavior analysis process.

In this way, the behavior analysis device 140 analyzes the behavior of individuals by tracking images of individuals identified using individual identification models corresponding to each individual group. Accurate and easy behavioral analysis of individuals corresponding to each dairy field with different conditions is possible. 1, the learning device 100 shown in FIG. 10, the individual identification device 120 shown in FIG. 12, and the behavior analysis system 140 shown in FIG. included in the category of

[Learning data generation method and behavior analysis method]
A learning data generation method according to an aspect of the present invention includes an image acquisition step of acquiring an image including an individual, a moving object detection step of detecting a moving object in the acquired image, and the individual detected in the moving object. a framing step of giving a framing to a body region including the entire body of the individual; an extracting step of extracting a body region image of the body region given the framing; and identifying an individual from the extracted body region image and a generation step of generating training data for the discriminative model. As an example, a learning data generating method according to an aspect of the present invention is realized by the learning data generating device according to an aspect of the present invention described above. Therefore, the description of the learning data generating method according to one aspect of the present invention conforms to the description of the learning data generating device according to one aspect of the present invention described above.

A behavior analysis method according to an aspect of the present invention includes an image acquisition step of acquiring an image including an individual, and using the learning data generated by the learning data generation method described above to identify the individual included in the image. an identification step of outputting individual identification information of an individual contained in the image by inputting the data of the acquired image as input data into an identification model for identifying an individual constructed by machine learning; and an analysis step of analyzing the behavior of the individual with reference to the acquired images. As an example, a behavior analysis method according to one aspect of the present invention is realized by the behavior analysis device according to one aspect of the present invention described above. Therefore, the description of the behavior analysis method according to one aspect of the present invention conforms to the description of the behavior analysis device according to one aspect of the present invention described above.

[Example of realization by software]
The control blocks of the learning data generation device 10, the learning device 100, the individual identification device 120, and the behavior analysis device 140 may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like. , may be implemented by software.

In the latter case, each device described above comprises a computer that executes program instructions, which are software that implements each function. This computer includes, for example, one or more processors, and a computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a "non-temporary tangible medium" such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. In addition, a RAM (Random Access Memory) for developing the above program may be further provided. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one aspect of the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

That is, the learning data generation device, the learning device, the individual identification device, and the behavior analysis device 140 according to each aspect of the present invention may be realized by a computer. A program that causes a computer to implement each of the above-described devices by operating as a software element, and a computer-readable recording medium recording the program are also included in the scope of the present invention.

Specifically, the following program also falls within the scope of the present invention:
A program for causing a computer to function as a learning data generation device according to each aspect of the present invention, comprising: an image acquisition step of acquiring an image containing an individual; and a moving object detection step of detecting a moving object in the acquired image. a framing step of giving a framing to a body region including the entire body of the individual in the detected moving body; an extracting step of extracting a body region image of the body region given the framing; A program characterized by executing a generation step of generating learning data for a discrimination model that identifies an individual from the body region image:
A program for causing a computer to function as a behavior analysis device according to each aspect of the present invention, comprising an image acquisition step of acquiring an image including an individual, and learning generated by the learning data generation device according to each aspect of the present invention By inputting the acquired data of the image as input data to an identification model for identifying an individual, which is constructed by machine learning so as to identify the individual contained in the image, using the data for the image an identification step of outputting individual identification information of an individual included in the identification step, an identification image obtaining step of obtaining time-series images of the identified individual, and an analysis step of analyzing the behavior of the individual by referring to the obtained images. A program characterized by executing and
A program for causing a computer to function as a learning device according to each aspect of the present invention, comprising: a learning data acquisition step for acquiring learning data generated by the learning data generation device according to each aspect of the present invention; A program characterized by:
A program for causing a computer to function as an individual identification device according to each aspect of the present invention, comprising an image acquisition step of acquiring an image including an individual, and the identification model constructed by the learning device according to each aspect of the present invention. and an identification step of outputting individual identification information of an individual contained in the image by inputting data of the acquired image as input data.

[Modification]
A learning data generation device according to another aspect of the present invention includes an image acquisition unit that acquires an image including an individual, a detection unit that detects the individual in the acquired image, and a whole body of the individual that has been detected. a body region specifying unit that specifies a body region including a body region, an extraction unit that extracts a body region image of the specified body region, and a generator that generates training data for a discrimination model that identifies an individual from the extracted body region image and

That is, in the learning data generation device, the method for detecting an individual in an image is not limited to the moving object detection method, and other methods for detecting an individual may be used. Another example of detecting individuals in images is a method using machine learning. Therefore, in the training data generation device, a configuration using a detection unit that detects an individual in an image using machine learning instead of the moving object detection unit is also included in the scope of the present invention.

When using machine learning to detect an individual in an image, it is preferable to create a machine learning model suitable for detecting the individual in accordance with the imaging direction of the image. For example, when an image of an individual photographed obliquely from above is used, a machine learning model generated so as to be suitable for detecting an individual in the image photographed obliquely from above is used. Methods for detecting individuals in images using machine learning include methods for detecting objects and methods for detecting skeletons of individuals using machine learning. Such a detection unit can also be used in the individual identification device according to one aspect of the present invention.

When using a machine learning model created to detect an object from an image, as shown in FIG. 17, by inputting image data of an individual cow into the object detection machine learning model as input data, the part of the individual cow in the image is obtained. is output. When using a machine learning model created to detect a skeleton from an image, as shown in FIG. 18, image data of an individual cow is input to the skeleton detection machine learning model as input data. As a result, the skeletal position of the bovine individual in the image is detected and grouped for each individual, and the detection result of the bovine individual part in the image is output.

In addition, in the learning data generation device, the method of specifying the body region including the entire body of the detected individual is not limited to the method of adding a frame, but can be realized by other methods of specifying the body region of the individual. good too. Another example of identifying the body region of an individual is a method using image segmentation, as shown in FIG. FIG. 19 is a diagram illustrating an example of a form of specifying a body region in the learning data generation device according to one aspect of the present invention. As shown in FIG. 19, a body region is specified by image segmentation in an image containing a bovine individual, and, for example, position coordinates indicating only the specified body region range are obtained. In this way, in the training data generation device, a configuration that uses a body region specifying unit that specifies the body region of an individual by image segmentation instead of the frame enclosing unit is also included in the scope of the present invention. Also, such a body region identifying unit can be used in an individual identification device according to one aspect of the present invention.

Furthermore, in the learning data generation device, the method of identifying the image of the whole body from the image representing the part of the body of the individual is not limited to the method of restoring the image representing the other part of the body. It may be realized by another method of identifying the image of the whole body from the image representing the . Another example of identifying an image of the whole body from an image representing a part of the body is a method of referring to a posture estimated based on bone structure detection using machine learning.

FIG. 20 shows an example of specifying an image of the whole body from an image representing a part of the body by referring to the posture estimated based on bone structure detection. FIG. 20 is a diagram illustrating an example of a posture estimation model used to identify an image of the entire body from images representing a part of the body in the learning data generation device according to one aspect of the present invention. As shown in FIG. 20, when image data containing only a part of a bovine individual is obtained, the posture estimated from the parts included in the image (visible parts) is referred to, and the parts not included in the image are extracted. Identify the position of (invisible parts).

For example, only the nose, left eye, right eye, left ear, right ear, left shoulder, and left waist are visible in the image data of FIG. When this image data is input to the posture estimation model, machine learning detects the skeleton of the visible parts. And a skeletal frame including the positions of 10 invisible parts (right shoulder, right hip, left front ankle, left front foot, left rear foot, left rear foot, right front foot, right front foot, right rear foot, right rear foot). Information is output as the result of the pose estimation. The parts of the bovine individual shown here are examples, and the types and number of parts included in the image and not included in the image are not limited to these, and can be set as appropriate.

This makes it possible to identify the entire body region and use it for analyzing the behavior of the individual, even if the image contains only a part of the individual due to an obstructing object. In this way, information on parts included in the image is used as information for estimating parts not included in the image. Also, as the input data, for example, the position of each part detected in images before and after the target image in time series may be used.

In this way, in the learning data generation device, the body region identifying unit detects the skeleton of the individual from an image representing a part of the body of the individual and estimates the posture of the individual by machine learning. The scope of the present invention also includes a configuration for identifying a body region representing the entire body of an individual by referring to the posture of the individual output by inputting the image data of the moving object as input data. Also, such a body region identifying unit can be used in an individual identification device according to one aspect of the present invention.

[Behavioral analysis system]
The behavior type (behavior type) of each individual animal is important information for managing animals. For example, eating behavior and resting behavior serve as indicators that maintenance behavior is ensured. In addition, in ruminant livestock, ruminant behavior, among other behavioral patterns, is regarded as useful information that serves as a barometer reflecting signs of various diseases including feed intake, recovery state, or stress state. Conventionally, systems that use wearable sensors and the like to grasp behavior patterns have been put into practical use, but there are problems in terms of cost and labor involved in management. On the other hand, with conventional image analysis, it is difficult to detect behaviors such as rumination, which require understanding. ADVANTAGE OF THE INVENTION According to one aspect of this invention, it is labor-saving and can acquire behavior type data easily by grasping|ascertaining the behavior type of an animal using image analysis. In addition, according to one aspect of the present invention, it is possible to detect even behavioral types that require grasping of actions.

A behavioral analysis system according to one aspect of the present invention will be described with reference to FIG. FIG. 21 is a block diagram showing the main configuration of a behavioral analysis system 200 according to one aspect of the present invention. The behavioral analysis system 200 includes a behavioral analysis device 220 . The behavioral analysis system 200 also includes a camera 20 , a learning data generation device 210 and a learning device 100 .

(Behavioral analysis device 220)
The behavioral analysis device 220 includes an image acquisition unit 221 , a detection unit 222 , an identification unit 223 , a position identification unit 224 , an information acquisition unit 225 and a behavioral analysis unit 226 . The behavioral analysis device 220 further includes an analysis result storage unit 227 . The behavioral analysis device 220 detects and analyzes behavioral patterns of individuals.

<Individual>
The individuals whose behavior patterns are analyzed by the behavior pattern analysis device 220 can be humans, animals, other moving bodies, etc., but they may be individuals managed in groups in a closed space, or individual It may be an individual managed by Such individuals include multi-housed domestic animals and single-housed domestic animals. The livestock individual may be a ruminant individual such as a cow individual, for example a dairy cow individual, more particularly a Holstein cow individual. The multi-headed individual may be an individual that is kept free-range indoors, or an individual that is tied up with a chain or the like. In addition, an individual that is reared with multiple animals may be an individual that is reared outdoors in a controlled area. In this embodiment, a case where the individual is a bovine individual will be described as an example.

<Image acquisition unit 221>
The image acquisition unit 221 acquires an image including the individual cow photographed by the camera 20 . The image acquisition unit 221 is configured similarly to the image acquisition unit 121 of the individual identification device 120 described above. Therefore, the description of the image acquisition unit 121 of the individual identification device 120 is used as the description of the image acquisition unit 221, and detailed description thereof is omitted. Since the camera 20 is also the same as the camera 20 described with respect to the individual identification device 120, detailed description thereof will be omitted.

<Detector 222>
The detection unit 222 detects individual cows in the image acquired by the image acquisition unit 221 . Examples of methods for detecting individual cows in images include moving object detection methods and methods using machine learning. When using the moving object detection method, the detection unit 222 can be configured in the same manner as the moving object detection unit 12 of the learning data generation device 10 . When using machine learning, machine learning models can be used, such as the object detection machine learning model shown in FIG. 17 and the skeleton detection machine learning model shown in FIG. If the image contains a single bovine individual, it is not necessary to detect the bovine individual, and the acquired image may be subjected to position identification, which will be described later.

<Identification unit 223>
The identification unit 223 identifies the bovine individuals detected by the detection unit 222 . The identification unit 223 is configured similarly to the identification unit 122 of the individual identification device 120 described above. Therefore, the description of the identification unit 122 of the individual identification device 120 is used as the description of the identification unit 223, and detailed description thereof is omitted. If the image contains a single bovine individual, it is not necessary to identify the bovine individual, and the acquired image may be subjected to position identification, which will be described later.

<Position specifying unit 224>
The position specifying unit 224 specifies the position of the body part of the identified individual. The position of the part of the body of the individual specified by the position specifying unit 224 is, for example, the tip of the nose, the mouth, the heel of the forelimb, the toe of the forelimb, and the like. FIG. 22 shows an example in which the position specifying unit 224 specifies body parts of an individual cow. The left side of FIG. 22 shows an example of identifying the nose tip, mouth, forelimb heels, and forelimb hooves of a sitting bovine individual, and the right side shows an example of identifying the nose tip, mouth, forelimb heels, and forelimb hooves of a bovine individual in a standing position. indicates The number of body parts of the individual to be specified is not limited to this, and may be smaller or larger.

The position identifying unit 224 may use machine learning to identify the position of the body part of the individual. For example, using an image including an individual as input data, inputting the image acquired by the image acquiring unit as input data to a position specifying model that has undergone machine learning to specify the position of the body part of the individual in the image. Specifies the position output by . By inputting an image including an individual to the localization model, an image in which body parts of the individual are specified as shown in FIG. 22 can be output.

The position specifying unit 224 may specify the position by acquiring the coordinates of the position of the body part of the individual. The position specifying unit 224 acquires the position coordinates of the body part of the individual in the image. The position specifying unit 224 may convert the acquired position coordinates into spatial coordinates of the space in which the bovine individual exists. The position specifying unit 224 acquires position coordinates in a plurality of frames of images captured at different times.

The position specifying unit 224 inputs the detected image data of the individual as input data to a posture estimation model that has undergone machine learning so as to detect the skeleton of the individual from an image representing a part of the body of the individual and estimate the posture. The position may be identified by referring to the posture of the individual output by The posture estimation model used in position specifying section 224 is the same as the posture estimation model shown in FIG. In addition, the position specifying unit 224 is a generation model that performs machine learning so as to generate an image representing the entire body of the individual from an image representing a part of the body of the individual used in the enclosing unit 13 of the learning data generation device 10. may be used. As a result, even if there is a shielding object in the image and only a part of the individual's body is included, it is possible to specify the body part of the individual, and to acquire the position coordinates. is also possible.

Note that when the image data includes only a part of the body of the individual, the position specifying unit 224 does not generate an image representing the entire body of the individual, but rather A part may be identified and position coordinates may be obtained. In this case, for example, the body part of the individual may be identified using a position identification model trained so as to be able to identify the body part of the individual from an image of a part of the body of the individual. Behavioral analysis can be performed by using a partial image of an individual's body as it is without restoring an image of the entire body from an image of the partial body of the individual.

<Information Acquisition Unit 225>
The information acquisition unit 225 acquires time-series information representing changes in the specified position within a predetermined period of time. The time-series information acquired by the information acquisition unit 225 is information representing changes in parts of the bovine individual obtained from consecutive images within a predetermined time, and each part obtained based on the position coordinates acquired by the position specifying unit 224. and the average displacement between image frames for each site.

<Behavioral Analysis Unit 226>
The behavior type analysis unit 226 refers to the acquired time-series information and analyzes the behavior type of the individual. By referring to the time-series information, the behavioral analysis unit 226 can detect the behavior of the bovine individual, and can also track the behavior of the bovine individual. As a result, it is possible to analyze the behavioral pattern from the movement and behavior of the individual cow at a predetermined timing and within a predetermined period of time. The behavior type analysis unit 226 stores the analysis result of the behavior type of the individual cow in the analysis result storage unit 217 .

The behavioral pattern of the bovine individual analyzed by the behavioral pattern analysis unit 226 is, for example, a type of behavior such as lying down resting, lying down rumination, standing rest, standing rumination, feeding, and the like. As shown in FIG. 23, by acquiring in advance information that associates time-series information obtained from consecutive images within a predetermined time period with behavioral patterns of individual cows corresponding to the time-series information, time-series information can be obtained. Based on the information, the behavioral pattern of the bovine individual can be analyzed.

The behavioral pattern analysis unit 226 may analyze behavioral patterns of individual bovines using an analysis model constructed using teacher data in which time-series information and behavioral patterns of individual bovines are associated with each other. By inputting time-series information as input data to such an analysis model, behavior patterns of individual cows are output. As shown in FIG. 23, the behavioral pattern analysis unit 226 may display the analyzed behavioral pattern of the bovine individual together with the image of the bovine individual on the display unit (not shown). Further, when the image data includes only a part of the body of the individual, the behavior analysis unit 226 analyzes the behavior of the individual using only the position coordinate information of the specified part of the body. may In this case, the behavior type of the individual may be analyzed using an analysis model trained so that the behavior type can be analyzed even if there is missing data in the position coordinates of the body parts.

<Learning data generation device 210>
The learning data generation device 210 included in the behavioral analysis system 200 includes an image acquisition unit 11 that acquires an image including an individual, a detection unit 212 that detects the individual in the acquired image, and a body region of the detected individual. an identifying unit 213 for identifying, an extracting unit 14 for extracting a body region image of the identified body region, and a learning data generating unit 15 for generating learning data for a discrimination model for identifying an individual from the extracted body region image. It has The learning data generation device 210 further includes a learning data storage unit 16 that stores the generated learning data.

The detection unit 212 detects an individual in the image using a moving object detection method, a method using machine learning, or the like. Methods for detecting individuals in images using machine learning include methods for detecting objects and methods for detecting skeletons of individuals using machine learning. The specifying unit 213 specifies the body region of the detected individual by using a method of adding a frame to the body region including the entire body of the detected individual, a method of using image segmentation, or the like.

The image acquisition unit 11, the extraction unit 14, and the learning data generation unit 15 are the same as these members in the learning data generation device 10 shown in FIG. That is, the learning data generation device 210 is configured in the same manner as the learning data generation device 10 . Therefore, the description of the learning data generation device 10 is used as the description of the learning data generation device 210, and detailed description thereof is omitted.

The learning data generation device 210 generates not only learning data for constructing a discriminant model but also learning data for constructing an analysis model used by the behavioral analysis unit 226 to analyze behavioral patterns. can also be used for

<Learning Device 100>
The learning device 100 uses the learning data generated by the learning data generation device 210 to construct a discriminative model, an analysis model, and the like. A learning device 100 included in the behavioral analysis system 200 is the same as the learning device 100 shown in FIG. Therefore, the description of the learning device 100 shown in FIG. 10 is used as the description of the learning device 100 included in the behavioral analysis system 200, and detailed description thereof is omitted.

<Behavioral analysis processing>
An example of behavioral analysis processing by the behavioral analysis device 220 will be described with reference to FIG. 24 . FIG. 24 is a flowchart showing behavioral analysis processing by the behavioral analysis device 220 of FIG.

First, in step S241, the image acquisition unit 221 acquires an image including an individual bovine. The behavioral analysis processing by the behavioral analysis device 220 may be started by receiving an input of captured image data from the camera 20, or by receiving an operation input for starting the behavioral analysis processing from the user. You may

Next, the detection unit 222 inputs the image data to the skeleton detection machine learning model, and detects a bovine individual in the image data (step S242). Then, the identification unit 223 inputs the image data of the detected bovine individual to the identification model, acquires the individual identification information of the bovine individual, and identifies the bovine individual (step S243).

The position specifying unit 224 inputs the image data of the identified bovine individual to the position specifying model, and obtains the position coordinates for specifying the part of the body of the bovine individual (step S244). Then, the information acquisition unit 225 acquires the specified position coordinates for several consecutive frames, and calculates the average distance between each part and the average displacement between frames of each part as time-series information (step S245).

The behavioral pattern analysis unit 226 inputs the time-series information into the analysis model and analyzes the behavioral pattern of the bovine individual (step S246). The behavioral analysis unit 226 stores the analysis result in the analysis result storage unit 227, and ends the behavioral analysis process.

In this way, the behavioral analysis device 220 can analyze the behavioral pattern of an individual by identifying the position of the body part of the individual from the image data and tracking changes in the position. As a result, it is possible to easily and labor-savingly analyze the behavior type, and it is possible to detect even behavior types such as rumination behavior that require grasping of the behavior.

The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

10 learning data generation device 11 image acquisition unit 12 moving object detection unit 13 frame enclosing unit 14 extraction unit 15 learning data generation unit (generation unit) 100 learning device 101 learning data acquisition unit 102 learning Unit 120 Individual identification device 121 Image acquisition unit 122 Identification unit 140 Behavior analysis device 141 Identification image acquisition unit 142 Analysis unit 200 Behavioral analysis system 220 Behavioral analysis device 221 Image acquisition unit 222 Detection Unit 223 Identification Unit 224 Location Identification Unit 225 Information Acquisition Unit 226 Behavior Analysis Unit

Claims (25)

an image acquisition unit that acquires an image containing an individual;
a moving body detection unit that detects a moving body in the acquired image;
a frame enclosing unit that encloses a body region including the entire body of the individual in the detected moving body;
an extracting unit that extracts a body region image of the body region to which the frame is attached;
A learning data generation device, comprising: a generation unit that generates learning data for a discrimination model that discriminates one individual included in a group of individuals to be identified from other individuals from the extracted body region image. The frame enclosing part is output by inputting image data of the detected moving object as input data to a discriminant model that has undergone machine learning so as to extract a body region that includes the entire body of the individual from an image that includes the individual. 2. The learning data generation device according to claim 1, wherein the body region is framed. The frame enclosing unit inputs image data of the detected moving object as input data to a generative model that performs machine learning so as to generate an image representing the entire body of the individual from an image representing a part of the body of the individual. 3. The learning data generation device according to claim 1, wherein an image representing the entire body of an individual output by is used as a body region, and a frame is added to the body region. 4. The learning data generation device according to any one of claims 1 to 3, wherein the individual is a livestock individual raised with multiple animals. The learning data generation device according to any one of claims 1 to 4, wherein the generation unit classifies and accumulates the body region images for each individual to generate the learning data. A program for causing a computer to function as the learning data generation device according to any one of claims 1 to 5,
an image acquisition step of acquiring an image containing the individual;
a moving body detection step of detecting a moving body in the acquired image;
a framing step of framing a body region including the entire body of the individual in the detected moving body;
an extracting step of extracting a body region image of the body region to which a frame is attached;
and a generation step of generating learning data for a discrimination model that discriminates one individual included in a group of individuals to be identified from other individuals from the extracted body region image. an image acquisition unit that acquires an image containing an individual;
An identification target constructed by machine learning to identify an individual contained in an image using learning data generated by the learning data generation device according to any one of claims 1 to 5. an identification unit that outputs the individual identification information of the individual contained in the image by inputting the data of the obtained image as input data to a identification model that identifies one individual contained in the population from other individuals; ,
an identification image acquisition unit that acquires time-series images of the identified individual;
and an analysis unit that analyzes the behavior of the individual by referring to the acquired image. A program for causing a computer to function as the behavior analysis device according to claim 7,
an image acquisition step of acquiring an image containing the individual;
An identification target constructed by machine learning to identify an individual contained in an image using learning data generated by the learning data generation device according to any one of claims 1 to 5. an identification step of outputting individual identification information of an individual included in the image by inputting data of the acquired image as input data to a discrimination model that identifies one individual included in the population from other individuals; ,
an identification image acquisition step of acquiring time-series images of the identified individual;
and an analysis step of analyzing the behavior of the individual with reference to the acquired image. A learning data acquisition unit that acquires learning data generated by the learning data generation device according to any one of claims 1 to 5;
Learning to construct a discrimination model that discriminates one individual contained in a group of individuals to be identified from other individuals by performing machine learning so as to identify individuals contained in images using the acquired learning data. A learning device comprising: A program for causing a computer to function as the learning device according to claim 9,
A learning data acquisition step of acquiring learning data generated by the learning data generation device according to any one of claims 1 to 5;
Learning to construct a discrimination model that discriminates one individual contained in a group of individuals to be identified from other individuals by performing machine learning so as to identify individuals contained in images using the acquired learning data. A program characterized by executing a step. an image acquisition unit that acquires an image containing an individual;
an identification unit that outputs individual identification information of an individual contained in the image by inputting data of the acquired image as input data to the identification model constructed by the learning device according to claim 9. identification device. A program for causing a computer to function as the individual identification device according to claim 11,
an image acquisition step of acquiring an image containing the individual;
and inputting data of the obtained image as input data to the identification model constructed by the learning device according to claim 9, and outputting individual identification information of the individual contained in the image. A program characterized by: A recording medium recording the program according to claim 6, 8, 10 or 12. an image acquisition step of acquiring an image containing the individual;
a moving body detection step of detecting a moving body in the acquired image;
a framing step of framing a body region including the entire body of the individual in the detected moving body;
an extracting step of extracting a body region image of the body region to which a frame is attached;
and a generating step of generating learning data for a discrimination model for discriminating one individual contained in a group of individuals to be identified from other individuals from the extracted body region image. an image acquisition step of acquiring an image containing the individual;
15. Using the learning data generated by the learning data generation method according to claim 14, one of the individuals included in the identification target population constructed by machine learning so as to identify the individuals included in the image an identification step of outputting individual identification information of an individual contained in the image by inputting the data of the acquired image as input data to a identification model that identifies the individual from other individuals;
an identification image acquisition step of acquiring time-series images of the identified individual;
and an analysis step of analyzing the behavior of the individual with reference to the acquired image. an image acquisition unit that acquires an image containing an individual;
a detection unit that detects the individual in the acquired image;
a body region identifying unit that identifies a body region including the entire body of the detected individual;
an extraction unit that extracts a body region image of the identified body region;
A learning data generation device, comprising: a generation unit that generates learning data for a discrimination model that discriminates one individual included in a group of individuals to be identified from other individuals from the extracted body region image. The detection unit inputs the image acquired by the image acquisition unit as input data to a detection model machine-learned to detect the individual in the image using an image including the individual as input data, and outputs the image. 17. The learning data generation device according to claim 16, which detects the individual who is being tested. 18. The learning data generation device according to claim 16, wherein the body region identifying unit identifies a body region including the entire body of the individual detected by the detecting unit by image segmentation. The body region identifying unit uses image data of the detected individual as input data for a posture estimation model that performs machine learning to detect the skeleton of the individual from an image representing a part of the body of the individual and to estimate the posture of the individual. 19. The learning data generation device according to any one of claims 16 to 18, wherein a body region representing the entire body of the individual is identified by referring to the posture of the individual that is output by the input. The learning data generation device according to any one of claims 16 to 19, wherein the generation unit classifies and accumulates the body region images for each same individual to generate the learning data. an image acquisition unit that acquires an image containing an individual;
a detection unit that detects the individual in the acquired image;
Using learning data generated by the learning data generation device according to any one of claims 1 to 5 and 16 to 20, machine learning is performed to identify individuals included in images. inputting the obtained data of the image as input data to a discrimination model that distinguishes one individual contained in the population to be identified from other individuals, thereby distinguishing the individual contained in the image from other individuals an identification unit that
a position specifying unit that specifies the position of the body part of the identified individual;
an information acquisition unit that acquires time-series information representing changes in the identified position within a predetermined time period;
and a behavioral analysis unit that analyzes the behavioral pattern of the individual by referring to the acquired time-series information. The position specifying unit uses an image including an individual as input data, and applies the image acquired by the image acquisition unit to a position specifying model that has undergone machine learning so as to specify the position of the body part of the individual in the image. 22. The behavioral analysis device according to claim 21, wherein the output position is specified by inputting as input data. 23. The behavioral analysis apparatus according to claim 21, wherein said position specifying unit acquires the coordinates of the position of the body part of said individual. The behavioral analysis unit inputs the time-series information acquired by the information acquisition unit to an analysis model that performs machine learning so as to estimate a behavioral pattern corresponding to the time-series information as input data, using the time-series information as input data. 24. The behavioral analysis apparatus according to any one of claims 21 to 23, which acquires as an analysis result a behavioral pattern output by inputting as . The position specifying unit inputs the detected image data of the individual as input data to a posture estimation model that is machine-learned so as to detect the skeleton of the individual from an image representing a part of the body of the individual and estimate the posture of the individual. 25. The behavioral analysis apparatus according to any one of claims 21 to 24, wherein the position is specified by referring to the posture of the individual output by doing.

Images