JP2022074009A - Behavior trajectory estimation device, behavior trajectory estimation program, and behavior trajectory estimation method - Google Patents

Abstract

To provide a behavior trajectory estimation device, a behavior trajectory estimation program and a behavior trajectory estimation method that highly accurately identify a moving body for each individual while achieving wide area monitoring of a lot of moving bodies, and can correctly estimate a behavior trajectory for each individual.SOLUTION: A behavior trajectory estimation device 1, which can estimate a behavior trajectory of a moving body for each individual, has: an individual area detection unit 53 that detects an individual area within time-sequence monitoring images; a trace processing implementation unit 54 that traces the moving body for each individual; a trajectory fragment acquisition unit 55 that acquires a trajectory fragment of the individual area; an individual characteristic extraction unit 57 that extracts an individual characteristic from the individual area; an individual characteristic collation unit 58 that collates the individual characteristic with an individual characteristic database 42, and acquires an individual ID corresponding to the individual characteristic; and a trajectory fragment integration unit 59 that integrates the trajectory fragments, and outputs the integrated trajectory fragment as the behavior trajectory for each individual ID.SELECTED DRAWING: Figure 2

Description

The present invention relates to a behavior trajectory estimation device, a behavior trajectory estimation program, and a behavior trajectory estimation method suitable for wide-area monitoring of moving objects such as animals and humans with a plurality of surveillance cameras.

Conventionally, as a method of monitoring an animal such as a dairy cow, a method of directly monitoring by a human, a method of attaching a sensor to a livestock for monitoring, a method of monitoring with a surveillance camera, and the like are known. For example, Japanese Patent Application Laid-Open No. 2019-24482 describes object information (cow contour information, etc.) detected from spatial information (infrared image, etc.) and cow placement information (individual identification information, etc.) in contact with each other. A technique for determining abnormal behavior of a cow based on the behavior information of the cow (walking distance, standing / sitting / lying time, number of contacts, etc.) generated from the above has been proposed (Patent Document 1).

Japanese Unexamined Patent Publication No. 2019-24482

However, in recent years, the scale of dairy farming has been increasing, and the mainstream method is to raise hundreds or more cows in a large barn while freely moving them, such as the free stall barn and the free burn barn. There is. For this reason, it is difficult to sufficiently monitor all cows without confusing them and manage their physical condition, etc. by the method of direct monitoring by humans.

In addition, in the method of attaching sensors to livestock for monitoring, it is necessary to prepare sensors for all the animals, so the larger the barn, the higher the cost. In addition, every time the sensor falls off, is damaged, or the battery runs out, it is necessary to find and replace the cow, which is troublesome and costly for maintenance. In addition, cows have a habit of swallowing anything, so there is a risk of swallowing a dropped sensor or battery with food.

Further, regarding the method of monitoring by a surveillance camera, there is a problem that a system for monitoring a small number of cows in a narrow cow cell, as in Patent Document 1, cannot support wide area monitoring of a large number of cows. In addition, if a surveillance camera equipped with a wide-angle lens is used to realize wide-area surveillance, the appearance will change significantly depending on the position of the cow and the posture of the cow in the image. For this reason, it becomes difficult to identify the cow for each individual from the wide-angle image, and there is a problem that the behavioral trajectory of each individual cannot be estimated correctly.

The present invention has been made to solve such a problem, and while realizing wide-area monitoring of a large number of moving objects, the moving objects are identified for each individual with high accuracy, and the behavioral trajectory of each individual is achieved. It is an object of the present invention to provide a behavior locus estimation device, a behavior locus estimation program, and a behavior locus estimation method capable of accurately estimating.

The behavior trajectory estimation device according to the present invention solves the problem of identifying a moving object for each individual with high accuracy and correctly estimating the behavior trajectory for each individual while realizing wide area monitoring of a large number of moving objects. , A behavior trajectory estimation device that can estimate the behavior trajectory of a moving object for each individual, and detects the image area occupied by the individual as an individual area in each surveillance image of the time series taken by a plurality of surveillance cameras. A tracking process execution unit that tracks the moving object for each individual by associating an individual area detection unit and an individual region having a similar position, movement direction, and speed between adjacent surveillance images in time series, and the tracking process execution unit. A locus fragment acquisition unit that acquires a time-series position coordinate group in a predetermined coordinate system of an individual region associated with the unit as a locus fragment, and an individual feature that is a feature vector based on the appearance of the individual is extracted from the individual region. The individual characteristics extracted by the individual characteristics extraction unit, the individual characteristics extracted by the individual characteristics extraction unit, and the individual characteristics database in which the individual characteristics are registered in association with the individual ID that identifies the individual are collated and extracted. An individual characteristic collation unit that acquires an individual ID corresponding to an individual characteristic, and a locus fragment integration unit that integrates the locus fragments having the highest continuity between the locus fragments and outputs them as an action locus for each individual ID. Have.

Further, as one aspect of the present invention, there is a moving body posture estimation unit that estimates the posture of the moving body included in the individual region in order to solve the problem of suppressing the deterioration of the identification accuracy for each individual. The individual feature extraction unit may extract the individual feature only from the individual region including the moving body in a posture suitable for identifying the individual.

Further, as one aspect of the present invention, in order to solve the problem of easily estimating the posture of the moving body with high accuracy, a posture estimable region in which the posture of the moving body can be estimated is provided for each surveillance camera. A posture that is often taken by a moving body existing in the posture estimable region is set, and the moving body posture estimation unit is in the posture estimable region when the individual region is included in the posture estimable region. The posture associated with the above may be estimated as the posture of the moving body included in the individual region.

Further, as one aspect of the present invention, in order to solve the problem of improving the identification accuracy of an individual, a moving body image is obtained by cutting out the individual region from the monitoring image for each individual ID based on the action locus. It may have an individual feature addition part that acquires a column, extracts an individual feature of each moving object image, and adds it to the individual feature database.

Further, as one aspect of the present invention, in order to solve the problem of estimating the action locus of each individual with high accuracy, the locus fragment integration unit has temporal continuity and spatial continuity between the locus fragments. The locus fragment with the highest continuity between the locus fragments may be identified based on the similarity between sex and individual characteristics.

Further, as one aspect of the present invention, in order to solve the problem of preventing erroneous integration of locus fragments of different individuals even among individuals having similar appearances, the locus fragment integration unit is used. If the individual IDs associated with the locus fragments having the highest continuity between the locus fragments are different, the locus fragments may not be integrated.

The behavior trajectory estimation program according to the present invention solves the problem of identifying a moving object for each individual with high accuracy and correctly estimating the behavior trajectory of each individual while realizing wide-area monitoring of a large number of moving objects. , A behavior trajectory estimation program that can estimate the behavior trajectory of a moving object for each individual, and detects the image area occupied by the individual as an individual area in each surveillance image of the time series taken by a plurality of surveillance cameras. A tracking process execution unit that tracks the moving object for each individual by associating an individual area detection unit and an individual region having a similar position, movement direction, and speed between adjacent surveillance images in time series, and the tracking process execution unit. A locus fragment acquisition unit that acquires a time-series position coordinate group in a predetermined coordinate system of an individual region associated with the unit as a locus fragment, and an individual feature that is a feature vector based on the appearance of the individual is extracted from the individual region. The individual characteristics extracted by the individual characteristics extraction unit, the individual characteristics extracted by the individual characteristics extraction unit, and the individual characteristics database in which the individual characteristics are registered in association with the individual ID that identifies the individual are collated and extracted. An individual characteristic collation unit that acquires an individual ID corresponding to an individual characteristic, and a locus fragment integration unit that integrates the locus fragments having the highest continuity between the locus fragments and outputs them as an action locus for each individual ID. To make the computer work.

The behavior trajectory estimation method according to the present invention solves the problem of identifying a moving object for each individual with high accuracy and correctly estimating the behavior trajectory for each individual while realizing wide-area monitoring of a large number of moving objects. , A behavior trajectory estimation method that can estimate the behavior trajectory of a moving object for each individual, and detects the image area occupied by the individual as an individual area in each surveillance image of the time series taken by a plurality of surveillance cameras. A tracking process execution step for tracking the moving object for each individual and a tracking process execution for each individual by associating the individual area detection step with individual areas having similar positions, movement directions, and speeds between adjacent surveillance images in time series. A locus fragment acquisition unit that acquires a time-series position coordinate group in a predetermined coordinate system of an individual region associated with a step as a locus fragment, and an individual feature that is a feature vector based on the appearance of the individual is extracted from the individual region. The individual feature extraction step, the individual feature extracted in the individual feature extraction step, and the individual feature database in which the individual feature is registered in association with the individual ID that identifies the individual are collated and extracted. An individual characteristic collation step for acquiring an individual ID corresponding to an individual characteristic, and a locus fragment integration step for integrating the locus fragments having the highest continuity between the locus fragments and outputting them as an action locus for each individual ID. Have.

The behavior trajectory estimation device according to the present invention identifies a moving object for each individual with high accuracy while realizing wide area monitoring of a large number of moving objects, correctly estimates the behavior trajectory for each individual, and relates to the estimation calculation. In order to solve the problem of improving processing efficiency, it is a behavior trajectory estimation device that can estimate the behavior trajectory of a moving object for each individual, and in each surveillance image of the time series taken by a plurality of surveillance cameras. Of the entire image consisting of the individual area detection unit that detects the image area occupied by the individual as the individual area and all the monitoring images taken at the same time, the position, movement direction, and position / movement direction between the adjacent overall images in time series. By associating individual areas with similar speeds, a tracking process execution unit that tracks the moving object for each individual and a time-series position coordinate group in a predetermined coordinate system of the individual area associated with the tracking process execution unit A locus fragment acquisition unit acquired as a locus fragment, an individual feature extraction unit that extracts an individual feature that is a feature vector based on the appearance of the individual from the individual region, an individual feature extracted by the individual feature extraction unit, and the above-mentioned individual characteristics. The individual characteristic collating unit that collates with the individual characteristic database in which the individual characteristics are registered in association with the individual ID that identifies the individual and acquires the individual ID corresponding to the extracted individual characteristics, and the locus fragment are described above. It has an action locus output unit that outputs as an action locus for each individual ID.

The behavior trajectory estimation program according to the present invention identifies a moving object for each individual with high accuracy while realizing wide-area monitoring of a large number of moving objects, correctly estimates the behavior trajectory for each individual, and relates to the estimation calculation. In order to solve the problem of improving processing efficiency, it is a behavior trajectory estimation program that can estimate the behavior trajectory of a moving body for each individual, and in each surveillance image of the time series taken by multiple surveillance cameras. Of the entire image consisting of the individual area detection unit that detects the image area occupied by the individual as the individual area and all the monitoring images taken at the same time, the position, movement direction, and position / movement direction between the adjacent overall images in time series. By associating individual regions with similar speeds, a tracking process execution unit that tracks the moving body for each individual and a time-series position coordinate group in a predetermined coordinate system of the individual region associated with the tracking processing execution unit A locus fragment acquisition unit acquired as a locus fragment, an individual feature extraction unit that extracts an individual feature that is a feature vector based on the appearance of the individual from the individual region, an individual feature extracted by the individual feature extraction unit, and the above-mentioned individual characteristics. The individual characteristic collating unit that collates with the individual characteristic database in which the individual characteristics are registered in association with the individual ID that identifies the individual and acquires the individual ID corresponding to the extracted individual characteristics, and the locus fragment are described above. The computer functions as an action locus output unit that outputs an action locus for each individual ID.

The behavior trajectory estimation method according to the present invention identifies a moving object for each individual with high accuracy while realizing wide-area monitoring of a large number of moving objects, correctly estimates the behavior trajectory for each individual, and relates to the estimation calculation. In order to solve the problem of improving processing efficiency, it is a behavior trajectory estimation method that can estimate the behavior trajectory of a moving object for each individual, and in each surveillance image of the time series taken by a plurality of surveillance cameras. Of the whole image consisting of the individual area detection step of detecting the image area occupied by the individual as the individual area and all the monitoring images taken at the same time, the position, the movement direction, and the position / movement direction / By associating individual regions with similar speeds, a tracking process execution step for tracking the moving object for each individual and a time-series position coordinate group in a predetermined coordinate system of the individual region associated with the tracking process execution step are The locus fragment acquisition unit acquired as a locus fragment, an individual feature extraction step for extracting an individual feature which is a feature vector based on the appearance of the individual from the individual region, an individual feature extracted in the individual feature extraction step, and the above-mentioned The individual characteristic collation step of collating with the individual characteristic database in which the individual characteristics are registered in association with the individual ID for identifying the individual and acquiring the individual ID corresponding to the extracted individual characteristics, and the locus fragment are described above. It has an action locus output step that outputs as an action locus for each individual ID.

According to the present invention, while realizing wide-area monitoring of a large number of moving objects, it is possible to identify the moving objects for each individual with high accuracy and correctly estimate the behavioral trajectory of each individual.

It is a figure which shows an example of the dairy cow (moving body) monitored by using the action locus estimation apparatus, the action locus estimation program and the action locus estimation method which concerns on this invention, and the barn. It is a block diagram which shows 1st Embodiment of the action locus estimation apparatus which concerns on this invention. It is a figure which shows an example of the data stored in the individual characteristic database of this 1st Embodiment. It is a figure which shows an example of the data stored in the locus fragment buffer of this 1st Embodiment. This is an example of a monitoring image showing an individual area of the first embodiment. It is a figure which shows an example of the data stored in the posture estimation database of this 1st Embodiment. It is a figure which shows an example of the data stored in the action locus database of this 1st Embodiment. This is an example of a device for capturing an image group required when registering individual characteristics of the first embodiment. It is a schematic diagram of an image of a dairy cow taken from different directions. It is a figure which shows (a) the tracking process by the tracking process execution part, and (b) the locus fragment for each individual in this 1st Embodiment. It is a schematic diagram of an image coordinate system (two-dimensional space). It is a schematic diagram of a barn coordinate system (three-dimensional space). It is a figure which shows an example of the locus fragment integrated by the locus fragment integration part of this 1st Embodiment. It is a figure which shows an example of the action locus estimated by the action locus estimation apparatus of this 1st Embodiment. It is a flowchart which shows the individual characteristic registration process of this 1st Embodiment. It is a flowchart which shows the action locus estimation process of this 1st Embodiment. It is a flowchart which shows the posture estimation process of this 1st Embodiment. It is a flowchart which shows the individual characteristic addition processing of this 1st Embodiment. It is a block diagram which shows the 2nd Embodiment of the action locus estimation apparatus which concerns on this invention. It is a figure which shows an example of the data which is stored in the locus fragment buffer of this 2nd Embodiment. It is a figure which shows an example of the data stored in the action locus database of this 2nd Embodiment. In the second embodiment, it is a figure which shows how the individual area in an image coordinate system is converted into the individual area in a barn coordinate system. It is a figure which shows (a) the individual area detected in each monitoring image, (b) the individual area detected in the whole image, and (c) the locus fragment for each individual in this 2nd Embodiment. It is a flowchart which shows the action locus estimation process of this 2nd Embodiment.

Hereinafter, embodiments of the action trajectory estimation device, the action trajectory estimation program, and the action trajectory estimation method according to the present invention will be described with reference to the drawings.

In the first embodiment, as shown in FIG. 1, a large number of dairy cows raised in a large barn, such as a free stall barn and a free burn barn, are monitored by a plurality of surveillance cameras 11, and each surveillance camera 11 is used. The case where the behavior locus of each individual is estimated by using the behavior locus estimation device 1, the behavior locus estimation program 1b, and the behavior locus estimation method of the first embodiment will be described as an example. In the first embodiment, a dairy cow is assumed as a moving object to be monitored, but the present invention is not limited to this, and all objects that move like a car, etc., in addition to other animals and humans, are books. It is included in the moving body according to the invention.

In the first embodiment, the action locus estimation device 1 is composed of a computer such as a personal computer, a smartphone or a tablet, and as shown in FIG. 2, mainly a display means 2, an input means 3, and a storage means. 4 and arithmetic processing means 5. Further, the above-mentioned plurality of surveillance cameras 11 are connected to the action locus estimation device 1. Hereinafter, each configuration means will be described in detail.

The surveillance camera 11 captures a moving object to be monitored. In the first embodiment, as shown in FIG. 1, a plurality of surveillance cameras 11 are fixed to the ceiling of the barn, and time-series surveillance images are taken at intervals of about one every few seconds. There is. Further, in the first embodiment, the surveillance camera 11 is configured by a moving image camera that outputs one frame of a moving image as a surveillance image, but may be a still image camera that intermittently captures the surveillance image.

The number of surveillance cameras 11 needs to be adjusted according to the size of the barn to be monitored and the height of the ceiling. Further, the surveillance camera 11 may be a near-infrared camera in order to increase the sensitivity during nighttime photography. Further, the connection between the action trajectory estimation device 1 and the surveillance camera 11 is not limited to the bus connection, but is connected by a wired network such as Ethernet (registered trademark) or a wireless network such as Wi-Fi (registered trademark). You may be.

The display means 2 is composed of a liquid crystal display or the like, and displays the action locus or the like estimated by the action locus estimation device 1. The input means 3 is composed of a keyboard, a mouse, or the like, and inputs instructions and selections by the user. In addition, instead of these display means 2 and input means 3, it may have a display input means such as a touch panel having both a display function and an input function.

The storage means 4 stores various data and functions as a working area when the arithmetic processing means 5 performs arithmetic processing. In the first embodiment, the storage means 4 is composed of a hard disk, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like, and as shown in FIG. 2, the program storage unit 41 and the storage means 4 It has an individual characteristic database 42, a locus fragment buffer 43, a posture estimation database 44, and an action locus database 45. Hereinafter, each component will be described in more detail.

The program storage unit 41 has an individual feature registration program for executing each of the three functions of the behavior trajectory estimation device 1 of the first embodiment, that is, the individual feature registration function, the behavior trajectory estimation function, and the individual feature addition function. 1a, the behavior trajectory estimation program 1b and the individual feature addition program 1c are installed. Then, the arithmetic processing means 5 executes each program to make the computer as the action locus estimation device 1 function as each component described later.

The usage patterns of the individual feature registration program 1a, the action trajectory estimation program 1b, and the individual feature addition program 1c are not limited to the above configuration. For example, each program may be stored in a non-temporary recording medium that can be read by a computer, such as a CD-ROM or a DVD-ROM, and read directly from the recording medium for execution. Further, it may be used by a cloud computing method or an ASP (Application Service Provider) method from an external server or the like.

The individual characteristic database 42 registers in advance individual characteristics, which are characteristic vectors based on the appearance of the individual, for each moving object to be monitored. In the first embodiment, as shown in FIG. 3, the individual characteristic database 42 contains the individual characteristics extracted by the individual characteristic registration unit 51, which will be described later, in association with the individual ID that identifies the individual, and the individual characteristics. The image file name used for the extraction of is stored.

In the individual feature database 42, since the individual feature matching unit 58, which will be described later, needs to search for the nearest individual feature at high speed, indexes for nearest neighbor search (cartesian quantization, hash index, kd tree index, etc.) ) Is preferably created. Further, the individual characteristic database 42 may have a function of deleting or updating individual characteristics when the individual is replaced or the like.

The locus fragment buffer 43 buffers and stores locus fragments constituting a part of the action locus of each individual for a predetermined time (several hours to one day). In the first embodiment, as shown in FIG. 4, the locus fragment buffer 43 acquires the camera number of the surveillance camera 11 that acquired the locus fragment and the locus fragment in association with the locus fragment ID that identifies the locus fragment. The time, the individual area (two-dimensional coordinate value in the image coordinate system), the position of the dairy cow (two-dimensional coordinate value in the barn coordinate system), the individual feature (feature vector), and the individual ID, which is the image area occupied by the individual. And are stored.

In the first embodiment, the individual region is detected by the individual region detection unit 53, which will be described later, and is detected as a rectangular image region occupied by the individual in the surveillance image, as shown in FIG. .. Then, the position coordinates (two-dimensional coordinate values in the image coordinate system) of the two diagonal points of the image area are stored as the individual area. Further, as will be described later, the position of the dairy cow is obtained by converting the center point of the individual region in the image coordinate system into a two-dimensional point in the barn coordinate system. However, the locus fragment is not limited to the above, and may include at least a time-series position coordinate group of an individual region in a predetermined coordinate system.

The posture estimation database 44 stores data used when the moving body posture estimation unit 56, which will be described later, estimates the posture. In the first embodiment, as shown in FIG. 6, the posture estimation database 44 has a posture estimable region (image) in which the posture of the moving body can be estimated in association with the camera number of the surveillance camera 11 used for monitoring. (Two-dimensional coordinate values of the coordinate system) and the posture that a moving body existing in the posture estimable region often takes are set.

The posture estimable area is set based on the layout of the barn. Specifically, to explain the monitoring image of FIG. 5 as an example, a bed is arranged in the upper half of the image, and a lying state (a posture in which a limb is thrown out and lying down) or a prone state (a knee is lowered and a chest is raised). The dairy cow in the posture) is shown. On the other hand, a feeding area is located in the lower half of the image, showing a foraging dairy cow eating feed while standing. In addition, a passage is located in the center of the image, showing a walking cow. Since the layout of the barn is fixed in this way and the surveillance camera 11 is also fixed, a region that often takes a posture suitable for identifying an individual in advance is set as a posture estimation ability region for each surveillance camera 11. can do.

The action locus database 45 stores the action locus for each individual ID output from the locus fragment integration unit 59, which will be described later. In the first embodiment, as shown in FIG. 7, in the action locus database 45, the camera number of the surveillance camera 11, the shooting time, and the individual area (image) are associated with the action locus ID that identifies the action locus. The two-dimensional coordinate value in the coordinate system), the position of the cow (two-dimensional coordinate value in the barn coordinate system), the individual feature (feature vector), and the individual ID are stored. That is, the individual ID stored in the locus fragment buffer 43 is a temporary individual ID before the locus fragments are integrated, but when each locus fragment is integrated by the locus fragment integration unit 59 described later, the final locus fragment is finalized. It is registered in the action locus database 45 as an individual ID.

Next, the arithmetic processing means 5 is composed of a CPU (Central Processing Unit) or the like, and by executing each program installed in the storage means 4, the action locus estimation device 1 is made to exert various functions. Is. Specifically, as shown in FIG. 2, the arithmetic processing means 5 functions as an individual feature registration unit 51 that exerts an individual feature registration function by executing the individual feature registration program 1a.

Further, the arithmetic processing means 5 executes the action locus estimation program 1b to exert the action locus estimation function, the monitoring image acquisition unit 52, the individual area detection unit 53, the tracking processing execution unit 54, and the locus fragment acquisition. It functions as a unit 55, a moving body posture estimation unit 56, an individual feature extraction unit 57, an individual feature matching unit 58, and a locus fragment integration unit 59. Further, the arithmetic processing means 5 functions as an individual feature addition unit 60 that exerts an individual feature addition function by executing the individual feature addition program 1c. Hereinafter, the components of each function will be described in more detail.

(1) Individual feature registration function The individual feature registration function is a function for pre-registering individual features, which are feature vectors based on the appearance of an individual, in the individual feature database 42 in association with the individual ID, and is performed by the individual feature registration unit 51. It works.

In the first embodiment, the individual feature registration unit 51 first acquires an image group of a dairy cow to be newly registered by using a still image camera, a moving image camera, or the like. For example, by fixing a plurality of cameras to the shooting frame as shown in FIG. 8 and installing sensors such as a passage sensor and a proximity sensor to control the shutter to be released at the timing when the cow passes, the whole body of the cow can be affected. Images taken from multiple directions are acquired at the timing when they fit on the screen.

The camera used when registering the individual characteristics is not a general RGB camera, but a depth camera capable of acquiring depth information can also be used. A three-dimensional model of a dairy cow is created from an RGB image with depth data acquired by a depth camera. Further, by rendering an image of the three-dimensional model taken from a plurality of directions with computer graphics, multi-viewpoint image data of a dairy cow taken from multiple directions is generated. Then, by extracting individual features from the multi-viewpoint image data and registering them in the individual feature database 42, the identification accuracy of the individual can be improved.

Next, the individual characteristic registration unit 51 acquires an individual ID for identifying an individual. In the first embodiment, the individual feature registration unit 51 acquires an individual ID assigned to a dairy cow's collar or ear tag by reading it using a wireless sensor or a camera. Alternatively, a human may visually confirm the individual ID given to the ear tag or the like, and the individual characteristic registration unit 51 may acquire the individual ID input from the input means 3.

Subsequently, the individual feature registration unit 51 calculates a feature vector representing the appearance such as a dairy cow's mottled pattern, coat color, and body shape from a group of images taken by a camera, and extracts it as an individual feature. The method for extracting individual characteristics is not particularly limited, but a vector having real values as elements from an input image using a machine learning model such as a convolutional neural network (CNN) (for example, 2048 dimensions). May be calculated. By optimizing the weight parameter of the CNN filter by learning in advance, the accuracy of individual identification can be improved. The weighting parameters are adjusted to minimize cross-entropy loss, triplet loss, etc. for the teacher data.

Here, FIG. 9 shows a schematic diagram of an image used when registering individual characteristics. The image of the camera 1 in which the dairy cow is photographed from directly above to the bottom shows a mottled pattern on the back. On the other hand, in the image of the camera 2 taken from the upper rear of the dairy cow, the mottled pattern on the buttocks is shown. There are some spots that are common to the images of the cameras 1 and 2, but there are also spots that are visible only in one camera. In addition, the appearance of the same mottled pattern changes depending on the shooting direction. Therefore, when extracting individual characteristics, it is desirable to use multi-viewpoint images taken from multiple directions in order to create a database of mottled patterns of the whole body of dairy cows.

It is difficult to photograph the whole body of a dairy cow with one camera at a time. Therefore, it can be expected that the accuracy of individual identification will be improved by inputting multi-viewpoint images generated from a plurality of cameras and quantifying the mottled pattern. Further, when extracting individual features from a plurality of images, a feature vector may be calculated using CNN for all the images, and the average or maximum value of each element of each feature vector may be used as the feature vector.

Finally, the individual feature registration unit 51 extracts individual features from all the input images and registers them in the individual feature database 42 in association with the individual ID. Further, the individual characteristic registration unit 51 extracts individual characteristics of all the dairy cows to be monitored and registers them in the individual characteristic database 42 in association with the individual ID.

(2) Behavior trajectory estimation function The behavior trajectory estimation function is a function for identifying a dairy cow for each individual from the surveillance image of the surveillance camera 11 and estimating the behavior trajectory for each individual, and the surveillance image acquisition unit 52 and the individual area detection. It is functioned by a unit 53, a tracking process execution unit 54, a locus fragment acquisition unit 55, a moving body posture estimation unit 56, an individual feature extraction unit 57, an individual feature matching unit 58, and a locus fragment integration unit 59. ..

The surveillance image acquisition unit 52 acquires time-series surveillance images from a plurality of surveillance cameras 11. In the first embodiment, the surveillance image acquisition unit 52 acquires time-series surveillance images from each surveillance camera 11 at intervals of about once every few seconds, and the camera number for identifying each surveillance camera 11 and the shooting time. Is designed to be associated with each other.

The individual area detection unit 53 detects an image area occupied by an individual as an individual area in each of the time-series surveillance images taken by a plurality of surveillance cameras 11. In the first embodiment, as shown in FIG. 5, the individual area detection unit 53 detects the dairy cow shown in the monitoring image acquired by the monitoring image acquisition unit 52 for each individual, and a rectangle circumscribing each individual. The image area of is detected as an individual area.

In the first embodiment, the individual region detection unit 53 detects an individual region using a deep learning-based object detector such as Faster R-CNN or Mask R-CNN, but is limited to this. Instead, it may be detected by other image processing or the like. Further, in the first embodiment, the individual region is represented as a rectangular shape, but the shape is not limited to this shape, and may be represented by a polygon, a circle, or the like.

The tracking process execution unit 54 tracks the moving object in the surveillance image for each individual. In the first embodiment, the tracking process execution unit 54 uses, for example, a moving object tracking algorithm called SORT (Simple Online and Realtime Tracking), and each of the individual regions detected by the individual region detection unit 53 is time-series. In, tracking is performed by associating individual areas having similar positions, moving directions, and velocities between the monitoring images (frames) before and after.

Specifically, as shown in FIG. 10A, in the time-series surveillance image, the individual region group detected from the surveillance image at time t is {B (t) 1, B (t) 2, B ( Let t) 3,… B (t) n _t }, and the individual region group detected from the monitoring image at time t-1 is {B (t-1) 1, B (t-1) 2, B (t-). 1) 3,… B (t-1) n _t-1 }. In this case, the tracking process execution unit 54 associates the individual regions between the frames at time t and time t-1, so that a locus fragment for each individual as shown in FIG. 10 (b) can be obtained.

For example, if the dairy cows contained in the surveillance image have hardly moved, the same number of individual regions are detected (that is, n _t = n _t-1 ), so all individual regions are associated with each other on a one-to-one basis. Can be (all cows can be tracked). On the other hand, if a new cow enters the surveillance image at time t, or if the cow appears from the shadow of an obstacle, it is detected as a new individual area that cannot be associated.

Then, when the tracking process execution unit 54 cannot associate the individual area (when the individual area corresponding to the individual area B (t) m does not exist in the monitoring image at time t-1), the monitoring image at time t It is determined that a new individual area has appeared in the area, and a new locus fragment ID is assigned and associated with it. On the other hand, when the tracking process execution unit 54 succeeds in associating the individual regions (when the individual regions B (t) m and the individual regions B (t-1) n correspond to each other), the locus fragment buffer 43 to the individual regions B (t-1) Acquire the locus fragment ID corresponding to n.

In the first embodiment, the tracking process execution unit 54 uses SORT (Simple Online and Realtime Tracking) as a moving object tracking algorithm, but the present invention is not limited to this, and other moving object tracking is performed. An algorithm may be used.

The locus fragment acquisition unit 55 acquires a time-series position coordinate group in a predetermined coordinate system of an individual region associated with the tracking process execution unit 54 as a locus fragment. In the first embodiment, the locus fragment acquisition unit 55 specifies a center point for each of the individual regions to which the same locus fragment ID is given by the tracking process execution unit 54, and the locus fragment acquisition unit 55 is two-dimensional in the image coordinate system of the center point. Calculate the coordinate values. Then, the two-dimensional coordinate value is converted into a three-dimensional coordinate value in the barn coordinate system, and the two-dimensional coordinate value excluding the Z value (height) is acquired as a locus fragment. Further, the locus fragment acquisition unit 55 associates the acquired locus fragment with the locus fragment ID and stores it in the locus fragment buffer 43.

Here, a method of obtaining the three-dimensional point P = (X, Y, Z) of the barn coordinate system from the two-dimensional point p = (x, y) of the image coordinate system will be described. FIG. 11 is a schematic diagram of an image coordinate system (two-dimensional space), and corresponds to an image taken by a camera installed on the ceiling of the barn, with the upper left corner as the origin. Further, FIG. 12 is a schematic diagram of the barn coordinate system (three-dimensional space), in which one point in the barn is the origin O, the vertical upward direction is Z, and the floor surface of the barn is the XY plane.

Using the pinhole camera model (perspective projection model), the correspondence between the two-dimensional point p = (x, y) in the image coordinate system and the three-dimensional point P = (X, Y, Z) in the barn coordinate system is as follows. It is expressed by the equations (1) and (2) of.

Since the camera parameters change depending on the camera model, settings, camera installation position, angle, etc., they are obtained by calibration processing when the camera is installed. As the camera calibration method, a general method called the Tsai or Zhang method can be used. Although the lens distortion is not considered in the equations (1) and (2), a camera model including the lens distortion parameter can also be used.

As shown in FIGS. 11 and 12, the point on the back of the cow is the three-dimensional point P ₁ = (X ₁ , Y ₁ , Z ₁ ) in the barn coordinate system and the two-dimensional point p ₁ = (x) in the image coordinate system. ₁ , y ₁ ). Assuming that the height (height from the floor to the back) H of the dairy cow is known, substituting it into the equation (2) and rearranging it gives the following equation (3).

Since the two-dimensional point p ₁ = (x ₁ , y ₁ ) in the image coordinate system is acquired as the center point of the individual region, for example, by solving the simultaneous linear equations in Eq. (3), the barn coordinate system The three-dimensional point P ₁ = (X ₁ , Y ₁ , Z ₁ ) is obtained. The height H of the dairy cow may be regarded as constant regardless of the individual, or may be estimated from the height or the like.

In the image coordinate system, it seems that the cow is walking slowly as it goes away from the camera, so even if it walks at a constant speed, it is not judged as a constant speed movement. On the other hand, in the barn coordinate system, if the dairy cow is moving at a constant speed, it can be determined that the cow is moving at a constant speed. Therefore, in the individual tracking process, it is preferable to use the barn coordinate system instead of the image coordinate system because the movement of the dairy cow can be accurately determined.

The moving body posture estimation unit 56 estimates the posture of the moving body included in the individual area. In the first embodiment, the moving body posture estimation unit 56 estimates the posture (standing, lying down, head orientation, etc.) of the dairy cow for each of the individual regions detected by the individual region detection unit 53, and identifies the individual. Determine if the posture is suitable for doing. Then, the individual feature extraction unit 57 extracts the individual feature only when the posture is suitable for identifying the individual.

In the first embodiment, as described above, the posture estimation ability region is set in the posture estimation database 44 in advance based on the barn layout. Therefore, the moving body posture estimation unit 56 first acquires the individual region detected by the individual region detection unit 53, and also acquires the camera number of the surveillance camera 11 that has captured the surveillance image including the individual region.

Next, the moving body posture estimation unit 56 refers to the posture estimation database 44, and acquires a posture estimable region and a posture set in association with the acquired camera number. Then, it is determined whether or not the acquired individual region is included in the posture estimable region, and if it is included, the posture associated with the posture estimable region is estimated as the posture of the moving body included in the individual region. Output. On the other hand, if the acquired individual area is not included in the posture estimable area, the posture cannot be estimated and the posture is not output.

In the first embodiment, if the output posture is in the standing state, it is determined that the individual is in a posture that can be identified, and the individual feature extraction unit 57 extracts the individual feature. On the other hand, if the output posture is in a lying state, or if the posture cannot be estimated, it is determined that the individual is not in a identifiable posture, and processing of other individual regions is performed without extracting individual characteristics. It is becoming a transition to.

In the first embodiment, the moving body posture estimation unit 56 considers only the inclusion relationship between the individual region and the posture estimable region. However, the condition is not limited to this condition, and the aspect ratio and area of the individual area, the position of the individual area in the immediately preceding monitoring image (that is, the moving direction and the amount of movement of the dairy cow), the overlap rate of different individual areas, etc. are the conditions. By adding it, the estimation accuracy of the posture can be improved. For example, when the overlap rate of individual regions is large, that is, when the overlap between adjacent individual regions is large, the dairy cows are in close contact with each other, so that individual identification may not be possible.

Further, in the first embodiment, when the individual region is not included in the posture estimable region, the individual characteristics for identifying the individual are not extracted. However, the present invention is not limited to such processing, and even when the individual region is not included in the posture estimable region, the individual characteristics are extracted and the individual ID corresponding to the individual characteristics has low reliability. It may be treated as data.

Further, in the first embodiment, the moving body posture estimation unit 56 estimates the posture of the moving body using a posture estimation ability region preset based on the barn layout. However, there are exceptional situations, such as when a dairy cow is in the feeding area but does not eat and turns sideways. In order to deal with such exceptional situations and postures that cannot be estimated by the method based on the barn layout, the posture of the cow may be estimated by image recognition.

Further, in the first embodiment, the moving body posture estimation unit 56 executes the posture estimation process for all the individual regions, but the present invention is not limited to this configuration. For example, when the change in the position or shape of the individual area to be processed is smaller than the preset threshold value as compared with the individual area in the immediately preceding monitoring image, the posture change of the dairy cow is small and the individual identification result is also obtained. Since there is a high possibility that they will be the same, the posture estimation process may not be performed. As a result, the time required for the extraction process and the collation process of individual characteristics is reduced.

The individual feature extraction unit 57 extracts individual features from the individual region. In the first embodiment, the individual feature extraction unit 57 extracts individual features from the individual region by the same processing as the individual feature registration unit 51 described above. Further, the individual feature extraction unit 57 extracts individual characteristics only from the individual region including the moving body having a posture suitable for identifying the individual, while the individual region including the moving body not having the posture suitable for identifying the individual. In addition, individual characteristics are not extracted for individual regions including moving objects whose posture cannot be estimated.

The individual characteristic collation unit 58 collates the extracted individual characteristics with the individual characteristic database 42 and identifies the individual. In the first embodiment, the individual feature collation unit 58 collates the individual features extracted by the individual feature extraction unit 57 with the individual features registered in the individual feature database 42, and corresponds to the extracted individual features. The individual ID to be used is acquired.

Specifically, the individual feature (feature vector) extracted by the individual feature extraction unit 57 is x _q (called a query), and the individual feature (feature vector set) registered in the individual feature database 42 is x _i (? When i = 1,2, ..., n), the individual feature matching unit 58 uses the distance d (x _i , x _q ) = || x _i -between the feature vectors represented by the following equation (4). Find the number i _min of the individual feature with the smallest (closest) x _q ||.

Then, the individual characteristic collation unit 58 acquires the individual ID associated with the _iminth individual characteristic as the individual identification result in the individual characteristic database 42. As the distance between the feature vectors, various distances such as the Euclidean distance and the cosine distance can be applied. Further, the distance d (x _i , x _q ) between the feature vectors takes a value of 0 or more, and the closer it is to 0, the more similar the individual features are, so that it may be used as the reliability of the individual identification result. .. Alternatively, the nearest neighbor distance d ₁ and the second nearest distance d ₂ may be calculated, and the distance ratio d ₂ / d ₁ of these may be used as the reliability.

The locus fragment integration unit 59 integrates the locus fragments having the highest continuity between the locus fragments and outputs them as an action locus for each individual ID. In the first embodiment, the locus fragment integrating unit 59 determines the continuity between the locus fragments accumulated in the locus fragment buffer 43 at predetermined time intervals, and integrates the locus fragments having the highest continuity. Then, the action locus with the final individual ID is output to the action locus database 45. At this time, the locus fragment integration unit 59 assigns a unique action locus ID to the integrated locus fragment group. In addition, only one individual ID corresponds to one action locus ID.

Further, in the first embodiment, the locus fragment integration unit 59 has the highest continuity between locus fragments based on temporal continuity, spatial continuity, and similarity between individual characteristics. Identify high trajectory fragments. Specifically, when the locus fragment integration unit 59 obtains the locus fragment T _j connected to the locus fragment T _i , the weighted sum d (T _i ,) of the distances between the locus fragments defined by the following equation (5). T _j ) identifies T _j , which has the smallest value, as the locus fragment with the highest continuity between the locus fragments.
d (T _i , T _j ) = w _t d _time (T _i , T _j ) + w _l d _loc (T _i , T _j ) + w _f d _feature (T _i , T _j ) ・ ・ ・ Expression (5) )

In the above equation (5), d _time (T _i , T _j ) is the difference between the end time e (T _i ) of the locus fragment T _i and the start time s (T _j ) of the locus fragment T _j , and is the locus fragment. Shows temporal continuity between. Also, d _loc (T _i , T _j ) is the difference between the final position P _e (T _i ) of the locus fragment T _i and the start position P _s (T _j ) of the locus fragment T _j , and the space between the locus fragments. Shows continuous continuity. Furthermore, d _feature (T _i , T _j ) is the distance between individual features x (T _i ), x (T _j ) associated with each of the locus fragments T _i , T _j , and the similarity between individual features. Is shown. However, d _time (T _i , T _j ) and d _loc (T _i , T _j ) shall be defined only when the locus fragments T _i , T _j are connected in this order. Further, the distance between the locus fragments is not limited to the above equation (5), and the difference in the moving speed and the moving direction may be evaluated.

Here, a specific procedure for integrating the locus fragments will be described with reference to the example of FIG. First, the integration process is started from the locus fragment T ₁ (provisional individual ID = 3) acquired by the first surveillance camera 11. Trajectory fragment T ₂ (temporary individual ID: unknown) acquired by the second surveillance camera 11 and locus fragment T ₃ (temporary individual ID = 4), T ₄ (temporary individual ID = 4) acquired by the third surveillance camera 11. The distance d (T _i , T _j ) (j = 2,3,4) for each of the provisional individual IDs = 3) is as follows. For the sake of simplicity, we set w _t = w _l = 1, w _f = 0.
d (T ₁ , T ₂ ) = | 11-10 | + √ ((19-20) ² + (12-12) ² ) = 2
d (T ₁ , T ₃ ) = | 20-10 | + √ ((9-20) ² + (12-12) ² ) = 21
d (T ₁ , T ₄ ) = | 20-10 | + √ ((9-20) ² + (8-12) ² ) = 21.7

As described above, since the distance d (T _i , T _j ) with respect to the locus fragment T ₁ is minimized in the locus fragment T ₂ , the two are combined to form the locus fragment {T ₁ , T ₂ }. Further, the above distance d (T _i , T _j ) (j = 3, 4) between the locus fragments {T ₁ , T ₂ } and the locus fragments T ₃ , T ₄ is as follows.
d (T ₂ , T ₃ ) = | 20-20 | + √ ((10-10) ² + (12-10) ² ) = 2
d (T ₂ , T ₄ ) = | 20-20 | + √ ((9-10) ² + (8-10) ² ) = 2.236

As described above, the distance d (T _i , T _j ) with respect to the locus fragment {T ₁ , T ₂ } is minimized because the locus fragment T ₃ is combined, and as a result, the action locus {T ₁ , T ₂ , T ₃ } is obtained. However, if the locus fragments are combined by the above procedure, a contradiction may occur in the temporary individual ID. That is, in the example of FIG. 13, the tentative individual ID corresponding to the locus fragment T ₁ is 3, and the tentative individual ID corresponding to the locus fragment T ₃ is different from 4, which causes a contradiction.

There are two possible causes of the above contradiction: an error in the locus fragment integration process and an error in the individual identification process. Therefore, if the accuracy and reliability of the individual identification process is high and reliable, T ₄ with the same temporary individual ID may be integrated instead of the locus fragment T ₃ with which the temporary individual ID does not match. In this way, when the individual IDs associated with each of the locus fragments having the highest continuity between the locus fragments are different, individuals having similar appearances are provided by setting a restriction that the locus fragments are not integrated. Even if they are one another, it is possible to prevent accidentally integrating the locus fragments of another individual. On the other hand, when the accuracy of the individual identification processing is not so high, even if the locus fragments have inconsistencies in the provisional individual IDs, the results of the locus fragment integration processing may be preferentially integrated.

Although the algorithm for integrating the locus fragments into greedy has been described in the first embodiment, it may be formulated and solved as a combinatorial optimization problem. By optimizing the fact that there is only one dairy cow with the same individual ID in the barn in addition to the constraint, the integration accuracy is further improved. Further, instead of setting a constraint so that the locus fragments having different temporary individual IDs are not combined with each other, the reliability of the temporary individual ID is calculated, and the temporary individual ID having the highest total reliability is integrated. May be adopted as. Furthermore, instead of resolving the contradiction of the temporary individual ID fully automatically, the operator is notified that the contradiction has occurred, and the operator confirms the monitoring image and the site and assigns the correct individual ID. It may be corrected manually.

(3) Individual feature addition function The individual feature addition function is a function for adding and expanding individual features not registered in the individual feature database 42 based on the behavior trajectory estimated by the behavior trajectory estimation function. It is functioned by the add-on 60.

In the first embodiment, the individual feature addition unit 60 first acquires a action locus from the action locus database 45 based on the individual ID of the individual to which the individual feature is to be added. Next, the individual feature addition unit 60 cuts out an individual area from the surveillance image based on the camera number, the shooting time, and the individual area in the acquired action locus data, and acquires it as a moving body image sequence. Then, the individual feature addition unit 60 extracts the individual features of each moving body image by the same processing as the individual feature extraction unit 57, and adds them in association with the individual ID registered in the individual feature database 42.

Here, FIG. 14 is a diagram showing an example of an action locus estimated by the action locus estimation device 1 of the first embodiment. Five dairy cow images (moving body images) with individual ID = X cut out from the surveillance camera 11 are picked up in chronological order. In this example, the first (t = 0) dairy cow image and the fifth (t = 80) dairy cow image are cut out from the posture presumable region, and the corresponding individual features are stored in the individual feature database 42 in advance. It is registered. On the other hand, the second (t = 10), third (t = 30) and fourth (t = 50) dairy cow images were not included in the postural estimable area, so the corresponding individual characteristics are individual characteristics. It is not registered in the database 42.

In the above situation, the individual feature addition unit 60 extracts individual features from the second (t = 10), third (t = 30), and fourth (t = 50) dairy cow images. , Each individual feature is added to the individual feature database 42. If all the dairy cow images are to be added to the individual feature database 42, the scale of the individual feature database 42 becomes large and the individual feature extraction process takes time. Therefore, clustering processing or the like is performed to reduce the number of individual features to be added. You may.

Further, the individual characteristic addition unit 60 deletes old individual characteristics and sequentially replaces the latest individual characteristics, so that the accuracy of individual identification is maintained even if the individual grows or the body shape changes. However, if the individual characteristic database 42 is updated when there is an error in the behavioral locus, the identification accuracy of the individual may decrease. Therefore, it is preferable to use only the highly reliable behavioral locus data. Alternatively, the individual characteristic database 42 may be updated after the operator confirms that there is no error in the action trajectory.

Next, the actions of the action locus estimation device 1, the action locus estimation program 1b, and the action locus estimation method of the first embodiment will be described.

When estimating the behavior trajectory of a dairy cow (moving body) using the behavior trajectory estimation device 1, the behavior trajectory estimation program 1b, and the behavior trajectory estimation method of the first embodiment, first, let's monitor using the individual characteristic registration function. The individual characteristics of all the dairy cows to be used are registered in the individual characteristics database 42.

Specifically, as shown in FIG. 15, the individual feature registration unit 51 acquires an image group of a dairy cow to be newly registered (step S1) and an individual ID of the dairy cow to be newly registered (step S1). Step S2). Then, when the individual feature registration unit 51 extracts the individual feature from the image group acquired in step S1 (step S3), the individual feature is registered in the individual feature database 42 in association with the individual ID acquired in step S2 (step S3). Step S4). As a result, the individual ID and the individual characteristics of a certain individual are registered in association with each other.

After repeating the above steps S3 to S4 for all the acquired images for a certain individual (step S5: YES), the above steps S1 to S5 are repeated for another individual. Then, when the individual characteristics of all the dairy cows to be monitored are registered (step S6: YES), this process is terminated. As a result, the individual characteristics of all the cows to be monitored are pre-registered in the individual characteristics database 42. Therefore, as long as the individual characteristics of each individual are known, the individual ID having the individual characteristics is specified, so that the individual dairy cow can be identified.

When the preparation of the individual characteristic database 42 is completed, the behavior trajectory of each dairy cow is estimated for each individual using the behavior trajectory estimation function. Specifically, as shown in FIG. 16, first, the surveillance image acquisition unit 52 acquires time-series surveillance images from a plurality of surveillance cameras 11 (step S11). Next, the individual area detection unit 53 detects the individual area of all dairy cows existing in each monitoring image (step S12), and the tracking process execution unit 54 executes the tracking process for each of the individual areas (step S12). S13).

When a new individual area appears in the surveillance image as a result of the tracking process (step S14: YES), the tracking process execution unit 54 pays out a new locus fragment ID (step S15), and associates the new individual area with the new individual area. On the other hand, when there is no new individual region in the surveillance image (step S14: NO), the tracking process execution unit 54 acquires the locus fragment ID corresponding to each individual region (step S16). As a result, in the time-series monitoring images, the same locus fragment ID is given to the individual regions that are considered to be the same individual, so that the behavior of all dairy cows is tracked for each individual.

Next, when the locus fragment acquisition unit 55 acquires the locus fragment of each individual region (step S17), the acquired locus fragment is stored in the locus fragment buffer 43 in association with the locus fragment ID (step S18). As a result, the behavioral loci of the dairy cow within the imaging range are collected as locus fragments for each of the plurality of surveillance cameras 11.

Subsequently, in the first embodiment, the moving body posture estimation unit 56 estimates the posture of the moving body included in each individual region in order to suppress the deterioration of the identification accuracy for each individual (step S19). Specifically, as shown in FIG. 17, the moving body posture estimation unit 56 first captures the individual region detected by the individual region detection unit 53 and the camera of the surveillance camera 11 that captures the surveillance image including the individual region. The number and the number are acquired (step S191).

Next, the moving body posture estimation unit 56 acquires the posture estimable region and the posture set in association with the acquired camera number (step S192), and the individual region acquired in step S191 becomes the posture estimable region. It is determined whether or not it is included (step S193). As a result of the determination, when the individual region is included in the posture estimable region (step S193: YES), the posture associated with the posture estimable region is output (step S194). On the other hand, when the individual region is not included in the posture estimable region (step S193: NO), the posture is not output. As a result, the posture of the moving body is easily estimated with high accuracy.

Subsequently, when the posture estimated by the moving body posture estimation unit 56 is a posture suitable for identifying an individual (step S20: YES), the individual feature extraction unit 57 extracts the individual characteristics of each individual region. (Step S21). On the other hand, if the posture is not suitable for identifying the individual (step S20: NO), the process proceeds to step S24 described later.

Next, the individual characteristic collation unit 58 collates the individual characteristics extracted in step S21 with the individual characteristic database 42, and acquires an individual ID corresponding to each individual region (step S22). As a result, the dairy cows included in each individual region are identified with high accuracy for each individual.

After that, the individual characteristics of each individual region extracted in step S21 and the individual ID corresponding to each individual region acquired in step S22 are stored in the locus fragment buffer 43 (step S23). Then, the processing from step S13 to step S23 described above is repeated for each individual region detected in step S12, and when the processing is completed for all individual regions (step S24: YES), the process proceeds to step S25. As a result, the locus fragments of the entire individual region included in a certain surveillance image are temporarily stored in the locus fragment buffer 43 in association with the individual characteristics and the individual ID.

Next, in step S25, the processes from step S11 to step S24 described above are repeated for the surveillance images of each surveillance camera 11, and when the processes for all the surveillance cameras 11 are completed (step S25: YES), a predetermined time elapses. The process from step S11 to step S24 described above is repeated until the above-mentioned process is performed. As a result, the locus fragments of all the individual regions included in all the surveillance images are temporarily stored in the locus fragment buffer 43 for a predetermined time.

Then, when a predetermined time elapses (step S26: YES), the locus fragment integration unit 59 integrates the locus fragments having the highest continuity between the locus fragments, and the action locus database is used as the final action locus for each individual ID. Output to 45 (step S27). As a result, even the locus fragments in the surveillance images taken by different surveillance cameras 11 are integrated for each individual with high accuracy. Therefore, even when a large number of dairy cows are monitored over a wide area, the dairy cows are identified for each individual with high accuracy, and the behavioral trajectory of each individual is correctly estimated.

Further, in the first embodiment, the individual feature addition function is used to add and expand individual features that are not registered in the individual feature database 42. Specifically, as shown in FIG. 18, the individual feature addition unit 60 acquires the action locus from the action locus database 45 based on the individual ID of the individual to which the individual feature is to be added (step S31).

Next, the individual feature addition unit 60 cuts out an individual area from the monitoring image and acquires it as a moving body image string based on the acquired action trajectory data (step S32). Then, when the individual feature addition unit 60 extracts the individual feature of each moving body image (step S33), the individual feature is added corresponding to the individual ID registered in the individual feature database 42 (step S34). As a result, individual characteristics of various postures are added, so that the accuracy of individual identification is improved.

According to the action locus estimation device 1, the action locus estimation program 1b, and the action locus estimation method of the first embodiment as described above, the following effects are obtained.
1. 1. While realizing wide-area monitoring of a large number of moving objects, it is possible to identify the moving objects for each individual with high accuracy and correctly estimate the behavioral trajectory of each individual.
2. 2. By extracting individual characteristics only from an individual region including a moving body having a posture suitable for individual identification, it is possible to suppress a decrease in identification accuracy for each individual.
3. 3. By setting the posture estimable area based on the layout of the barn or the like, the posture of the moving body can be easily estimated with high accuracy.
4. By adding individual characteristics of various postures based on the behavior trajectory data, it is possible to improve the identification accuracy of the individual.
5. By integrating the locus fragments based on the temporal and spatial continuity between the locus fragments and the similarity between individual characteristics, the behavioral locus of each individual can be estimated with high accuracy.
6. By preventing the locus fragments having inconsistencies in the individual IDs from being integrated, it is possible to prevent the locus fragments of different individuals from being erroneously integrated even if the individuals have similar appearances.

Next, a second embodiment of the action trajectory estimation device, the action trajectory estimation program, and the action trajectory estimation method according to the present invention will be described. Of the configurations of the second embodiment, the same or equivalent configurations as those of the first embodiment described above are designated by the same reference numerals, and the description thereof will be omitted again.

In the first embodiment described above, the tracking processing execution unit 54 executes tracking processing on the time-series surveillance images taken by the same surveillance camera 11 in the image coordinate system, and the locus fragment acquisition unit 55 respectively. The position coordinate group in the image coordinate system in the surveillance image is converted into the position coordinate group in the barn coordinate system and acquired as a locus fragment, and the locus fragment integration unit 59 takes a locus in the surveillance image taken by a different surveillance camera 11. The action trajectory was created by integrating the fragments.

On the other hand, the feature of the second embodiment is that the tracking process execution unit 54 previously detects the position coordinate group in the image coordinate system of the individual area detected by the individual area detection unit 53, and the position coordinate group in the barn coordinate system. Convert to. Then, among the whole images composed of all the monitoring images taken at the same time, the action trajectory in the whole barn is acquired by executing the tracking process between the whole images adjacent to each other in the time series.

Specifically, as shown in FIG. 19, the action locus estimation device 10 of the second embodiment is obtained from the whole image instead of the locus fragment integrating unit 59 that integrates the locus fragments between different surveillance cameras 11. It has an action locus output unit 61 that outputs the obtained locus fragment as an action locus as it is. Further, since the data contents of the locus fragment buffer 43 and the action locus database 45, and the processing contents of the tracking processing execution unit 54 and the locus fragment acquisition unit 55 are different from those of the first embodiment, the differences will be described.

In the second embodiment, as shown in FIG. 20, in the locus fragment buffer 43, as a rectangular individual area occupied by an individual, a two-dimensional coordinate value in the barn coordinate system is used instead of the two-dimensional coordinate value in the image coordinate system. Is saved. Further, in the second embodiment, as shown in FIG. 21, the action locus database 45 stores the two-dimensional coordinate values in the barn coordinate system instead of the two-dimensional coordinate values in the image coordinate system as the individual area. The coordinates. The other data stored in the locus fragment buffer 43 and the action locus database 45 are the same as those in the first embodiment.

The tracking process execution unit 54 tracks each individual moving object existing in the adjacent overall image in the time series among the entire images composed of all the monitoring images taken at the same time. In the second embodiment, the tracking process execution unit 54 first converts the position coordinate group in the image coordinate system of the individual area detected by the individual area detection unit 53 into the position coordinate group in the barn coordinate system. The coordinate conversion process from the image coordinate system to the barn coordinate system is the same as the process (paragraphs [0054] to [0058]) executed by the locus fragment acquisition unit 55 of the first embodiment.

Further, as shown in FIG. 22, the rectangular individual area detected by the individual area detection unit 53 is transformed into a distorted quadrangle when converted from the image coordinate system to the barn coordinate system. Therefore, in the second embodiment, the tracking process execution unit 54 uses the circumscribed rectangle for the distorted quadrangle converted to the barn coordinate system as the individual area in the barn coordinate system. The shape of the individual region is not limited to the circumscribed rectangle, and the inscribed rectangle or the polygonal region may be used as the individual region as it is. In addition, when it is determined that a plurality of individual regions detected in two or more surveillance images overlap (for example, when the overlap of individual regions is large, or when the appearance characteristics of the individual regions are similar), It is also possible to add a process of integrating multiple individual areas into one.

Next, the tracking process execution unit 54 executes the tracking process of each individual area between the entire images adjacent to each other in the time series. Specifically, as shown in FIGS. 23 (a) and 23 (b), in the entire time-series images taken by all surveillance cameras, the individual region group detected from the entire image at time t is the barn coordinate system. The one converted to {B (t) 1, B (t) 2, B (t) 3,… B (t) n _t }, and the individual area group detected from the entire image at time t-1 is the barn. Let {B (t-1) 1, B (t-1) 2, B (t-1) 3,… B (t-1) n _t-1 } be converted into a coordinate system. In this case, the tracking process execution unit 54 associates the individual regions between the entire images at time t and time t-1, so that the locus fragments for each individual are all as shown in FIG. 23 (c). Obtained while straddling the surveillance image. The tracking process execution unit 54 uses a moving object tracking algorithm such as SORT (Simple Online and Realtime Tracking), which is the same as that of the first embodiment. However, it differs from the first embodiment in that the tracking process is performed not in the individual monitoring image (image coordinate system) but in the entire image consisting of all the monitoring images (barn coordinate system).

Further, in the second embodiment, the locus fragment acquisition unit 55 specifies a center point for each of the individual regions to which the same locus fragment ID is given by the tracking processing execution unit 54, and the locus fragment acquisition unit 55 in the barn coordinate system of the center point. Calculate the two-dimensional coordinate value. Then, the two-dimensional coordinate value is acquired as it is as a locus fragment. Further, the locus fragment acquisition unit 55 associates the acquired locus fragment with the locus fragment ID and stores it in the locus fragment buffer 43.

The action locus output unit 61 outputs a locus fragment as an action locus for each individual ID. In the second embodiment, the action locus output unit 61 associates the locus fragment accumulated in the locus fragment buffer 43 by the locus fragment acquisition unit 55 with the individual ID acquired by the individual feature collation unit 58, and finally. It is output to the action locus database 45 as an action locus with a specific individual ID.

Next, the actions of the action locus estimation device 10, the action locus estimation program 1b, and the action locus estimation method of the second embodiment will be described.

Specifically, as shown in FIG. 24, when the surveillance image acquisition unit 52 acquires time-series surveillance images from a plurality of surveillance cameras 11 (step S11), the individual area detection unit 53 exists in each surveillance image. The individual area of the whole dairy cow is detected (step S12). Then, the processes from step S11 to step S12 described above are repeated for the surveillance image of each surveillance camera 11 (step S41). As a result, the individual area included in all the surveillance images acquired from all the surveillance cameras 11 is detected.

Next, when the processing for the surveillance images of all the surveillance cameras 11 is completed (step S41: YES), the tracking processing execution unit 54 is adjacent in time series among all the surveillance images taken at the same time. The tracking process of each individual area is executed between the whole images to be performed (step S13). At this time, in the second embodiment, the tracking process execution unit 54 converts the position coordinate group of the individual area from the image coordinate system to the barn coordinate system in advance, so that the tracking process between the entire images becomes possible. As a result, a locus fragment for each individual can be obtained in a state of straddling all the surveillance images.

After that, from step S14 to step S24, and in step S26, the same processing as in the first embodiment is executed. As a result, the locus fragments of all the individual regions included in the entire image of the entire barn are temporarily stored in the locus fragment buffer 43 for a predetermined time.

Then, when the predetermined time elapses (step S26), the action locus output unit 61 outputs the locus fragment as an action locus for each individual ID (step S42). That is, in the second embodiment, all the monitoring images taken at the same time are taken as the whole image, and the tracking process is executed between the whole images, so that the locus fragment becomes the action locus as it is. Therefore, unlike the first embodiment, it is not necessary to integrate the locus fragments, and the processing efficiency related to the estimation calculation is improved.

According to the second embodiment as described above, in addition to the action and effect of the first embodiment described above, it is possible to improve the processing efficiency related to the estimation calculation of the action locus.

The action trajectory estimation devices 1 and 10, the action trajectory estimation program 1b, and the action trajectory estimation method according to the present invention are not limited to the above-described embodiments, and can be appropriately changed.

For example, in the above-described embodiment, the moving body posture estimation unit 56 estimates the posture of the moving body included in each individual region in order to suppress the deterioration of the identification accuracy for each individual. However, if the identification accuracy for each individual can be guaranteed to some extent, it is not necessary to execute the posture estimation process by the moving body posture estimation unit 56.

Further, in the above-described embodiment, the individual feature registration unit 51 and the individual feature extraction unit 57 use the machine learning model, but the parameters of the machine learning model may be optimized by using the action locus data. Specifically, teacher data (milk cow image with individual ID) is created from behavioral trajectory data with individual ID, and the convolutional neural network (CNN) uses the teacher data for cross entropy loss and triplet. It may be trained to minimize the loss and the like.

Further, in the present embodiment described above, the locus fragments for several hours to one day are buffered and stored in the locus fragment buffer 43. However, when actually implementing the action trajectory estimation program 1b, it is preferable to set it as short as a few minutes. In this case, each time the action locus is output, the continuity with the action locus created so far may be determined, and the action loci having the highest continuity may be integrated. As described above, the determination of continuity is performed based on temporal continuity, spatial continuity, and similarity between individual characteristics.

1 Behavior trajectory estimation device (first embodiment)
1a Individual feature registration program 1b Behavior trajectory estimation program 1c Individual feature addition program 2 Display means 3 Input means 4 Storage means 5 Arithmetic processing means 10 Action trajectory estimation device (second embodiment)
11 Surveillance camera 41 Program storage 42 Individual feature database 43 Trajectory fragment buffer 44 Attitude estimation database 45 Behavior locus database 51 Individual feature registration unit 52 Surveillance image acquisition unit 53 Individual area detection unit 54 Tracking processing execution unit 55 Trajectory fragment acquisition unit 56 Move Body posture estimation part 57 Individual feature extraction part 58 Individual feature matching part 59 Trajectory fragment integration part 60 Individual feature addition part 61 Behavior locus output part

Claims (11)

It is a behavior trajectory estimation device that can estimate the behavior trajectory of a moving body for each individual.
An individual area detection unit that detects the image area occupied by the individual as an individual area in each time-series surveillance image taken by a plurality of surveillance cameras.
A tracking process execution unit that tracks the moving object for each individual by associating individual regions having similar positions, moving directions, and velocities between adjacent surveillance images in chronological order.
A locus fragment acquisition unit that acquires a time-series position coordinate group as a locus fragment in a predetermined coordinate system of an individual region associated with the tracking process execution unit.
An individual feature extraction unit that extracts individual features, which are feature vectors based on the appearance of the individual, from the individual region.
The individual feature extracted by the individual feature extraction unit is compared with the individual feature database in which the individual feature is registered in association with the individual ID that identifies the individual, and the individual ID corresponding to the extracted individual feature is collated. With the individual feature collation unit to acquire
A locus fragment integration unit that integrates the locus fragments having the highest continuity between the locus fragments and outputs them as an action locus for each individual ID.
An action locus estimation device. It has a moving body posture estimation unit that estimates the posture of the moving body included in the individual area.
The behavior trajectory estimation device according to claim 1, wherein the individual feature extraction unit extracts the individual feature only from an individual region including a moving body having a posture suitable for identifying the individual. For each of the surveillance cameras, a posture estimable region in which the posture of the moving body can be estimated and a posture often taken by the moving body existing in the posture estimable region are set.
The moving body posture estimation unit estimates the posture associated with the posture estimable region as the posture of the moving body included in the individual region when the individual region is included in the posture estimable region. 2. The action locus estimation device according to 2. Based on the behavioral trajectory, for each individual ID, the individual area is cut out from the monitoring image to acquire a moving body image sequence, and the individual characteristics of each moving body image are extracted and added to the individual characteristic database. The action trajectory estimation device according to any one of claims 1 to 3, which has a feature addition unit. The locus fragment integration unit identifies the locus fragment having the highest continuity between the locus fragments based on the temporal continuity, spatial continuity, and similarity between individual characteristics between the locus fragments. The action trajectory estimation device according to any one of claims 1 to 4. Any of claims 1 to 5, wherein the locus fragment integrating unit does not integrate the locus fragments when the individual IDs associated with each of the locus fragments having the highest continuity between the locus fragments are different. The behavior trajectory estimation device described in Crab. It is a behavior trajectory estimation program that can estimate the behavior trajectory of a moving body for each individual.
An individual area detection unit that detects the image area occupied by the individual as an individual area in each time-series surveillance image taken by a plurality of surveillance cameras.
A tracking process execution unit that tracks the moving object for each individual by associating individual regions having similar positions, moving directions, and velocities between adjacent surveillance images in chronological order.
A locus fragment acquisition unit that acquires a time-series position coordinate group as a locus fragment in a predetermined coordinate system of an individual region associated with the tracking process execution unit.
An individual feature extraction unit that extracts individual features, which are feature vectors based on the appearance of the individual, from the individual region.
The individual feature extracted by the individual feature extraction unit is compared with the individual feature database in which the individual feature is registered in association with the individual ID that identifies the individual, and the individual ID corresponding to the extracted individual feature is collated. With the individual feature collation unit to acquire
A locus fragment integration unit that integrates the locus fragments having the highest continuity between the locus fragments and outputs them as an action locus for each individual ID.
An action trajectory estimation program that makes a computer function. It is a behavior trajectory estimation method that can estimate the behavior trajectory of a moving body for each individual.
An individual area detection step for detecting an image area occupied by an individual as an individual area in each of the time-series surveillance images taken by a plurality of surveillance cameras.
A tracking process execution step for tracking the moving object for each individual by associating individual regions having similar positions, moving directions, and velocities between adjacent monitoring images in a time series.
A locus fragment acquisition unit that acquires a time-series position coordinate group as a locus fragment in a predetermined coordinate system of the individual region associated with the tracking process execution step.
An individual feature extraction step for extracting individual features, which are feature vectors based on the appearance of the individual, from the individual region,
The individual features extracted in the individual feature extraction step are collated with the individual feature database in which the individual features are registered in association with the individual ID that identifies the individual, and the individual ID corresponding to the extracted individual features is collated. And the individual feature matching step to acquire
A locus fragment integration step that integrates the locus fragments having the highest continuity between the locus fragments and outputs them as an action locus for each individual ID.
A method for estimating a behavioral trajectory. It is a behavior trajectory estimation device that can estimate the behavior trajectory of a moving body for each individual.
An individual area detection unit that detects the image area occupied by the individual as an individual area in each time-series surveillance image taken by a plurality of surveillance cameras.
The moving object is tracked for each individual by associating individual regions having similar positions, moving directions, and velocities between the entire images that are adjacent to each other in the time series among the entire images composed of all the monitoring images taken at the same time. Tracking process execution unit and
A locus fragment acquisition unit that acquires a time-series position coordinate group as a locus fragment in a predetermined coordinate system of an individual region associated with the tracking process execution unit.
An individual feature extraction unit that extracts individual features, which are feature vectors based on the appearance of the individual, from the individual region.
The individual feature extracted by the individual feature extraction unit is compared with the individual feature database in which the individual feature is registered in association with the individual ID that identifies the individual, and the individual ID corresponding to the extracted individual feature is collated. With the individual feature collation unit to acquire
An action locus output unit that outputs the locus fragment as an action locus for each individual ID,
An action locus estimation device. It is a behavior trajectory estimation program that can estimate the behavior trajectory of a moving body for each individual.
An individual area detection unit that detects the image area occupied by the individual as an individual area in each time-series surveillance image taken by a plurality of surveillance cameras.
The moving object is tracked for each individual by associating individual regions having similar positions, moving directions, and velocities between the entire images that are adjacent to each other in the time series among the entire images composed of all the monitoring images taken at the same time. Tracking process execution unit and
A locus fragment acquisition unit that acquires a time-series position coordinate group as a locus fragment in a predetermined coordinate system of an individual region associated with the tracking process execution unit.
An individual feature extraction unit that extracts individual features, which are feature vectors based on the appearance of the individual, from the individual region.
The individual feature extracted by the individual feature extraction unit is compared with the individual feature database in which the individual feature is registered in association with the individual ID that identifies the individual, and the individual ID corresponding to the extracted individual feature is collated. With the individual feature collation unit to acquire
An action locus output unit that outputs the locus fragment as an action locus for each individual ID,
An action trajectory estimation program that makes a computer function. It is a behavior trajectory estimation method that can estimate the behavior trajectory of a moving body for each individual.
An individual area detection step for detecting an image area occupied by an individual as an individual area in each of the time-series surveillance images taken by a plurality of surveillance cameras.
The moving object is tracked for each individual by associating individual regions having similar positions, moving directions, and velocities between the entire images that are adjacent to each other in the time series among the entire images composed of all the monitoring images taken at the same time. Tracking process execution step and
A locus fragment acquisition unit that acquires a time-series position coordinate group as a locus fragment in a predetermined coordinate system of the individual region associated with the tracking process execution step.
An individual feature extraction step for extracting individual features, which are feature vectors based on the appearance of the individual, from the individual region,
The individual features extracted in the individual feature extraction step are collated with the individual feature database in which the individual features are registered in association with the individual ID that identifies the individual, and the individual ID corresponding to the extracted individual features is collated. And the individual feature matching step to acquire
An action locus output step that outputs the locus fragment as an action locus for each individual ID,
A method for estimating a behavioral trajectory.

Images