JP6346007B2 - Motion recognition device and motion recognition method - Google Patents

Description

  The present invention relates to a motion recognition device, a motion recognition method, and the like that automatically recognize the motion of a moving object such as a laboratory animal.

  Capturing and analyzing the behavior of experimental animals represented by mice and the like is an experimental technique widely used in drug efficacy tests and the like in addition to basic research such as animal behavior studies. Classically, a method has been adopted in which an experimenter visually observes a photographed image, and detects and records several types of behavior, but this method is caused by visual observation. There are many problems, such as high labor costs and low reproducibility due to the subjective judgment of the observer.

In order to solve such problems, various automatic analysis softwares have been developed conventionally. For example, a device that detects the center of gravity of a mouse from a photographed image from above and tracks the position and moving speed at each time point, the moving path within a predetermined time range, and the like is widely used.
The “center of gravity” here is a point obtained with respect to a two-dimensional figure detected by an appropriate method from a captured image, and does not take into account the volume or weight of the mouse. Hereinafter, the term “center of gravity” used in this specification has the above-mentioned meaning.

  Furthermore, in recent years, a technique using model fitting has been disclosed as a technique for observing mouse behavior in more detail. Examples of this include physical model fitting (see Non-Patent Document 1 and FIG. 10A) and elliptical fitting (see Non-Patent Document 2 and FIG. 10B). According to these methods, it is possible to track the movement of the mouse, the direction of the body, and the like without visual observation.

JP 2012-190280 A

Nature Methods, USA, Nature Publishing Group, Vol. 9, p. 410-417 Journal of Neuroscience Methods, Elsevier B.V., Volume 219, p. 10-19

  However, a mouse is an animal that expresses its internal state with a wide variety of actions, and indices for evaluating actions are not limited to the above movements and body orientations.

  For example, a mouse may move its nose tip very finely after receiving an electric shock, which is an indicator of anxiety behavior. Ellipse fitting treats the mouse as the ellipse that best matches the shape of the whole body (ie, excludes some pixels that protrude from the ellipse), which can cause this detection of small nose movement detection. is there.

  In addition, since the mouse is a very flexible animal, the neck and trunk are often bent greatly. The above physical model fitting is basically a model based on the ideal shape of a mouse in a stationary state, and it is difficult to correctly detect a large deformation of the body part. The same problem can be pointed out for ellipse fitting, for example, when the mouse bends the neck or torso, the shape of the ellipse is greatly deformed, and the wrong position on the circumference of the ellipse is determined as the base of the nose or tail There is a risk of being.

  The present invention has been made in view of the above circumstances, and its object is to improve tracking accuracy in behavior analysis of experimental animals such as mice.

The motion recognition apparatus according to the present invention, which has been made to solve the above-mentioned problems, is a recognition apparatus that automatically recognizes the motion of a moving object such as a laboratory animal from a plurality of frames of the moving object imaged by an imaging unit. There,
a) an overall center of gravity specifying means for specifying an overall center of gravity that is a center of gravity of the moving object region corresponding to the moving object in the image;
b) area dividing means for dividing the moving object area into a plurality of areas using a machine learning algorithm;
c) end point specifying means for specifying a point farthest from the overall center of gravity as an end point of the moving object in a predetermined area among the plurality of areas divided by the area dividing means;
It is characterized by providing.

The “animal body” includes, in addition to quadruped animals such as mice and rats, birds such as poultry, cells in a predetermined environment, and the like. The cell that can be included in the moving body of the present invention is a cell that moves with polarity, such as various animal cells typified by nerve cells, cancer cells, immune cells, and the like.
Further, the “predetermined area” may be determined based on an instruction from the observer, or may be automatically determined using an appropriate image recognition method. As the latter example, specifically, among the plurality of regions, the region that is the furthest away from the overall center of gravity, or the region that has a characteristic shape that serves as a material for determining information about moving objects is automatically described above. It may be determined as a predetermined area. For example, when a moving body is a mouse, a certain area has a characteristic shape. For example, when a moving object is a mouse, a nose tip contour is detected and a region including the contour of the nose is defined as the predetermined region. You can also

According to the above configuration, the moving object region is divided into a plurality of regions by the machine learning algorithm, and a point that is within the predetermined region and is farthest from the entire center of gravity is specified as the end point of the moving object. Thus, for example, when the moving object is a mouse, the algorithm divides the moving object region into a plurality of regions in the front-rear direction, of which the foremost region is a predetermined region, and the farthest end from the total center of gravity in the region is the tip of the nose of the mouse Can be specified as The number of divisions of the moving body region and which of the plurality of divided regions is set as the predetermined region may be appropriately determined according to the nature and shape of the motion or deformation of the moving body.
In this configuration, since an image of a moving object is not applied to a specified model, important (specifically, end point) pixels in the moving object region are excluded and do not leak from the detection target. Further, since the entire center of gravity of the moving object region is obtained for each image, even if the moving object is deformed by bending or the like, the end point can be specified based on the distance from the deformed entire center of gravity. Therefore, tracking accuracy is improved in behavior analysis of experimental animals such as mice.

The moving body is an animal having a longitudinal axis;
The area dividing means divides the moving object area in the front-rear direction,
The end point specifying means is configured to specify a point farthest from the overall center of gravity as an end point of the moving body in an end region which is a region located at one end in the front-rear direction among the plurality of regions. Is preferred.

Here, “having a longitudinal axis” specifically has a head and tail (or abdomen), and, as a normal movement mode, the property of moving in the direction from the overall center of gravity to the center of gravity of the head. It is synonymous with having.
According to the above configuration, high-accuracy tracking can be performed for many experimental animals having a longitudinal axis by specifying the end points represented by the nose of the mouse. In addition, by reversing the “one end portion” in the front-rear direction, it is possible to specify not only the end point located at the front part (the nose tip of the mouse) but also the end point located at the rear part (for example, the base of the tail). is there.

  It has been known for a long time that the mouse has the habit of performing exploratory behavior at the tip of the nose, but as a result of recent research, the mouse is acting much more visually than previously thought I understand. Moreover, the various behaviors supporting this are particularly prominent when there are two or more mice in the field. With these discoveries, the gaze, that is, how much the head is tilted with respect to the trunk, is beginning to be recognized as an important evaluation index of social behavior in mouse experiments.

Therefore, the motion recognition device further includes
d) It is preferable to include an end portion center of gravity specifying means for specifying the center of gravity of the end region.

According to said structure, in a moving body area | region, the end point, the gravity center of an edge part area | region, and the at least 3 feature point of the whole gravity center are specified. Thereby, the software which performs image analysis can obtain detailed information about the shape of the moving object. Specifically, for example, consider a case where a moving body is a mouse and a region located at the forefront in the front-rear axis is defined as an end region. At this time, even if the number of divisions by the region dividing means is set to 2 which is the smallest, the center of gravity of the end region is obtained near the back of the head, so the end point (nose tip) and the center of gravity of the end region (back of the head) The tilt angle of the head with respect to the body of the mouse can be obtained with high accuracy from the positional relationship between the three points of the nearby points) and the overall center of gravity. If the number of divisions by the region dividing means increases, the center of gravity of the end region approaches the end point, so that it is less affected by bending (for example, neck) in the region, and the accuracy is further improved.
As described above, the region located at the rearmost part in the front-rear axis may be an end region, and the center of gravity of the end region may be specified. Therefore, in addition to the above three points in the mouse, if the center of gravity of the end region on the rear side (or the base of the tail which is the end point on the rear side) is obtained, the posture of the torso can be grasped more accurately, and the head with respect to the torso The tilt angle is required with higher accuracy.
Therefore, according to the said structure, the very useful evaluation method in the analysis of the social behavior of a mouse | mouth can be provided.
As a conventional analysis method focusing on the orientation of the head, for example, in Patent Document 1, the mouse top view image is binarized to obtain a center of gravity, and the mouse contour information is obtained by polar coordinate conversion based on the obtained center of gravity. And specifying the direction of the tail and the direction of the head based on the contour information. However, according to the above configuration of the present invention, the orientation of the head can be obtained with high accuracy by a simple method without obtaining contour information by polar coordinate conversion every time.

Preferably, the end portion center-of-gravity specifying means sets the center of gravity of the end region specified immediately before as an initial value of the center of gravity of the end region to be specified in the next frame.
In a general captured video, it is rare that the moving object is reversed between successive frames, so the above configuration makes it possible to misrecognize an end point (that is, mistakenly determine a rear end point as a front end point). Is prevented.

As a machine learning algorithm used by the region dividing means, for example, k-means clustering can be used.
Thereby, a moving body area | region can be easily divided | segmented into a some area | region by a high-speed process.

Here, as an example, if an image of a mouse is divided into too many (for example, 10 or more) areas, the moving body area is divided as intended by the observer (for example, in the front-rear direction) due to the body width of the mouse. May not be.
Therefore, the area dividing means may determine the number of divisions of the moving object area based on the result of the preliminary experiment.
This prevents area division that does not meet the purpose of behavior analysis.

The motion recognition method according to the present invention made to solve the above problems is a recognition method that automatically recognizes the motion of a moving object such as a laboratory animal from a plurality of frames of the moving object imaged by the imaging unit. There,
a) an overall center of gravity specifying step for specifying an overall center of gravity that is a center of gravity of a moving object region corresponding to the moving object in the image;
b) a region dividing step of dividing the moving object region into a plurality of regions using a machine learning algorithm;
c) an end point specifying step for specifying a point farthest from the overall center of gravity as an end point of the moving body in a predetermined region among the plurality of regions divided in the region dividing step;
It is characterized by including.

  According to the present invention, tracking accuracy is improved in behavior analysis of experimental animals such as mice.

1 is a block diagram showing a schematic configuration of a behavior analysis system including a motion recognition device according to a first embodiment of the present invention. The flowchart which shows the flow of the process which the operation | movement recognition apparatus which concerns on the same embodiment performs. Explanatory drawing which shows the procedure of the area | region division of the mouse image by the k-means clustering which the motion recognition apparatus which concerns on the embodiment performs. The flowchart which shows the flow of the process which sets the initial value of each cluster center shown in FIG.3 (c). The graph which compared the tracking accuracy of the nose tip of the mouse | mouth by the same embodiment with the conventional method. Explanatory drawing which shows the procedure of the area | region division | segmentation of the mouse image by the k-means clustering which the motion recognition apparatus which concerns on the 2nd Embodiment of this invention performs. It is explanatory drawing which shows the method of evaluating the recognition accuracy of the orientation of a mouse head, (a) is the orientation of the head visually identified by the present inventors from the original mouse image, (b) is the conventional The head orientation estimated by the ellipse fitting is shown. (C) is the head orientation estimated based on the three points of the nose tip, the entire center of gravity, and the base of the tail specified by the embodiment. The head directions estimated based on the five points of the nose tip, the front center of gravity, the total center of gravity, the rear center of gravity, and the base of the tail specified by the embodiment are indicated by broken or solid arrows, respectively. The graph which shows the angle difference of the head direction which the user specified in Fig.7 (a), and the head direction estimated by the same figure (b), (c) and (d), respectively. The graph for a comparison which added the result by 1st Embodiment to FIG. (A) The figure which shows an example of the conventional physical model fitting. (B) The figure which shows an example of the conventional ellipse fitting.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the following description, members having the same functions as those in the above-described drawings are denoted by the same reference numerals, and description thereof is omitted.

[First Embodiment]
FIG. 1 is a block diagram showing a schematic configuration of a behavior analysis system including a motion recognition device according to an embodiment of the present invention. This behavior analysis system includes an observation box 10 and an information processing device 20 connected to the observation box 10 for analyzing and processing the behavior of a mouse in the observation box 10 (the motion recognition device of the present invention). Equivalent).

  The observation box 10 has a field that can be searched by the mouse 12, and the edge of the field is surrounded by a wall plate having a predetermined height. A camera 11 (corresponding to the imaging unit of the present invention) is provided inside the top plate of the observation box 10, whereby the action of the mouse 12 is photographed from above. Note that the observation box 10 includes a lighting device (not shown), for example, inside the top plate. The observation box 10 may further include a wiring for presenting an electric shock on the floor board.

  The actual state of the information processing apparatus 20 is a computer such as a workstation or a PC (Personal Computer), and a CPU (Central Processing Unit) 21 that is a central processing unit and a memory including a volatile storage device such as a RAM (Random Access Memory). 22, a display unit 23 composed of an LCD (Liquid Crystal Display), an input unit 24 composed of a keyboard, a mouse, etc., and a storage unit 30 composed of a mass storage device such as an HDD (Hard Disc Drive) or SSD (Solid State Drive) Are connected to each other. The storage unit 30 stores an OS (Operating System) 31 and an action analysis program 32. The information processing apparatus 20 further includes an interface (I / F) 25 for direct connection with an external apparatus and connection with the external apparatus via a network such as a LAN (Local Area Network). The I / F 25 is connected via a network cable NW (or wireless LAN) to a camera 11 provided in the observation box 10 or a lighting device (not shown). As another configuration example, the I / F 25 and the observation box 10 may be connected via a control device (not shown) dedicated to the observation box 10.

  The behavior analysis program 32 is a program having various functions for analyzing the behavior of the mouse 12 taken by the camera 11. However, in the following description, detailed descriptions of functions that deviate from the gist of the present invention (for example, a method of detecting a type behavior and a classification method) are omitted.

  In FIG. 1, as related to the behavior analysis program 32, the imaging control unit 41, the image acquisition unit 42, the preprocessing unit 43, the overall center of gravity specifying unit 44, and the region dividing unit 45 (the region dividing means and the end portion center of gravity of the present invention). An end point specifying unit 46 and a frame information storage unit 47 are shown. These are basically functional means realized by software by the CPU 21 reading the behavior analysis program 32 into the memory 22 and executing it. The behavior analysis program 32 is not necessarily a single program, and may be a function incorporated in a part of a program for controlling various machine elements provided accompanying the observation box 10, for example. Well, the form is not particularly limited. Further, a part of the functions provided in the behavior analysis program 32 (for example, the imaging control unit 41) may be provided in the control device dedicated to the observation box 10 as described above.

The photographing control unit 41 controls photographing inside the observation box 10 by the camera 11. For example, the start or end of shooting is instructed using a user instruction via the input unit 24 as a trigger.
The image acquisition unit 42 acquires an image inside the observation box 10 captured by the camera 11 (a plurality of frames constituting a moving image having a predetermined length). The plurality of frames of images acquired by the image acquisition unit 42 may be stored in a storage area (not shown) in the storage unit 30 in addition to the memory 22.

The preprocessing unit 43 performs preprocessing such as binarization of the image acquired by the image acquisition unit 42 and noise removal. A known technique can be used for these processes.
Here, image preprocessing will be described with reference to FIGS. FIG. 3A illustrates a small section centered on the mouse 12 in the image of a certain frame acquired by the image acquisition unit 42 for explanation. On the other hand, when the preprocessing unit 43 performs binarization and noise removal as necessary, an image as shown in FIG. Among these, the area shown in white is a mouse area 13 (corresponding to a moving object area of the present invention) that is a target of the overall center of gravity and area division described later.

  The description will be given with reference to FIG. 1 again. The overall center of gravity specifying unit 44 specifies the overall center of gravity C (see FIG. 3B), which is the center of gravity of the mouse region 13 as a two-dimensional figure, based on the preprocessed image. The identification of the overall center of gravity C is also possible by applying a known technique. Note that “specific (or setting)” of the center of gravity (and the cluster center and each end described later) means that the coordinates in the image are uniquely determined in the present embodiment.

The area dividing unit 45 divides the mouse area 13 into k areas by k-means clustering. In this embodiment, k = 2 and the mouse area 13 is divided into two areas. The procedure of area division by k-means clustering will be described later.
The area dividing unit 45 also specifies the center of each cluster at the time when the clustering has converged as each center of gravity of the preceding and following areas.

  The end point specifying unit 46 specifies the nose tip 51 and the base of the tail 52 of the mouse 12 (see FIG. 3H). Specifically, the pixels farthest from the overall center of gravity C are specified as the nose tip 51 and the base of the tail 52 for the pixels in the two regions before and after being divided by the region dividing unit 45, respectively. In the present embodiment, the tip of the nose 51 and the base 52 of the tail each correspond to an end point of the present invention.

  The frame information storage unit 47 holds and accumulates the processing results obtained by the preprocessing unit 43, the overall center-of-gravity specifying unit 44, the region dividing unit 45, and the end point specifying unit 46 as information relating to an image of one frame. When information related to a predetermined number of frames of images is accumulated in the frame information storage unit 47 or by arbitrary triggering by the user, behavior analysis such as detection and classification of the type behavior of the mouse 12 is performed.

  Hereinafter, the flow of processing performed by the information processing apparatus 20 according to the present embodiment will be described with reference to FIG. 2 which is a flowchart of the main routine, and FIG. 3 which is a flowchart of a subroutine as needed. Now, it is assumed that the camera 11 captures a moving image inside the observation box 10 according to an instruction from the shooting control unit 41, and the image acquisition unit 42 acquires images of I consecutive frames constituting the moving image.

  First, the behavior analysis program 32 sets the variable i indicating the frame number (1 ≦ i ≦ I) to 1 (step S101), and the preprocessing unit 43 first selects the first frame image acquired by the image acquisition unit 42. (For example, FIG. 3A) is acquired (step S102), and the image is binarized (step S103). If noise remains in the image after binarization in step S103, it is removed by an appropriate method. Subsequently, the overall center of gravity specifying unit 44 specifies the overall center of gravity C (see FIG. 3B) of the mouse region 13 (step S104: corresponding to the overall center of gravity specifying step of the present invention).

Next, the region dividing unit 45 determines the initial values fC _{i, 1} and rC _{i, 1} of the two cluster centers fC _i and rC _{i in} the mouse region 13 (step S105). Although details of the processing related to this step will be described later, since the processing related to the first frame is currently being performed, these initial values fC _1,1 and rC _1,1 are arbitrarily determined (for example, FIG. ) Cluster center fC ₁ and rC ₁ ). Here, it has been described that the initial values fC _1,1 and rC _1,1 are arbitrarily determined, but it is preferable that fC _1,1 is determined at least in front of rC _1,1 . Specifically, for example, the user may specify via the input unit 24 based on visual judgment, or by automatically recognizing the front and back of the mouse region 13 using another image recognition method, fC _1,1 , rC ₁ and ₁ may be set at random positions on the front and rear, respectively. By predefining only the front and back of the cluster center in this way, it is possible to prevent a detection error in which the tip of the nose and the base of the tail are reversed in the specific processing of the front and rear ends described later.

Subsequently, the region dividing unit 45 applies k-means clustering to the pixels in the mouse region 13, and first divides the mouse region 13 into two regions having fC _1,1 and rC _1,1 as centroids, respectively. (Step S106: equivalent to the region dividing step of the present invention).

  The procedure of region division by k-means clustering starting from this step will be described with reference to FIG.

First, the initial values fC ₁ and rC _{1 of the} two cluster centers fC and rC were determined at given positions as shown in FIG. 3C by the above-described steps S102 to S105. Since FIG. 3 shows a procedure common to all frames, the frame number i in the code is omitted.

Next, as shown in FIG. 4D, each pixel is assigned to the front and rear two regions F ₁ and R ₁ . The regions F ₁ and R ₁ have a perpendicular bisector (Voronoi boundary) connecting the cluster center fC ₁ defined as the initial value and rC ₁ as a boundary. When the pixels are assigned, the cluster centers are updated as fC ₂ and rC ₂ as shown in FIG. fC ₂ and rC ₂ are determined at positions that minimize the sum of the distances from the respective pixels in the regions F ₁ and R ₁ . Subsequently, Voronoi boundaries are determined for these fC ₂ and rC ₂ , and pixels are assigned to the regions F ₂ and R ₂ based on this boundary ((f) in the figure). Then, the cluster centers are updated as fC ₃ and rC ₃ for the regions F ₂ and R ₂ ((g) in the figure).

When the processing described above is repeated and the amount of change in the data value due to the update falls below a predetermined threshold (for example, fC _{n + 1} −fC _n <Th and rC _{n + 1} −rC _n <Th for the coordinate values of the cluster centers fC and rC are Established; Th is a predetermined threshold value), the area dividing unit 45 determines that the clustering has converged (Yes in S107). If the amount of change is equal to or greater than the threshold (No in S107), the region dividing unit 45 continues the above-described cluster center update and region division.

An example of a frame image at the time when it is determined that the clustering has converged is shown in FIG. Based on this image information, the region dividing unit 45 determines F _n as the front region, R _n as the rear region, and fC _n and rC _n as the front and rear centroids (step S108: end centroid of the present invention). Equivalent to a specific step).

Then, the end point identifying section 46, the point farthest from the total center of gravity C of the front region _{F n} (the exact pixel) identified as nose 51 of the mouse 12 (step S109), the whole of the rear region _{R n} A point (pixel) farthest from the center of gravity C is identified as the base 52 of the tail of the mouse 12 (step S110). Each of these steps corresponds to an end point specifying step of the present invention, and the execution order may be reversed.

  Next, the preprocessing unit 43 increments i (step S111), and if i does not exceed the number I of frames constituting the moving image (No in step S112), executes steps S102 to S112 for the next frame image.

On the other hand, if i exceeds I as a result of the increment in step S111 (Yes in step S112), the determination result is the identification of the entire center of gravity, the region division, the identification of each center of gravity before and after, Since this means that the identification of the base of the tail has been completed, the processing is terminated.
The processing results in steps S104 and S108 to S110 are sequentially stored in the frame information storage unit 47, and after the information about all the frames is accumulated, it becomes original data used for various behavior analysis.

According to the processing described above, the image is binarized for each frame constituting the moving image, the mouse region 13 is defined, and the entire center of gravity C is specified. The mouse region 13 is divided into the front region F _n and the rear region R _n by k-means clustering (k = 2), and the two cluster centers fC _n and rC _n are specified as the front and rear centroids. The Furthermore, the points farthest from the overall center of gravity C of the front region F _n and the rear region R _n are specified as the nose tip 51 and the base of the tail 52 of the mouse 12, respectively.

Here, the process of determining the initial values fC _{i, 1} and rC _{i, 1} of the two front and rear cluster centers in step S105 will be described with reference to FIG.
First, it is determined whether i is equal to 1, that is, whether the image currently being processed is the first frame in the moving image (step S201). If i is equal to 1 (Yes in S201), the initial values fC _1,1 and rC ₁ are set based on the user input (or the setting using the image recognition method described above) as described in the description of step S105 above. _{, 1} is determined (step S202).
On the other hand, when it is determined that the value of i is 2 or more (No in S201), this determination result means that the cluster centers before and after the previous frame have already been determined. At this time, the region dividing unit 45 uses the cluster centers fC _{i−1, n} and rC _{i−1, n} determined as the front center of gravity and the rear center of gravity in the previous frame as the initial values fC _{i, n} of the currently processed frame _{. 1} and rC _{i, 1} are set (step S203).

In normal captured moving images, it is rare that the posture changes so rapidly that the front and rear of the mouse 12 are reversed between successive frames. Therefore, by using the cluster centers fC _{i−1, n} and rC _{i−1, n} determined in the previous frame as the initial values fC _{i, 1} and rC _{i, 1} in this way, the user can visually check Is not detected, the detection mistake of the nose tip 51 and the base of the tail 52 due to the reverse of the front-rear recognition is prevented. Moreover, according to this structure, the process like patent document 1 which recognizes the outline of the mouse | mouth area | region 13 and identifies a nose tip from the shape of this outline can be made unnecessary.

[Experimental example 1: Tracking of nose tip]
In order to confirm the effect of this embodiment, an evaluation experiment was performed on the tracking accuracy of the nose tip of the mouse 12 under the following conditions.
As a data set, images of five consecutive frames were sampled from 100 moving images obtained by photographing the free movement of the mouse 12 over 30 seconds. That is, the total number of analysis target images is 500. For each of these 500 images, the distance from the nose tip position manually marked (annotated) by the present inventors based on visual observation is used as an evaluation index, and conventionally used elliptical fitting and currently widely used commercial fittings are used. The tracking accuracy of the tip of the nose was compared between EthoVision XT (registered trademark) manufactured by Noldus, which is software, and the proposed method to which this embodiment is applied.

  The results of the evaluation experiment are shown in FIG. The horizontal axis represents the distance (number of pixels) from the annotation position, and the vertical axis represents the cumulative distribution (%). As shown in the figure, the ratio of the identified nose tip position within 10 pixels (indicated by the alternate long and short dash line) from the annotation position was 45% and 74% in elliptical tracking and commercial software. On the other hand, with the proposed method using this embodiment, the nose tip of the mouse 12 can be traced with a very high accuracy of 94%.

As described above, according to the present embodiment, the mouse area 13 is divided into front and rear by k-means clustering, simple method of determining the furthest point from the total center of gravity C mouth area 13 in the front region F _n Thus, the nose tip 51 can be tracked with high accuracy. In addition, since tracking according to the present embodiment does not perform fitting to a physical model or an ellipse, there is little error due to bending or fine movement of the body, and high robustness is maintained.

[Second Embodiment]
In the first embodiment, the configuration in which the mouse region 13 is divided into the front and rear regions has been described. However, the mouse region 13 may be divided into more regions, for example, three regions. That is, k = 3 may be set in step S106 of the flowchart shown in FIG. However, in this embodiment, one of the three cluster centers is fixed to the entire center of gravity C, and only the other cluster centers fC and rC are updated.

With reference to FIG. 6, the division | segmentation procedure of the mouse | mouth area | region 13 using the k-means clustering in this embodiment is demonstrated.
6 (a) and 6 (b) correspond to FIGS. 3 (b) and 3 (c). However, in FIG. 3C, the entire center of gravity C is not shown for the sake of simplification of description and easy viewing of the drawing.
In the present embodiment, the processing ahead is different. FIG. 6C shows how the mouse region 13 is divided by the three cluster centers of fC ₁ , C and rC ₁ . By a so-called Voronoi boundary explained in the first embodiment, the mouse area 13 is first divided into front region F ₁ and central area M ₁ and the rear region R _1. Next, the cluster centers fC ₁ and rC ₁ are updated to fC ₂ and rC ₂ based on these three regions F ₁ , M ₁ and R ₁ (FIG. 6 (d)). As described above, the entire center of gravity C is a fixed cluster center and is not an update target. Thereafter, the pixels corresponding to the regions F ₂ , M _2, and R ₂ (not shown) are allocated and the fC ₂ and rC ₂ are updated. After the same processing is repeated, each pixel in the three regions and three Clustering converges when the sum of the distances from the cluster center (including the entire center of gravity C) is minimized (FIG. 6 (e)).

[Experimental example 2: Estimation of head orientation]
In order to confirm the effect of this embodiment, an evaluation experiment was performed on the estimation accuracy of the orientation of the head of the mouse 12 under the following conditions. This will be described with reference to FIGS.
The data set is the same as Experimental Example 1 in the first embodiment. For each of the 500 images, the head orientation visually identified by the present inventors (hereinafter referred to as “actual head orientation 60” for convenience) (an example is a broken line in FIG. 7A). An angle difference θ (°) from an arrow) is used as an evaluation index, and is based on ellipse fitting and three points (the nose tip 51, the entire center of gravity C, and the base of the tail 52) specified by the proposed method using this embodiment. And the case based on the five points (the nose tip 51, the front center of gravity fC _n , the total center of gravity C, the rear center of gravity rC _n and the tail base 52) specified by the proposed method, Compared.

Ellipse fitting, the proposed method using this embodiment (3 points), and head orientations 61, 62 and 63 estimated by the proposed method (5 points) are shown in FIGS. ) With solid arrows. As can be seen from FIG. 5B, in the ellipse fitting, both ends of the ellipse in the extending direction are the nose tip 51a and the base 52a of the tail, and therefore the head orientation estimated from the positional relationship between the overall center of gravity C and the nose tip 51a. 61 is greatly deviated from the actual head direction 60. On the other hand, in the proposed method (three points), the head orientation 62 estimated from the positional relationship between the total center of gravity C and the nose tip 51 is not different from the actual head orientation 60 compared to the above-described elliptic fitting. Although alleviated, it is difficult to say that it is sufficient to measure the line of sight of the mouse 12. This “proposed method (3 points)” is reference data assuming an estimation result by the commercial software (algorithm not disclosed).
On the other hand, in the proposed method (5 points), as shown in FIG. 4D, the head direction 63 estimated from the positional relationship between the front center of gravity fC _n and the nose tip 51 is The angle difference θ (not shown) becomes smaller as it almost coincides with the direction 60.

  The results of the evaluation experiment described above are shown in FIG. In this experimental example, a tendency similar to that of Experimental Example 1 was observed, and in the proposed method (5 points) using this embodiment, a very high estimation accuracy was recognized at least as compared with the elliptical fitting.

Then, referring to the positional relationship of the five points of the nose tip 51, the front center of gravity fC _n , the total center of gravity C, the rear center of gravity rC _n and the base of the tail 52, it is good to know what posture the mouse 12 takes as a whole. I can grasp. Accordingly, the angle difference between the head direction 63 obtained as described above and the body part direction (e.g., estimated from the positional relationship between the overall center of gravity C and the rear center of gravity rC _n (or tail base 52)). That is, the tilt angle of the head with respect to the trunk of the mouse 12 can be obtained with high accuracy.

In the present experimental example, similar effects can be obtained even if the first embodiment (k = 2) is used. The result obtained by using the first embodiment is shown by a one-dot chain line in FIG. As shown in the figure, a good result is obtained although it is slightly inferior to the present embodiment in which k = 3. This is considered to be because the mouse 12 is a moving body in which the ratio of the head length to the body length (head torso length) is relatively large. That is, the position of the front center of gravity fC _n which is shifted due to the decrease in the number of divisions of the mouse region 13 from 3 to 2 is near the back of the head (slightly behind FIG. 7 (d)) in the mouse 12, and thus the neck It is hard to be affected by the bending of On the other hand, it is considered that an increase in the number of divisions in the estimation of the head orientation is more effective for a moving body (for example, a cat or the like) that has a small ratio of the head length to the body length. Thus, it is preferable that the value of k is appropriately determined depending on the form of the moving object to be observed and the characteristics of the operation. According to the verification performed by the present inventors, when the purpose is to track the nose tip 51 or to estimate the head direction with respect to the mouse 12, the value of k seems to be most suitable. If the value of k is increased too much, the mouse region 13 may be divided in the body width direction.

As described above, according to the present embodiment, the orientation of the head of the mouse can be estimated with higher accuracy than before. That is, it becomes possible to track the line of sight of the mouse. Furthermore, by obtaining the tilt angle of the head with respect to the body part, it is possible to correctly detect an important operation such as moving the line of sight largely from the traveling direction and directing attention to the subject. As a result, it is possible to provide an evaluation method that is extremely useful in analyzing social behaviors of mice that have attracted attention in recent years.
Further, by specifying the rear center of gravity rC _n , it is possible to detect not only the direction of the head but also the posture such as the extension of the body.

[Example of change]
The present invention is not limited to the above-described embodiments, and appropriate modifications are allowed within the scope of the gist of the present invention.
For example, in each of the above-described embodiments, k-means clustering is used as a method for dividing the mouse region 13, but region division can also be performed using another machine learning algorithm. Examples of such an algorithm include the Ward method and a mixed Gaussian distribution.
Furthermore, in k-means clustering, examples have been given in which the value of k is set to 2 or 3, but the specific value of k is appropriately changed depending on the shape of the moving object to be observed and the purpose of observation. Just do it.
In addition, the estimation of the head orientation is useful for detecting estrus behavior and social behavior of livestock, but when such analysis is not performed, the step of specifying the front and back centroids may be omitted.

[Application of the present invention]
In each of the above-described embodiments, the mouse 12 is used as the moving body. However, the moving body of the present invention is not limited to this, and not only rats but also quadruped animals such as dogs, cats and horses, birds, and some cells. (Cells that move with polarity in a predetermined environment, such as nerve cells, cancer cells, and immune cells). As an example of the above birds, in some poultry such as chickens, the whole group gathers toward the breeding wall induced by the running behavior of an individual caused by some kind of stimulus, and this causes many There is a problem that individuals are overwhelmed. According to the present invention, it is possible to prevent economic loss due to crushing of poultry by a method such as notifying the breeding manager of a warning by detecting such a collective action.

DESCRIPTION OF SYMBOLS 10 ... Observation box 11 ... Camera 12 ... Mouse 13 ... Mouse area | region 20 ... Information processing apparatus 21 ... CPU
22 ... Memory 23 ... Display 24 ... Input 25 ... I / F
DESCRIPTION OF SYMBOLS 30 ... Memory | storage part 32 ... Behavior analysis program 41 ... Shooting control part 42 ... Image acquisition part 43 ... Pre-processing part 44 ... Whole gravity center specific part 45 ... Area division | segmentation part 46 ... End point specific part 47 ... Frame information storage part 51, 51a ... Nose tip 52, 52a ... Tail root 60 ... Actual head orientation 61, 62, 63 ... Estimated head orientation C ... Overall center of gravity fC ... Cluster center fCn ... Front center of gravity F ... Front region M ... Middle Area rC ... Cluster center rCn ... Rear center of gravity R ... Rear area θ ... Angular difference

Claims (13)

A recognition device that automatically recognizes the motion of a moving object from a plurality of frames of images of the moving object imaged by an imaging unit,
a) an overall center of gravity specifying means for specifying an overall center of gravity that is a center of gravity of the moving object region corresponding to the moving object in the image;
b) area dividing means for dividing the moving object area into a plurality of areas using a machine learning algorithm;
c) end point specifying means for specifying a point farthest from the overall center of gravity as an end point of the moving object in a predetermined area among the plurality of areas divided by the area dividing means;
A motion recognition apparatus comprising: The moving body is an animal having a longitudinal axis;
The area dividing means divides the moving object area in the front-rear direction,
The end point specifying means specifies, as an end point of the moving object, a point farthest from the overall center of gravity in an end region that is a region located at one end in the front-rear direction among the plurality of regions. The motion recognition apparatus according to claim 1. The motion recognition apparatus according to claim 2, further comprising: d) an end portion center of gravity specifying unit that specifies a center of gravity of the end region.   4. The motion recognition according to claim 3, wherein the end portion center-of-gravity specifying unit sets the center of gravity of the end region specified immediately before as an initial value of the center of gravity of the end region to be specified in a next frame. apparatus.   The motion recognition apparatus according to claim 1, wherein the machine learning algorithm used by the region dividing unit is k-means clustering.   The motion recognition apparatus according to claim 1, wherein the region dividing unit determines the number of divisions of the moving object region based on a result of a preliminary experiment. A recognition method for automatically recognizing a motion of a moving object from a plurality of frames of images of the moving object imaged by an imaging unit,
a) an overall center of gravity specifying step for specifying an overall center of gravity that is a center of gravity of a moving object region corresponding to the moving object in the image;
b) a region dividing step of dividing the moving object region into a plurality of regions using a machine learning algorithm;
c) an end point specifying step for specifying a point farthest from the overall center of gravity as an end point of the moving body in a predetermined region among the plurality of regions divided in the region dividing step;
A motion recognition method comprising: The moving body is an animal having a longitudinal axis;
In the region dividing step, the moving object region is divided in the front-rear direction,
In the end point specifying step, a point farthest from the overall center of gravity is specified as an end point of the moving body in an end region that is a region located at one end in the front-rear direction among the plurality of regions. The motion recognition method according to claim 7.
The motion recognition method according to claim 8, further comprising: d) an end portion center of gravity specifying step of specifying a center of gravity of the end region.   The operation according to claim 9, wherein, in the end portion center of gravity specifying step, the center of gravity of the end region specified immediately before is set as an initial value of the center of gravity of the end region to be specified in the next frame. Recognition method.   The motion recognition method according to claim 7, wherein the machine learning algorithm used in the region dividing step is k-means clustering.   The motion recognition method according to claim 7, wherein, in the region dividing step, the number of divisions of the moving object region is determined based on a result of a preliminary experiment.   The control program for functioning a computer as each means of the action recognition apparatus in any one of Claims 1-6.