CN114333068A - Training method and training device - Google Patents

Abstract

The application provides a training method and a training device. The method comprises the following steps: acquiring a training sample, wherein the training sample is an image sequence for recording the movement of the rodent; inputting the training sample into a neural network model to obtain a recognition result of the rodent posture, wherein the recognition result comprises key points in the rodent posture; and training the neural network model by using a gradient descent method by using a loss function according to the recognition result of the posture of the rodent. Wherein the loss function includes a temporal constraint term for constraining the position of keypoints in the rodent's pose between adjacent image frames in the sequence of images. When the neural network model trained according to the method is used for recognizing the postures of rodents such as mice and the like, the phenomenon that the recognition result shakes in the time domain can be effectively avoided.

Description

Training method and training device

technical field

The present application relates to the technical field of artificial intelligence, and in particular, to a training method and a training device.

Background technique

Gesture recognition refers to the use of neural network models to identify and/or extract the actions and/or key points of creatures in images or videos.

In the prior art, when identifying key points in an image sequence in which rodents are moving, the key points in a single image or a single image frame are usually extracted. For example, the key points can be joints of rodents, etc. . The recognition results obtained in this way are prone to jitter, which makes the movement trajectory of key points not smooth enough.

SUMMARY OF THE INVENTION

In view of this, the embodiments of the present application provide a training method and a training device, so as to improve the accuracy and recognition efficiency of a neural network model in rodent gesture recognition, and ensure the continuity of recognition results in the time domain.

In a first aspect, a training method is provided, the method comprising: acquiring a training sample, where the training sample is an image sequence recording rodent movements; inputting the training sample into a neural network model to obtain the rodent The recognition result of the posture of the rodent, and the recognition result includes the key points in the posture of the rodent; according to the recognition result of the posture of the rodent, the neural network model is subjected to the gradient descent method using the loss function. train. Wherein, the loss function includes a time constraint term, and the time constraint term is used to constrain the positions of key points in the pose of the rodent between adjacent image frames in the image sequence.

Optionally, the neural network model includes HRNet network.

Optionally, before the training of the neural network model, the training method further includes: according to the error between the position of the key point obtained by using the tracking method and the position of the key point in the recognition result, determining the time constraints.

Optionally, the tracking method includes Lucas-Kanade optical flow method.

中的第一帧

作为初始帧，进行前向跟踪，得到前向跟踪结果

确定所述前向跟踪结果与所述跟踪样本中的第m个图像帧的识别结果的差值为：

其中ω为每一个图像帧上的关键点的个数；将所述跟踪样本的识别结果中的最后一帧

作为终止帧，进行后向跟踪，得到后向跟踪结果

确定所述后向跟踪结果与所述跟踪样本中的第1个图像帧的识别结果的差值为

将所述跟踪样本的识别结果中的第一帧

作为初始帧

进行前向跟踪，得到前向跟踪结果

以所述前向跟踪结果

作为初始帧，进行后向跟踪，得到第二后向跟踪结果

所述第二后向跟踪结果与所述初始帧

的差值为

确定时间约束项为：

其中E₁为阈值。Optionally, the determining the time constraint item according to the error between the position of the key point obtained by using the tracking method and the position of the key point in the recognition result includes: selecting m samples from the training samples as tracking sample, where m is a positive integer greater than or equal to 2; the identification result of the tracking sample

first frame in

As the initial frame, forward tracking is performed to obtain the forward tracking result

It is determined that the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample is:

where ω is the number of key points on each image frame; the last frame in the recognition result of the tracking sample

As the termination frame, backward tracking is performed to obtain the backward tracking result

Determine the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample as

The first frame of the identification result of the tracking sample

as initial frame

Perform forward tracking and get forward tracking results

Take the forward trace result

As the initial frame, backward tracking is performed to obtain the second backward tracking result

the second backtracking result and the initial frame

The difference is

Determine the time constraints as:

where E ₁ is the threshold.

Optionally, the loss function further includes a space constraint term, and the space constraint term is used to define the positions of the key points in the pose of the rodent in the same frame of image.

Optionally, before the training of the neural network model, the training method further includes: determining the space constraint item according to the difference between the positions of a plurality of key points in the recognition result.

中两两关键点的距离

所述距离

符合高斯分布，确定所述距离

的均值μ和方差σ²，其中ω为每一个图像帧上的关键点的个数；确定空间约束项为：Optionally, the determining the space constraint item according to the difference between the positions of multiple key points in the recognition result includes: selecting p samples from the training samples, where p is greater than or A positive integer equal to 2; determine the recognition result of the p samples

The distance between two key points in the middle

the distance

fit a Gaussian distribution, determine the distance

The mean μ and variance σ ² of , where ω is the number of key points on each image frame; the spatial constraint term is determined as:

Optionally, the loss function further includes an error constraint term, and the error constraint term is used to constrain the errors of the key points in the pose of the rodent in the recognition result and the labeling result.

Optionally, the error loss term is a mean square error loss term.

In a second aspect, a training device is provided, comprising: an acquisition module for acquiring training samples, where the training samples are image sequences recording rodent movements; an input module for inputting the training samples into a neural network model, Obtain the recognition result of the posture of the rodent, where the recognition result includes key points in the posture of the rodent; the training module is configured to use a loss function according to the recognition result of the posture of the rodent, using a loss function, The neural network model is trained using gradient descent. Wherein, the loss function includes a time constraint term, and the time constraint term is used to constrain the positions of key points in the pose of the rodent between adjacent image frames in the image sequence.

Optionally, the neural network model includes HRNet network.

Optionally, before the training of the neural network model, the training device further includes: a first determination module, configured to determine the difference between the position of the key point obtained by using the tracking method and the key point in the identification result. The error of the position determines the time constraint term.

Optionally, the tracking method includes Lucas-Kanade optical flow method.

中的第一帧

作为初始帧，进行前向跟踪，得到前向跟踪结果

确定所述前向跟踪结果与所述跟踪样本中的第m个图像帧的识别结果的差值为：

其中ω为每一个图像帧上的关键点的个数；将所述跟踪样本的识别结果中的最后一帧

作为终止帧，进行后向跟踪，得到后向跟踪结果

确定所述后向跟踪结果与所述跟踪样本中的第1个图像帧的识别结果的差值为

将所述跟踪样本的识别结果中的第一帧

作为初始帧

进行前向跟踪，得到前向跟踪结果

以所述前向跟踪结果

作为初始帧，进行后向跟踪，得到第二后向跟踪结果

所述第二后向跟踪结果与所述初始帧

的差值为

确定时间约束项为：

其中E₁为阈值。Optionally, the first determining module is configured to: select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2;

first frame in

As the initial frame, forward tracking is performed to obtain the forward tracking result

It is determined that the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample is:

where ω is the number of key points on each image frame; the last frame in the recognition result of the tracking sample

As the termination frame, backward tracking is performed to obtain the backward tracking result

Determine the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample as

The first frame of the identification result of the tracking sample

as initial frame

Perform forward tracking and get forward tracking results

Take the forward trace result

As the initial frame, backward tracking is performed to obtain the second backward tracking result

the second backtracking result and the initial frame

The difference is

Determine the time constraints as:

where E ₁ is the threshold.

Optionally, the loss function further includes a space constraint term, and the space constraint term is used to define the positions of the key points in the pose of the rodent in the same frame of image.

Optionally, the training device further includes: a second determination module, configured to determine the space constraint item according to the difference between the positions of the multiple key points in the recognition result.

中两两关键点的距离

所述距离

符合高斯分布，确定所述距离

的均值μ和方差σ²，其中ω为每一个图像帧上的关键点的个数；确定空间约束项为：Optionally, the second determining module is configured to: select p samples from the training samples, where p is a positive integer greater than or equal to 2; determine the identification results of the p samples

The distance between two key points in the middle

the distance

fit a Gaussian distribution, determine the distance

The mean μ and variance σ ² of , where ω is the number of key points on each image frame; the spatial constraint term is determined as:

Optionally, the loss function further includes an error constraint term, and the error constraint term is used to constrain the errors of the key points in the pose of the rodent in the recognition result and the labeling result.

Optionally, the error loss term is a mean square error loss term.

In the present application, by introducing time constraints in the training process of the neural network model, the neural network model can have a high accuracy rate when processing occlusion and blurred images, and at the same time, it can effectively suppress the recognition results of the neural network model in the time domain. jitter phenomenon.

Description of drawings

FIG. 1 is a schematic flowchart of a training method provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of a method for determining a time constraint item according to an embodiment of the present application.

FIG. 3 is a schematic flowchart of a method for determining a space constraint item according to an embodiment of the present application.

FIG. 4 is a schematic flowchart of a method for determining an error constraint term provided by an embodiment of the present application.

FIG. 5 is a schematic block diagram of a training apparatus provided by an embodiment of the present application.

FIG. 6 is a schematic block diagram of a training apparatus provided by another embodiment of the present application.

FIG. 7 is a schematic block diagram of an application scenario of an embodiment of the present application.

Detailed ways

The methods and apparatuses in the embodiments of the present application can be applied to various scenarios based on gesture recognition of rodents in image sequences. The sequence of images can be multiple image frames in the video. The multiple image frames may be consecutive multiple image frames in the video. The image sequence may also be multiple images of the animal captured by an image capturing device such as a camera. The rodent may be, for example, a mouse or the like.

In order to facilitate the understanding of the embodiments of the present application, the background of the present application is firstly described in detail with examples.

The behavior of biological neurons is closely related to the activities of animals, and changes in animal posture usually cause corresponding changes in neurons. Therefore, the exploration of the connection and interaction of complex networks of neurons under specific behaviors is of great importance to the fields of neuroscience and medicine. Quantitative analysis methods are generally used in the art, that is, by acquiring animal posture information and neuron behaviors, to determine the corresponding relationship.

The behavior of animal neurons can be obtained using methods such as ray scanning and miniaturized multiphoton microscopy.

There are many ways to obtain the posture information of animals. For example, the pose information of animals can be obtained by manually annotating key points in an image sequence. However, in the face of massive data, the efficiency of manual processing is low and error-prone, and the accuracy of the obtained attitude information cannot be guaranteed.

For another example, markers (such as displacement or acceleration sensors) may also be set at key points of the animal's body, and changes in the animal's posture can be determined according to changes in information such as the position of the markers. However, in rodents, due to their small size, setting markers interferes with their natural behavior, resulting in less accurate data collected.

For another example, a depth camera can be used to locate an animal in space to obtain its pose information. However, this method is sensitive to imaging conditions and scene changes, and is not suitable for all occasions.

With the development of artificial intelligence, animal gesture recognition methods based on neural networks are gradually replacing traditional techniques. Current neural network models typically do not take into account the time-dependent movement patterns of rodent keypoints in image sequences. These neural network models have the following problems in the gesture recognition process:

When recognizing animal poses in image sequences, neural network models usually perform pose recognition based on each frame of image itself. For example, the image sequence to be recognized includes a first frame of images and a second frame of images in time sequence. The neural network model recognizes the animal pose in the first frame image according to the first frame image, and obtains a first pose recognition result corresponding to the first frame image. The animal gesture in the second frame image is recognized according to the second frame image, and the second gesture recognition result corresponding to the second frame image is obtained. By adopting the above-mentioned method of directly using the current frame image to recognize the animal pose, the accuracy of the obtained recognition result is low, and the time is not smooth enough. In addition, when the image frames in the acquired image sequence are blurred or occluded, such as when the rodent's tail is curled or occluded, the accuracy of the keypoint location information output by the neural network model is low.

In addition, the existing neural network models usually construct a loss function based on the error between the recognition result and the manual labeling result, and use the back-propagation algorithm for training. This neural network model does not consider the continuous changes of key points in the time domain during training, which leads to the problem of low accuracy when performing rodent pose recognition. On the other hand, constructing a loss function to train a neural network model using the error between the above recognition results and manual annotation results usually makes the initial training process slow.

In view of the above problems, embodiments of the present application provide a training method and a training device. The method provided by the embodiment of the present application effectively suppresses the jitter phenomenon in the time domain of the recognition result of the neural network model by introducing a time constraint in the training process of the neural network model.

The training method provided by the embodiment of the present application will be described in detail below with reference to FIG. 1 to FIG. 4 . FIG. 1 is a schematic flowchart of a training method provided by an embodiment of the present application. The training method shown in FIG. 1 may include steps S11-S13.

Step S11, acquiring training samples.

In one embodiment of the present application, the training samples may include image sequences recording rodent movements and labelling results. It can be understood that the marking result may include the position information of a preset number of key points on the rodent body. For example, the key points can be various joint points and key parts of the body, such as joint points on the limbs of a mouse, tail, eyes, nose, ears, and the like. The location information may be coordinate information of key points.

The embodiment of the present application does not limit the acquisition manner of the pre-marked result. For example, an image frame in an image sequence can be annotated frame by frame using a manual annotation method. As a possible implementation, other methods with higher confidence can also be used for labeling.

There may be many ways to obtain training samples, which are not limited in this embodiment of the present application. For example, as an implementation, an image sequence that can be directly acquired by, for example, an image acquisition device (eg, a camera, a camera, a medical imaging device, a lidar, etc.) image. For another example, training samples may be obtained from a server (eg, a local server or a cloud server, etc.). Alternatively, training samples can also be obtained on the Internet or other content platforms, for example, open-source training datasets such as MSCOCO dataset, MPII dataset, and POSETTRACK dataset can be used; .

In step S12, the training samples obtained in the foregoing step S11 are input into the neural network model to obtain the recognition result of the posture of the rodent.

The embodiments of the present application do not specifically limit the neural network model, and any neural network model capable of realizing the gesture recognition described in the present application may be used. For example, the neural network model can be a 2D convolutional neural network such as VGG, ResNet, HRNet, etc.

Optionally, HRNet (HighResolution Network) can maintain high resolution throughout the feature extraction process, and can perform poor fusion of features of different resolutions during the feature extraction process. It is especially suitable for use in semantic segmentation, human pose, image classification, facial landmark detection, general target recognition and other scenarios.

Wherein, the identification result may include position information of a preset number of rodent body key points identified by the neural network model (also referred to as identification positions for short).

In step S13, according to the identification result in step S12, using the loss function, the neural network model is trained by using the gradient descent method.

Wherein, the loss function may include a temporal constraint term L _temporal .

The method for determining the time constraint item L _temporal will be described in detail below with reference to FIG. 2 . Referring to FIG. 2, FIG. 2 shows a method for determining a time constraint item.

A temporal constraint term L _temporal may be used to constrain the positions of keypoints in the rodent's pose between adjacent image frames in the image sequence.

In some embodiments, the temporal constraint term L _temporal may be determined according to the error between the position information of the key points obtained by using the tracking method and the position information of the key nodes in the recognition result.

In the training method provided by the embodiment of the present application, the tracking method may be an unsupervised tracking method, for example, the Lucas-Kanade optical flow method.

The method shown in FIG. 2 may include steps S1311-S1315.

Step S1311, select m images from the training samples as tracking samples.

The m images are any m images in the training sample. The m images may be consecutive m images in the training sample. It can be understood that the m images can also be all images in the training sample.

Step S1312, taking the first image frame of the m images in the training sample as an initial frame, and using the identification result of the initial frame to perform forward tracking to obtain a first forward tracking result, and determine the first forward tracking result. The first difference between the forward tracking result and the recognition result of the mth image frame, where the first forward tracking result includes the tracking position of the key point in the mth image frame. Among them, m is a positive integer greater than or equal to 2. In other words, the first difference value may be the difference value between the tracked position and the recognized position of the same key point in the mth image frame.

(i＝1,2…,m；ω＝1,2…,h)，其中，ω为每个图像帧中的关键点的个数。For the convenience of description, the set composed of m images will be denoted as I _1,i (i=1,2,...,m), and the recognition result of the set I _1,i will be denoted as

(i=1,2...,m; ω=1,2...,h), where ω is the number of key points in each image frame.

进行前向跟踪，得到第一前向跟踪结果

确定第一前向跟踪结果和集合I_1,i中的第m帧的识别结果

之间的差值F₁为：Take the first frame of the m images as the initial frame, and use the recognition result of the initial frame

Perform forward tracking to get the first forward tracking result

Determine the first forward tracking result and the recognition result of the mth frame in the set I _1,i

The difference between F ₁ is:

Step S1313, taking the m-th image frame in the m images as a termination frame, and using the identification result of the termination frame to perform backward tracking to obtain a first backward tracking result, where the first backward tracking result includes: Tracked locations of keypoints in the first image frame. It can be understood that the m th image may also be referred to as the last image frame in the m images. A second difference between the first backward tracking result and the identification result of the first image frame is determined. In other words, the second difference value may be the difference value between the tracked position and the recognized position of the same key point in the first image frame.

进行后向跟踪，得到第一后向跟踪结果

确定第一后向跟踪结果

和集合I_1,i中的第一帧的识别结果

之间的差值F₂为：Take the last frame in the m images as the end frame, and use the recognition result of the end frame

Perform backward tracking to get the first backward tracking result

Determine the first backtracking result

and the recognition result of the first frame in the set I _1,i

The difference between F ₂ is:

利用该初始帧的识别结果

进行前向跟踪，得到第一前向跟踪结果

再以第一前向跟踪结果

作为终止帧，进行后向跟踪，确定第二后向跟踪结果

确定第二后向跟踪结果

与初始帧

的差值F₃为：In step S1314, the first frame of the m images is taken as the initial frame

Use the recognition result of this initial frame

Perform forward tracking to get the first forward tracking result

Then trace the result in the first forward direction

As a termination frame, backward tracking is performed, and the second backward tracking result is determined

Determine the second backtracking result

with the initial frame

_The difference F3 is:

In step S1315, a time constraint item is determined.

与初始帧

的差值F₃。即所述时间约束项为

When both the first difference and the second difference are less than or equal to a preset threshold, the time constraint is determined to be 0; when the first difference and/or the second difference is greater than the predetermined threshold When the preset threshold is set, it is determined that the time constraint item is the second backward tracking result

with the initial frame

The difference F ₃ . That is, the time constraint term is

Among them, E ₁ is a preset threshold, and the preset threshold is related to the movement characteristics of living things. It should be noted that, compared with the prediction results of the neural network model, the tracking results obtained by using the tracking method can ensure that the tracking position of the same key point changes smoothly in the time domain. Therefore, when the difference value (such as the first difference value or the second difference value) is smaller than the preset threshold, it means that the recognition result is close to the tracking result, and the recognition result of the neural network model is relatively smooth in the time domain, and no time constraint can be set at this time. item. And when the difference is greater than the preset threshold, it means that the recognition result is quite different from the tracking result. That is to say, the recognition result is relatively jittery in the time domain. At this time, the neural network model can be trained by setting the time constraint item, so as to make the recognition result output by the neural network model smoother.

The embodiments of the present application do not specifically limit the manner of determining the time constraint item. For example, the first difference can be used as a time constraint. For another example, the second difference can be used as a time constraint. For another example, backward tracking can be performed on the first forward tracking result to obtain a second backward tracking result; according to the difference between the second backward tracking result and the recognition result of the first image frame, the time constraint is determined. . For another example, forward tracking may be performed on the first backward tracking result to obtain a second forward tracking result; according to the difference between the second forward tracking result and the recognition result of the first image frame, the time constraint is determined. .

In some embodiments, the loss function further includes a spatial constraint term, the spatial constraint term is used to define the positions of the key points in the pose of the rodent in the same frame of image.

Referring to Fig. 3, Fig. 3 shows a method for determining a space constraint item.

The spatial constraint term L _spatial can be used to define the positions of key points in the pose of the creature in the same image frame. In some embodiments, the spatial constraint term L _spatial may be determined according to the difference between the positions of a plurality of key points in the recognition result.

The method for determining a spatial constraint item L spatial provided by an embodiment of the present application may include steps _S1321 -S1322.

Step S1321, select p samples from the training samples, where p is a positive integer greater than or equal to 2;

The p images are any p images in the training sample. The p images may be consecutive p images in the training sample. It can be understood that the p images can also be all images in the training sample. Among them, p is a positive integer greater than or equal to 2.

(i＝1,2…,p；ω＝1,2…,h)，其中，ω为每个图像帧中的关键点的个数。For the convenience of description, the set composed of p images will be denoted as I _2,j (j=1,2,...,p), and the recognition result of the set I _2,j will be denoted as

(i=1,2...,p; ω=1,2...,h), where ω is the number of key points in each image frame.

Step S1322, determining the distance between two key points in the same image in the training sample.

(j＝1,2,…,p,ω＝1,2,…h)中两两关键点的距离

(j＝1,2,…,p,ω＝1,2,…h)Determine the recognition results of the p samples

(j=1,2,...,p,ω=1,2,...h) The distance between the key points in pairs

(j=1,2,…,p,ω=1,2,…h)

Step S1323, determine the space constraint item.

符合高斯分布，确定所述距离

的均值μ和方差σ²，其中ω为每一个图像帧上的关键点的个数；above distance

fit a Gaussian distribution, determine the distance

The mean μ and variance σ ² of , where ω is the number of key points on each image frame;

Determine the space constraints as:

It should be noted that the method for determining the spatial constraint item L spatial provided in the above steps S1321- _S1323 is only an example, and can also be determined in other ways. For example, the space constraint item may also be determined based on the error between the distance between the key points in the recognition result and the distance between the key points in the corresponding labeling result, which is not limited in this application.

In some embodiments, the loss function may also include an error constraint term L _MSE . In some embodiments, the error constraint term L _MSE may be determined according to the error between the recognition result of the training sample and the position information of the same key point in the labeling result. Referring to FIG. 4, taking the mean square error as an example, determining the error constraint term may include steps S1331-S1333.

Step S1331, select n images from the training samples obtained in step S11 to form a sample set I _3,k (k=1, 2, . . . , n), where n is a positive integer greater than or equal to 1.

The n images are any n images in the training sample. The n images may be consecutive n images in the training sample. It can be understood that the n images can also be all images in the training sample.

(k＝1,2…,n；ω＝1,2…,h)以及标注结果

(k＝1,2…,n；ω＝1,2…,h)。Step S1332, determine the identification result of the sample set I _3,k

(k=1,2...,n; ω=1,2...,h) and labeling results

(k=1,2...,n; ω=1,2...,h).

和标注结果

的均方误差，确定误差损失项为：Step S1333, calculate the aforementioned recognition result

and labeling results

The mean square error of , determines the error loss term as:

For the error loss term, in addition to the mean square error loss, cross entropy loss, 0-1 loss, absolute value loss, etc. commonly used in the art can also be used. The methods shown in the above steps S1331-S1333 are only examples, and do not have a limiting effect on the protection scope of the present application.

In some embodiments, the loss function may also be determined by a weighted summation of the aforementioned error constraint term L _MSE , temporal constraint term L _temporal , and spatial constraint term L _spatial . That is, the loss function L=L _MSE + aL _temporal + bL _spatical , where a and b are hyperparameters whose values are greater than or equal to 0.

An embodiment of the training device provided by the present application will be described in detail below with reference to FIG. 5 . It should be understood that the descriptions of the apparatus embodiments and the foregoing method embodiments correspond to each other. Therefore, the parts not described in detail can refer to the foregoing method embodiments.

FIG. 5 is a schematic block diagram of a training apparatus 50 provided by an embodiment of the present application. It should be understood that the apparatus 50 shown in FIG. 5 is only an example, and the apparatus 50 in this embodiment of the present invention may further include other modules or units.

It should be understood that the apparatus 50 can perform various steps in the methods of FIGS. 1-4 , and in order to avoid repetition, details are not described herein again.

As a possible implementation manner, the device includes:

The acquiring module 51 is used for acquiring training samples.

Wherein, the training samples and the acquisition method thereof may be consistent with step S11 of the foregoing method, and details are not described herein again.

The input module 52 is configured to input the training sample into a neural network model to obtain a recognition result of the posture of the rodent, where the recognition result includes key points in the posture of the rodent.

The training module 53 is configured to use a loss function to train the neural network model by using a gradient descent method according to the recognition result of the posture of the rodent.

Optionally, the neural network model includes HRNet network.

Optionally, before the training of the neural network model, the training device further includes: a first determination module, configured to determine the difference between the position of the key point obtained by using the tracking method and the key point in the identification result. The error of the position determines the time constraint term.

Optionally, the tracking method includes Lucas-Kanade optical flow method.

(i＝1.2…m,ω＝1,2,…,h)中的第一帧

作为初始帧，进行前向跟踪，得到前向跟踪结果

确定所述前向跟踪结果与所述跟踪样本中的第m个图像帧的识别结果的差值为：

其中ω为每一个图像帧上的关键点的个数；将所述跟踪样本的识别结果中的最后一帧

作为终止帧，进行后向跟踪，得到后向跟踪结果

确定所述后向跟踪结果与所述跟踪样本中的第1个图像帧的识别结果的差值为

将所述跟踪样本的识别结果中的第一帧

作为初始帧

进行前向跟踪，得到前向跟踪结果

以所述前向跟踪结果

作为初始帧，进行后向跟踪，得到第二后向跟踪结果

所述第二后向跟踪结果与所述初始帧

的差值为

确定时间约束项为：

其中E₁为阈值。Optionally, the first determining module is configured to: select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2;

The first frame in (i=1.2...m,ω=1,2,...,h)

As the initial frame, forward tracking is performed to obtain the forward tracking result

It is determined that the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample is:

where ω is the number of key points on each image frame; the last frame in the recognition result of the tracking sample

As the termination frame, backward tracking is performed to obtain the backward tracking result

Determine the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample as

The first frame of the identification result of the tracking sample

as initial frame

Perform forward tracking and get forward tracking results

Take the forward trace result

As the initial frame, backward tracking is performed to obtain the second backward tracking result

the second backtracking result and the initial frame

The difference is

Determine the time constraints as:

where E ₁ is the threshold.

Optionally, the loss function further includes a space constraint term, and the space constraint term is used to define the positions of the key points in the pose of the rodent in the same frame of image.

Optionally, the training device further includes: a second determination module, configured to determine the space constraint item according to the difference between the positions of the multiple key points in the recognition result.

(j＝1,2,…,p,ω＝1,2,…h)中两两关键点的距离

(j＝1,2,…,p,ω＝1,2,…h)，所述距离

符合高斯分布，确定所述距离

的均值μ和方差σ²，其中ω为每一个图像帧上的关键点的个数；确定空间约束项为：Optionally, the second determining module is configured to: select p samples from the training samples, where p is a positive integer greater than or equal to 2; determine the identification results of the p samples

(j=1,2,...,p,ω=1,2,...h) The distance between the key points in pairs

(j=1,2,...,p,ω=1,2,...h), the distance

fit a Gaussian distribution, determine the distance

The mean μ and variance σ ² of , where ω is the number of key points on each image frame; the spatial constraint term is determined as:

Optionally, the loss function further includes an error constraint term, and the error constraint term is used to constrain the error of the key points in the posture of the creature in the recognition result and the labeling result.

Optionally, the error loss term is a mean square error loss term.

It should be understood that the apparatus 50 for training a neural network model here is embodied in the form of functional modules. The term "module" here can be implemented in the form of software and/or hardware, which is not specifically limited. For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implement the above-mentioned functions. The hardware circuits may include application specific integrated circuits (ASICs), electronic circuits, processors (eg, shared processors, proprietary processors, or group processors) for executing one or more software or firmware programs etc.) and memory, merge logic and/or other suitable components to support the described functions.

As an example, the apparatus 50 for training a neural network model provided by the embodiment of the present invention may be a processor or a chip, so as to execute the method described in the embodiment of the present invention.

FIG. 6 is a schematic block diagram of a training apparatus 60 provided by another embodiment of the present application. The apparatus 60 shown in FIG. 6 includes a memory 61 , a processor 62 , a communication interface 63 and a bus 64 . The memory 61 , the processor 62 , and the communication interface 63 are connected to each other through the bus 64 for communication.

The memory 61 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 61 may store a program, and when the program stored in the memory 61 is executed by the processor 62, the processor 62 is configured to execute each step of the training method provided in this embodiment of the present invention. For example, the implementation shown in FIG. 1 to FIG. 4 may be executed. each step of the example.

The processor 62 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits for executing relevant programs to The training method of the method embodiment of the present invention is implemented.

The processor 62 can also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the training method provided by the embodiment of the present invention may be completed by an integrated logic circuit of hardware in the processor 62 or an instruction in the form of software.

The above-mentioned processor 62 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates. Or transistor logic devices, discrete hardware components. Various methods, steps, and logical block diagrams disclosed in the embodiments of the present invention can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 61, and the processor 62 reads the information in the memory 61 and, in combination with its hardware, completes the functions required to be performed by the units included in the gesture recognition apparatus in the embodiment of the present invention, or performs the training of the method embodiment of the present invention method. For example, various steps/functions of the embodiments shown in FIGS. 1-4 may be performed.

The communication interface 63 can use, but is not limited to, a transceiver such as a transceiver to implement communication between the device 60 and other devices or a communication network.

Bus 64 may include a pathway for communicating information between various components of device 60 (eg, memory 61, processor 62, communication interface 63).

It should be understood that the apparatus 60 shown in the embodiment of the present invention may be a processor or a chip, so as to execute the method described in the embodiment of the present invention.

It should be understood that the processor in this embodiment of the present invention may be a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), application-specific integrated circuits (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The specific application of the embodiment of the present application will be introduced below with reference to the application scenario of FIG. 7 . It should be noted that the following description about FIG. 7 is only an example and not a limitation, and the method in this embodiment of the present application is not limited thereto, and may also be applied to other gesture recognition scenarios.

The application scenario in FIG. 7 may include an image acquisition device 71 and an image processing device 72 .

Among them, the image acquisition device 71 can be used to acquire a sequence of images of rodents. The image processing apparatus 72 may be integrated in an electronic device, and the electronic device may be a server or a terminal or other device, which is not limited in this embodiment of the present application. For example, a server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a basic cloud that provides cloud services, cloud computing, cloud storage, cloud communication, and big data and artificial intelligence platforms. Cloud servers for computing services. Terminals can be smartphones, tablets, computers, and smart IoT devices. The terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

A neural network model can be deployed in the image processing device 72, which can be used to identify the image by using the neural network model after using the image sequence obtained by the above-mentioned image acquisition device 71 to obtain position information of key points in the image to be processed. The location information of the key points may include, for example, the location coordinate information of the rodent body joints, torso or facial features, and the like.

The above electronic device may also use the image acquisition device 71 to acquire training samples, and use the loss function to train the neural network model according to the recognition results of the training samples and the results of human annotation. The image processing device 72 can also recognize the image to be processed through the trained neural network model, thereby achieving the purpose of accurately recognizing the image.

The above-described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. The description order of the above embodiments is not intended to limit the preferred order of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

It should be understood that, in this embodiment of the present application, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is only determined according to A, and B may also be determined according to A and/or other information.

It should be understood that the term "and/or" in this document is only an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, and A and B exist simultaneously , there are three cases of B alone. In addition, the character "/" in this text generally indicates that the related objects are an "or" relationship.

It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be read by a computer or a data storage device such as a server, a data center, etc. that includes one or more available media integrated. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, digital video discs (DVDs)), or semiconductor media (eg, solid state disks (SSDs)) )Wait.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited to this. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (20)

1. a training method, is characterized in that, comprises: obtaining a training sample, the training sample is an image sequence recording rodent movement; Inputting the training sample into a neural network model to obtain a recognition result of the rodent's posture, where the recognition result includes key points in the rodent's posture; According to the recognition result of the posture of the rodent, use the loss function to train the neural network model by using the gradient descent method; Wherein, the loss function includes a time constraint term, and the time constraint term is used to constrain the positions of key points in the pose of the rodent between adjacent image frames in the image sequence. 2. The method of claim 1, wherein the neural network model comprises a HRNet network. 3. The training method according to claim 2, wherein, before the neural network model is trained, the training method further comprises: The time constraint item is determined according to the error between the position of the key point obtained by the tracking method and the position of the key point in the recognition result. 4. The training method according to claim 3, wherein the tracking method comprises a Lucas-Kanade optical flow method. 5. The training method according to claim 3, wherein the time constraint is determined according to the error between the position of the key point obtained by using the tracking method and the position of the key point in the identification result, comprising: Select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The identification results of the tracking samples

first frame in

As the initial frame, forward tracking is performed to obtain the forward tracking result

It is determined that the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample is:

where ω is the number of key points on each image frame; The last frame in the recognition result of the tracking sample

As the termination frame, backward tracking is performed to obtain the backward tracking result

Determine the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample as

The first frame of the identification result of the tracking sample

as initial frame

Perform forward tracking and get forward tracking results

Take the forward trace result

As the initial frame, backward tracking is performed to obtain the second backward tracking result

the second backtracking result and the initial frame

The difference is

Determine the time constraints as:

where E ₁ is the threshold. 6. training method according to claim 1, is characterized in that, The loss function further includes a spatial constraint term, the spatial constraint term is used to define the position of the key points in the pose of the rodent in the same frame of image. 7. The training method according to claim 6, wherein, before the neural network model is trained, the training method further comprises: The space constraint item is determined according to the difference between the positions of a plurality of key points in the recognition result. 8. The training method according to claim 7, wherein the determining the space constraint item according to the difference between the positions of multiple key points in the recognition result comprises: select p samples from the training samples, where p is a positive integer greater than or equal to 2; Determine the recognition results of the p samples

The distance between two key points in the middle

the distance

fit a Gaussian distribution, determine the distance

The mean μ and variance σ ² of , where ω is the number of key points on each image frame; Determine the space constraints as:

9 . The training method according to claim 1 , wherein the loss function further comprises an error constraint term, and the error constraint term is used to constrain the key points in the pose of the rodent in the recognition result. 10 . and errors in the annotation results. 10. The training method according to claim 9, wherein the error loss term is a mean square error loss term. 11. A training device, characterized in that, comprising: an acquisition module for acquiring a training sample, where the training sample is an image sequence for recording rodent movements; an input module, configured to input the training sample into a neural network model to obtain a recognition result of the posture of the rodent, where the recognition result includes key points in the posture of the rodent; a training module for training the neural network model by using a loss function and gradient descent method according to the recognition result of the posture of the rodent; Wherein, the loss function includes a time constraint term, and the time constraint term is used to constrain the positions of key points in the pose of the rodent between adjacent image frames in the image sequence. 12. The training device according to claim 11, wherein the neural network model comprises a HRNet network. 13. The training device according to claim 12, wherein before the training of the neural network model, the training device further comprises: The first determination module is configured to determine the time constraint item according to the error between the position of the key point obtained by using the tracking method and the position of the key point in the recognition result. 14. The training device according to claim 13, wherein the tracking method comprises a Lucas-Kanade optical flow method. 15. The training device according to claim 14, wherein the first determination module is used for: Select m samples from the training samples as tracking samples, where m is a positive integer greater than or equal to 2; The identification results of the tracking samples

first frame in

As the initial frame, forward tracking is performed to obtain the forward tracking result

It is determined that the difference between the forward tracking result and the recognition result of the mth image frame in the tracking sample is:

where ω is the number of key points on each image frame; The last frame in the recognition result of the tracking sample

As the termination frame, backward tracking is performed to obtain the backward tracking result

Determine the difference between the backward tracking result and the recognition result of the first image frame in the tracking sample as

The first frame of the identification result of the tracking sample

as initial frame

Perform forward tracking and get forward tracking results

Take the forward trace result

As the initial frame, backward tracking is performed to obtain the second backward tracking result

the second backtracking result and the initial frame

The difference is

Determine the time constraints as:

where E ₁ is the threshold. 16 . The training device according to claim 11 , wherein the loss function further comprises a spatial constraint term, and the spatial constraint term is used to limit the key points in the pose of the rodent to be in the same frame of image. 17 . s position. 17. The training device of claim 16, wherein the training device further comprises: The second determination module is configured to determine the space constraint item according to the difference between the positions of the multiple key points in the identification result. 18. The training device according to claim 17, wherein the second determination module is used for: select p samples from the training samples, where p is a positive integer greater than or equal to 2; Determine the recognition results of the p samples

The distance between two key points in the middle

the distance

fit a Gaussian distribution, determine the distance

The mean μ and variance σ ² of , where ω is the number of key points on each image frame; Determine the space constraints as:

19 . The training device according to claim 11 , wherein the loss function further comprises an error constraint term, and the error constraint term is used to constrain key points in the posture of the rodent in the recognition result. 20 . and errors in the annotation results. 20. The training device according to claim 19, wherein the error loss term is a mean square error loss term.

Images