WO2022041129A1

WO2022041129A1 - Three-dimensional capturing apparatus, method and system for ethology recording, and application of system

Info

Publication number: WO2022041129A1
Application number: PCT/CN2020/112156
Authority: WO
Inventors: 陈可; 黄康; 韩亚宁; 蔚鹏飞; 王立平
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2020-08-28
Filing date: 2020-08-28
Publication date: 2022-03-03

Abstract

A three-dimensional capturing apparatus for ethology recording, the apparatus comprising: cameras (101), a fixing frame (102), and a cage (103), wherein the cage (103) is arranged in the fixing frame (102); and a plurality of cameras (101) are respectively arranged at a plurality of positions on the fixing frame (102). A three-dimensional capturing method for ethology recording, the method comprising: calibrating a plurality of cameras (101); controlling the plurality of cameras (101) to synchronously photograph an animal to obtain a plurality of two-dimensional skeleton points of the animal; then, combining the plurality of two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence; using a triangulation algorithm to map the two-dimensional skeleton sequences into a three-dimensional space, and performing calculation to obtain three-dimensional skeletons; and connecting the three-dimensional skeletons by means of lines, and displaying, on a screen, the three-dimensional skeletons frame by frame. By means of the three-dimensional capturing apparatus, method and system for ethology recording, and the application of the system, various body actions of an animal can be completely and synchronously photographed, a two-dimensional skeleton of the body of the animal can be stably tracked and customized, and three-dimensional skeleton points can be stably reconstructed from a plurality of two-dimensional skeleton points and can be visually represented in a three-dimensional space.

Description

A three-dimensional capture device, method, system and application for animal behavior recording

technical field

The invention relates to the field of target tracking, in particular to a three-dimensional capture device, method, system and application for animal behavior recording.

Background technique

At present, the activities of animals are mainly recorded by shooting two-dimensional videos. Researchers analyze and draw conclusions through the two-dimensional videos. Animals have multiple degrees of freedom, rich and highly complex behaviors. Body parts are easily occluded, and it is difficult to guarantee the accuracy and reliability of the data obtained by recording animals by shooting two-dimensional videos.

The prior art 3D capture devices mostly use dual cameras and triple cameras for 3D capture, but the less the number of cameras, the more difficult it is to capture in all directions, the body parts of the animals being captured are easily blocked, and the cameras are large in size and difficult to install. In a cage, an excessively large camera may cause the animal's alertness, which is not conducive to the normal monitoring of animal behavior, and the camera has a low frame rate, making it difficult to quickly track animal behavior.

At present, the feature point matching algorithm is mostly used to obtain the corresponding feature points in the images captured by the two cameras, but this method is extremely unstable, and the number and matching positions of the matching feature points in each frame of the video may be different, which leads to the instability of the reconstructed 3D point cloud. , it is difficult to analyze this 3D point cloud.

SUMMARY OF THE INVENTION

In order to overcome the deficiencies of the prior art, the present invention provides a three-dimensional capture device, method, system and application for animal behavioral recording. The specific technical scheme is as follows:

A three-dimensional capture device for animal behavior recording, comprising a camera, a fixed frame, and a cage, wherein the cage is arranged in the fixed frame;

The number of the cameras is multiple, and the multiple cameras are respectively arranged at multiple positions of the fixed frame to photograph all the animals in the cage.

In a specific embodiment, at least one of the cameras is arranged at the center of the top of the fixed frame, and the cameras face vertically downward, and the rest of the cameras are arranged around the center, and The camera is disposed obliquely downward.

In a specific embodiment, the number of the cameras includes 5, and the cage includes a stainless steel glass mixing cage;

It should be noted that the number of the cameras including 5 is only a preferred number of the content of the present invention, and there are other preferred numbers, such as 2, 3, 4, 6, 7, etc.;

The stainless steel-glass blend is only one preferred material of the present invention, and there are other preferred materials, such as aluminum alloy-glass blend, wood-glass blend, rubber-glass blend, and the like.

The fixing frame includes a rectangular frame, and a plurality of the cameras are respectively arranged on each top corner of the upper surface of the rectangular frame;

It should be noted that the rectangular frame is only a preferred shape of the content of the present invention, and there are other preferred shapes, such as a trapezoidal frame, a conical frame, a spherical frame, etc.;

Arrangement on each top corner of the upper surface of the rectangular frame is only a preferred arrangement of the present invention, and there are other preferred arrangements, such as arrangement on each side of the upper surface of the rectangular frame, arrangement on all on each side of the rectangular frame perpendicular to the ground, etc.

Preferably, the fixing frame comprises an aluminum alloy fixing frame;

It should be noted that aluminum alloy is only a preferred material for the content of the present invention, and there are other preferred materials, such as stainless steel, wood, and the like.

A three-dimensional capture method for animal behavior records, comprising the following steps:

calibrating a plurality of the cameras;

Control a plurality of the cameras to synchronously shoot the active videos of the animals at different positions and angles, obtain the required multiple animal pictures according to the preset frame rate, image quality and shooting time, and compare the corresponding animal pictures between the multiple animal pictures. The feature points are marked, and a plurality of two-dimensional skeleton points of the animal are obtained by the fusion of the marks;

Train a deep neural network to complete the tracking of multiple two-dimensional skeleton points of the animal, and then combine a plurality of the two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence. Specifically, use the DeepLabCut toolbox to train a deep neural network The network makes the animal's two-dimensional skeleton sequence more robust and accurate when the ambient light changes, the image tone changes, and the background changes slightly.

Using the triangulation algorithm to map the two-dimensional skeleton sequence in combination with the calibration result into the three-dimensional space to calculate the three-dimensional skeleton, and specifically, use the SVD method to perform least squares calculation, so that the data of the three-dimensional skeleton is more accurate;

Rotate, translate, and invert a plurality of the three-dimensional skeleton coordinates without distortion, and align them to the horizontal line, take the vertical direction of the main camera as the Z-axis, and align the Z-axis to the height of the real world, so that the three-dimensional The features of the skeleton are more accurate and realistic;

The data of a plurality of the three-dimensional skeletons are read into the workspace, and a line is formed between the plurality of the three-dimensional skeletons, and displayed on the screen frame by frame.

In a specific embodiment, the method for "controlling a plurality of the cameras to synchronously capture the activity videos of animals at different positions and angles" includes: using five of the cameras, and dividing the five cameras into four pairs , one of the described cameras is the main camera, and the other four described cameras are sub-cameras, and the 5 described cameras are controlled to synchronously shoot the activity videos of animals at different positions and angles;

The method of "using the triangulation algorithm to map the two-dimensional skeleton sequence combined with the calibration result into the three-dimensional space to calculate the three-dimensional skeleton" includes: using the triangulation algorithm to reconstruct the two-dimensional skeleton sequence in pairs, and for each animal's skeleton. Four spatial point positions are reconstructed from each of the feature points, and finally the least squares method is used to optimize the spatial point positions to obtain the optimal spatial point positions, and the two-dimensional skeleton sequence composed of the optimal spatial point positions The three-dimensional skeleton is obtained by combining the calibration results and mapping them into three-dimensional space.

In a specific embodiment, the implementation method of "calibrating a plurality of the cameras" includes:

Start a plurality of the cameras, the cameras are automatically initialized, the calibration board includes a pattern for calibration, the size of the pattern is measured, and the cameras synchronously shoot several calibration boards at different positions and angles. Next, obtain a plurality of pictures of the calibration plate pattern, and obtain a plurality of rotation matrices and translation vectors of the cameras. Specifically, the patterns used for calibration include checkerboard patterns, solid circle array patterns, and the like.

In a specific embodiment, the "obtaining multiple required animal pictures according to the preset frame rate, image quality and shooting duration" further includes:

The frame rate is 40-70FPS;

The picture quality is 440*280-740*580;

The shooting time is 10-20 minutes.

A three-dimensional capture system for animal behavior records, comprising: a camera, a fixed frame, and a cage, wherein the cage is arranged in the fixed frame;

The number of the cameras is multiple, and the multiple cameras are respectively arranged at multiple positions of the fixed frame to photograph all the animals in the cage;

Also includes:

a calibration module for calibrating a plurality of the cameras;

The picture acquisition module is used to control a plurality of the cameras to synchronously shoot the motion videos of animals at different positions and angles, obtain the required animal pictures according to the preset frame rate, picture quality and shooting time, and analyze the multiple animal pictures. The corresponding feature points between the animal pictures are marked, and a plurality of two-dimensional skeleton points of the animal are obtained through the fusion of the markings;

The two-dimensional skeleton extraction module is used to train a deep neural network, complete the tracking of multiple two-dimensional skeleton points of the animal, and then combine a plurality of the two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence, specifically , using the DeepLabCut toolbox to train the deep neural network, so that the animal's two-dimensional skeleton sequence has stronger robustness and accuracy when the ambient light changes, the image tone changes, and the background slightly changes.

The three-dimensional skeleton extraction module is used to use the triangulation algorithm to map the two-dimensional skeleton sequence combined with the calibration result into the three-dimensional space to calculate the three-dimensional skeleton. more precise;

The processing module is used to rotate, translate, and invert a plurality of the three-dimensional skeleton coordinates without distortion, and align them to the horizontal line, and use the vertical direction of the main camera as the Z-axis to align the Z-axis to the height of the real world , so that the features of the three-dimensional skeleton are more accurate and real;

The visualization module is used for reading the data of a plurality of the three-dimensional skeletons into the workspace, connecting the plurality of the three-dimensional skeletons into lines, and displaying them frame by frame on the screen.

In a specific embodiment, the method of the "picture acquisition module" includes: using five cameras, dividing the five cameras into four pairs, one of the cameras is the main camera, and the other four cameras are the main camera. The described camera is a sub-camera, and 5 described cameras are controlled to synchronously shoot active videos of animals at different positions and angles;

The method of the "three-dimensional skeleton extraction module" includes: using a triangulation algorithm to reconstruct the two-dimensional skeleton sequence in pairs, reconstructing four spatial point positions for each of the feature points of the animal, and finally using the least squares method. Optimizing the spatial point positions is performed to obtain the optimal spatial point positions, and the two-dimensional skeleton sequence composed of the optimal spatial point positions is mapped to the three-dimensional space in combination with the calibration results to calculate the three-dimensional skeleton.

In a specific embodiment, the "calibration module" is further used for:

In a specific embodiment, the "picture acquisition module" further includes:

The frame rate is 40-70FPS;

The picture quality is 440*280-740*580;

The shooting time is 10-20 minutes.

In a specific embodiment, the three-dimensional capture system for animal ethology recording is applied in the ethology recording of non-human primates.

The present invention has at least the following beneficial effects:

According to a three-dimensional capture device for animal behavior recording provided by the present invention, it includes a camera, a fixed frame and a cage, the cage is arranged in the fixed frame, the number of cameras is multiple, and the multiple cameras are respectively arranged in multiple parts of the fixed frame. At least one of the cameras is set at the center of the top of the fixed frame, and the camera faces vertically downward, and the rest of the cameras are set around the center position, and the cameras are set diagonally downward; Based on the technical solution of the present invention, researchers can use multiple cameras to quickly track animal behaviors from different angles and positions, and to photograph and store various body movements of animals in an all-round, complete and synchronous manner.

According to a three-dimensional capture method for animal behavior records using the animal behavior record three-dimensional capture device provided by the present invention, the method includes the following steps: calibrating a plurality of cameras; Activity video, according to the preset frame rate, image quality and shooting time to obtain the required multiple animal pictures, and mark the corresponding feature points between the multiple animal pictures, and obtain multiple two-dimensional animal pictures through the fusion of marks Skeleton point; train a deep neural network to complete the tracking of multiple 2D skeleton points of animals, and then combine multiple 2D skeleton points in each frame to form a 2D skeleton sequence; use the triangulation algorithm to combine the 2D skeleton sequence with calibration The result is mapped to the three-dimensional space to calculate the three-dimensional skeleton; the coordinates of multiple three-dimensional skeletons are rotated, translated, and inverted without distortion, and aligned to the horizontal line, and the vertical direction of the main camera is used as the Z-axis, and the Z-axis is aligned to the real world height; read the data of multiple 3D skeletons into the workspace, connect multiple 3D skeletons into lines, and display them frame by frame on the screen;

Based on the technical solution of the present invention, through the synchronous shooting of multiple cameras, various body movements and behaviors of animals can be photographed and saved in an all-round and complete manner; The 2D skeleton points of the body can be stably tracked and customized; the deep neural network is trained to complete the tracking of multiple 2D skeleton points of the animal, and then the multiple 2D skeleton points in each frame are combined to form a 2D skeleton sequence. The animal's two-dimensional skeleton sequence has stronger robustness and accuracy when the ambient light changes, the image tone changes, and the background slightly changes; the two-dimensional skeleton sequence is mapped to the three-dimensional space using the triangulation algorithm and the calibration result is calculated. The 3D skeleton makes the data of the 3D skeleton more accurate; through the rotation, translation and inversion of multiple 3D skeleton coordinates without distortion, the features of the 3D skeleton are more accurate and more realistic, and are visualized in the 3D space, and then This enables researchers to use the 3D skeleton data for further analysis.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

1 is a diagram of a three-dimensional capture device for animal behavior recording in an embodiment;

Fig. 2 is the flow chart of the three-dimensional capture method of animal behavior record in the embodiment;

Fig. 3 is the concrete flow chart of multi-camera calibration in Fig. 3;

Description of main component symbols:

101-camera; 102-fixed frame; 103-cage; 201-multi-camera calibration; 202-multi-camera synchronous video shooting; 203-2D skeleton extraction; 204-3D skeleton reconstruction; 205-3D skeleton rotation, translation and inversion ; 206 - 3D skeleton visualization; 301 - start the camera; 302 - camera initialization; 303 - measure the size of the pattern on the calibration plate; 304 - shoot the pattern on the calibration plate.

detailed description

Example

This embodiment provides a three-dimensional capture device, method, system and application for animal behavior recording. As shown in FIG. 1 , the three-dimensional capture device for animal behavior recording includes a camera 101 , a fixed frame 102 , a cage 103 , and the cage 103 set in the fixed frame 102;

The number of cameras 101 is multiple, and the multiple cameras 101 are respectively arranged at multiple positions of the fixed frame 102 to photograph the animals in the whole cage 103 .

In this embodiment, at least one of the cameras 101 is arranged at the center of the top of the fixed frame 102, and the cameras 101 face vertically downward, and the other cameras 101 are arranged around the center, and the cameras 101 are arranged obliquely downward.

In this embodiment, the number of cameras 101 includes five, and the cage 103 includes a stainless steel glass mixing cage;

It should be noted that the number of cameras 101 including 5 is only a preferred number of the present invention, and there are other preferred numbers, such as 2, 3, 4, 6, 7, etc.;

Stainless steel glass blend is only one preferred material for the content of the present invention, and there are other preferred materials, such as aluminum alloy glass blend, wood glass blend, rubber glass blend, and the like.

The fixed frame 102 includes a rectangular frame, and a plurality of cameras 101 are respectively arranged on each top corner of the upper surface of the rectangular frame;

Setting on each top corner of the upper surface of the rectangular frame is only a preferred setting of the present invention, and there are other preferred settings, such as setting on each side of the upper surface of the rectangular frame, setting the rectangular frame perpendicular to the ground and so on on each side.

Preferably, the fixing frame 102 includes an aluminum alloy fixing frame;

As shown in Figure 2, the three-dimensional capture method of the animal behavior record comprises the following steps:

Step 201: multi-camera calibration; including calibrating multiple cameras 101;

Step 202: Multi-camera synchronous video shooting; including controlling multiple cameras 101 to synchronously shoot motion videos of animals at different positions and angles, obtaining the required multiple animal pictures according to the preset frame rate, image quality and shooting time, and making Corresponding feature points between multiple animal pictures are marked, and multiple two-dimensional skeleton points of animals are obtained through the fusion of markings;

Step 203: two-dimensional skeleton extraction; including training a deep neural network, completing the tracking of multiple two-dimensional skeleton points of the animal, and then combining multiple two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence, specifically, using DeepLabCut The toolbox trains deep neural networks to make animal 2D skeleton sequences more robust and accurate in response to ambient light changes, image tone changes, and slight background changes.

Step 204: 3D skeleton reconstruction; including using the triangulation algorithm to map the 2D skeleton sequence in combination with the calibration result into the 3D space to calculate the 3D skeleton, specifically, using the SVD method to perform least squares calculation to make the data of the 3D skeleton more accurate;

Step 205 : rotating, translating, and reversing the three-dimensional skeleton; including rotating, translating, and reversing multiple three-dimensional skeleton coordinates without distortion, and aligning them to the horizontal line, taking the vertical direction of the main camera 101 as the Z axis, and aligning the Z axis to the real The height of the world makes the features of the 3D skeleton more accurate and real;

Step 206 : 3D skeleton visualization; including reading the data of the multiple 3D skeletons into the workspace, connecting the multiple 3D skeletons into lines, and displaying them frame by frame on the screen.

In this embodiment, the method for "controlling multiple cameras to synchronously shoot animal videos at different positions and angles" includes: using 5 cameras, dividing the 5 cameras into four pairs, one of which is the main camera, and the remaining four The camera is a sub-camera, which controls 5 cameras to synchronously shoot the activity videos of animals at different positions and angles;

The method of "using the triangulation algorithm to map the two-dimensional skeleton sequence combined with the calibration results into the three-dimensional space to calculate the three-dimensional skeleton" includes: using the triangulation algorithm to reconstruct the two-dimensional skeleton sequence in pairs, and reconstructing each feature point of the animal separately. Four space point positions, and finally use the least squares method to optimize the space point positions, find the optimal space point positions, and map the two-dimensional skeleton sequence composed of the optimal space point positions to the three-dimensional space for calculation. Get a 3D skeleton.

In this embodiment, as shown in Figure 3, the implementation method of "multi-camera calibration" includes:

Step 301: start the camera; including starting a plurality of cameras 101;

Step 302: camera initialization; including automatic initialization of multiple cameras 101;

Step 303: Measure the size of the pattern on the calibration plate; include the pattern included on the calibration plate for calibration, and measure the size of the pattern;

Step 304 : photographing the pattern on the calibration plate; including multiple cameras 101 synchronously photographing the calibration plate at different positions and angles for several times, obtaining multiple pictures of the calibration plate pattern, and obtaining the rotation matrix and translation vector of the multiple cameras 101, specifically Typically, patterns used for calibration include checkerboard patterns, solid circle array patterns, and the like.

In this embodiment, "obtaining the required multiple animal pictures according to the preset frame rate, image quality and shooting time" further includes:

The frame rate is 40-70FPS;

The picture quality is 440*280-740*580;

Shooting time is 10-20 minutes.

A three-dimensional capture system for animal behavior recording, comprising: a camera 101, a fixing frame 102, a cage 103, and the cage 103 is arranged in the fixing frame 102;

The number of cameras 101 is multiple, and the multiple cameras 101 are respectively arranged at multiple positions of the fixed frame 102 to photograph the animals in the whole cage 103;

Also includes:

a calibration module for calibrating the plurality of cameras 101;

The picture acquisition module is used to control the multiple cameras 101 to synchronously shoot the motion videos of animals at different positions and angles, obtain the required multiple animal pictures according to the preset frame rate, picture quality and shooting time, and analyze the multiple animal pictures. The corresponding feature points between them are marked, and multiple two-dimensional skeleton points of the animal are obtained by the fusion of the marks;

The 2D skeleton extraction module is used to train a deep neural network, complete the tracking of multiple 2D skeleton points of animals, and then combine multiple 2D skeleton points in each frame to form a 2D skeleton sequence. Specifically, the DeepLabCut tool is used. The deep neural network is trained by the box, so that the animal's two-dimensional skeleton sequence has stronger robustness and accuracy when the ambient light changes, the image tone changes, and the background changes slightly.

The three-dimensional skeleton extraction module is used to map the two-dimensional skeleton sequence combined with the calibration result into the three-dimensional space by using the triangulation algorithm to calculate the three-dimensional skeleton. Specifically, the SVD method is used to perform the least square calculation to make the data of the three-dimensional skeleton more accurate;

The processing module is used to rotate, translate, and invert multiple three-dimensional skeleton coordinates without distortion, and align them to the horizontal line, take the vertical direction of the main camera 101 as the Z axis, and align the Z axis to the height of the real world, so that the three-dimensional The features of the skeleton are more accurate and realistic;

The visualization module is used to read the data of multiple 3D skeletons into the workspace, connect the multiple 3D skeletons into lines, and display them frame by frame on the screen.

In this embodiment, the method of the "picture acquisition module" includes: using 5 cameras, dividing the 5 cameras into four pairs, one of the cameras is the main camera, the other four cameras are the sub-cameras, and the 5 cameras are controlled to shoot animals synchronously Activity videos in different locations and angles;

The method of the "3D skeleton extraction module" includes: using the triangulation algorithm to reconstruct the 2D skeleton sequence in pairs, reconstructing four spatial point positions for each feature point of the animal, and finally using the least squares method to optimize the spatial point positions, Find the optimal spatial point position, and map the two-dimensional skeleton sequence composed of the optimal spatial point positions into the three-dimensional space to calculate the three-dimensional skeleton.

In this embodiment, the "calibration module" is also used for:

The multiple cameras 101 are activated, the multiple cameras 101 are automatically initialized, the calibration plate includes a pattern for calibration, and the size of the pattern is measured. From the picture of the pattern, the rotation matrices and translation vectors of the plurality of cameras 101 are obtained. Specifically, the patterns used for calibration include checkerboard patterns, solid circle array patterns, and the like.

In this embodiment, the "picture acquisition module" further includes:

The frame rate is 40-70FPS;

The picture quality is 440*280-740*580;

Shooting time is 10-20 minutes.

In this embodiment, the three-dimensional capture system for animal behavior recording is applied in the behavior recording of non-human primates.

The present invention has at least the following beneficial effects:

A three-dimensional capture device for animal behavior recording provided according to the present invention includes a camera 101, a fixed frame 102, and a cage 103. The cage 103 is arranged in the fixed frame 102. The number of cameras 101 is multiple, and the multiple cameras 101 are respectively Set at multiple positions of the fixed frame 102 to photograph the animals in the full cage 103, at least one of the cameras 101 is set at the central position of the top of the fixed frame 102, and the camera 101 faces vertically downward, and the rest of the cameras 101 are set at Around the center position, and the cameras 101 are set obliquely downward; based on the technical solution of the present invention, researchers can use multiple cameras 101 to quickly track animal behaviors from different angles and positions, and shoot a variety of animals in an all-round and complete synchronization. Body moves and stores.

According to a three-dimensional capture method for animal behavior records using the animal behavior record three-dimensional capture device provided by the present invention, the method includes the following steps: calibrating a plurality of cameras 101; For the active video of the angle, obtain the required multiple animal pictures according to the preset frame rate, image quality and shooting time, and mark the corresponding feature points between the multiple animal pictures, and obtain multiple animal pictures through the fusion of the marks. Two-dimensional skeleton points; train a deep neural network to complete the tracking of multiple two-dimensional skeleton points of animals, and then combine multiple two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence; use the triangulation algorithm to combine the two-dimensional skeleton sequences The calibration result is mapped into the three-dimensional space to obtain the three-dimensional skeleton; the coordinates of the multiple three-dimensional skeletons are rotated, translated, and inverted without distortion, and aligned to the horizontal line, and the Z axis is aligned with the vertical direction of the main camera 101 as the Z axis. To the height of the real world; read the data of multiple 3D skeletons into the workspace, connect multiple 3D skeletons into lines, and display them frame by frame on the screen;

Based on the technical solution of the present invention, through the synchronous shooting of multiple cameras 101, various body movements and behaviors of animals can be photographed and saved in an all-round and complete manner; The 2D skeleton points of the animal body can be stably tracked and customized; the deep neural network is trained to complete the tracking of multiple 2D skeleton points of the animal, and then the multiple 2D skeleton points in each frame are combined to form a 2D skeleton sequence , so that the animal's two-dimensional skeleton sequence has stronger robustness and accuracy when the ambient light changes, the image tone changes, and the background slightly changes; use the triangulation algorithm to map the two-dimensional skeleton sequence combined with the calibration results to the three-dimensional space to calculate The 3D skeleton is obtained, so that the data of the 3D skeleton is more accurate; through the rotation, translation and inversion of multiple 3D skeleton coordinates without distortion, the characteristics of the 3D skeleton are more accurate and more realistic, and are visualized in the 3D space. This in turn enables researchers to use the 3D skeleton data for further analysis.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred implementation scenario, and the modules or processes in the accompanying drawing are not necessarily necessary to implement the present invention.

Those skilled in the art can understand that the modules in the device in the implementation scenario may be distributed in the device in the implementation scenario according to the description of the implementation scenario, or may be located in one or more devices different from the implementation scenario with corresponding changes. The modules of the above implementation scenarios may be combined into one module, or may be further split into multiple sub-modules.

The above serial numbers of the present invention are only for description, and do not represent the pros and cons of the implementation scenarios.

The above disclosures are only a few specific implementation scenarios of the present invention, however, the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

A three-dimensional capture device for animal behavior recording, characterized in that it comprises a camera, a fixing frame, and a cage, and the cage is arranged in the fixing frame;

The number of the cameras is multiple, and the multiple cameras are respectively arranged at multiple positions of the fixed frame to photograph all the animals in the cage.
The three-dimensional capture device for animal behavior recording according to claim 1, characterized in that: at least one of the cameras is arranged at the center of the top of the fixed frame, and the camera faces vertically downward, and the rest of the cameras The cameras are arranged around the central position, and the cameras are arranged obliquely downward.
The three-dimensional capture device for animal behavior recording according to claim 1, wherein the number of the cameras comprises 5, and the cage comprises a stainless steel glass mixing cage;

The fixing frame includes a rectangular frame, and a plurality of the cameras are respectively arranged on each top corner of the upper surface of the rectangular frame;

Preferably, the fixing frame includes an aluminum alloy fixing frame.
A three-dimensional capture method for animal behavior records, characterized in that, using the three-dimensional capture device for animal behavior records according to claim 1 or 2, comprising the following steps:

calibrating a plurality of the cameras;

Control a plurality of the cameras to synchronously shoot the active videos of the animals at different positions and angles, obtain the required multiple animal pictures according to the preset frame rate, image quality and shooting time, and compare the corresponding animal pictures between the multiple animal pictures. The feature points are marked, and a plurality of two-dimensional skeleton points of the animal are obtained by the fusion of the marks;

Train a deep neural network to complete the tracking of multiple two-dimensional skeleton points of the animal, and then combine a plurality of the two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence;

Using a triangulation algorithm to map the two-dimensional skeleton sequence in combination with the calibration result into a three-dimensional space to calculate a three-dimensional skeleton;

Rotate, translate, and invert a plurality of the three-dimensional skeleton coordinates without distortion, and align them to the horizontal line, and use the vertical direction of the main camera as the Z-axis, and align the Z-axis to the height of the real world;

The data of a plurality of the three-dimensional skeletons are read into the workspace, and a line is formed between the plurality of the three-dimensional skeletons, and displayed on the screen frame by frame.
The three-dimensional capture method for animal behavior records according to claim 4, wherein the method of "controlling a plurality of the cameras to synchronously capture the activity videos of animals at different positions and angles" comprises: using five of the cameras , the 5 described cameras are divided into four pairs, one of the described cameras is the main camera, and the other four described cameras are sub-cameras, and the 5 described cameras are controlled to synchronously shoot the activity videos of animals at different positions and different angles;

The method of "using the triangulation algorithm to map the two-dimensional skeleton sequence combined with the calibration result into the three-dimensional space to calculate the three-dimensional skeleton" includes: using the triangulation algorithm to reconstruct the two-dimensional skeleton sequence in pairs, and for each animal's skeleton. Four spatial point positions are reconstructed from each of the feature points, and finally the least squares method is used to optimize the spatial point positions to obtain the optimal spatial point positions, and the two-dimensional skeleton sequence composed of the optimal spatial point positions The three-dimensional skeleton is obtained by combining the calibration results and mapping them into three-dimensional space.
The three-dimensional capture method for animal behavior records according to claim 4, wherein the implementation method of "calibrating a plurality of the cameras" comprises:

Start a plurality of the cameras, the cameras are automatically initialized, the calibration board includes a pattern for calibration, the size of the pattern is measured, and the cameras synchronously shoot several calibration boards at different positions and angles. Next, obtain a plurality of pictures of the calibration plate pattern, and obtain a plurality of rotation matrices and translation vectors of the cameras.
The three-dimensional capture method for animal behavior records according to claim 4, wherein the "obtaining required multiple animal pictures according to the preset frame rate, image quality and shooting time" further comprises:

The frame rate is 40-70FPS;

The picture quality is 440*280-740*580;

The shooting time is 10-20 minutes.
A three-dimensional capture system for animal behavior records, characterized in that:

It includes a camera, a fixed frame, and a cage, and the cage is arranged in the fixed frame;

The number of the cameras is multiple, and the multiple cameras are respectively arranged at multiple positions of the fixed frame to photograph all the animals in the cage;

Also includes:

a calibration module for calibrating a plurality of the cameras;

The picture acquisition module is used to control a plurality of the cameras to synchronously shoot the motion videos of animals at different positions and angles, obtain the required animal pictures according to the preset frame rate, picture quality and shooting time, and analyze the multiple animal pictures. The corresponding feature points between the animal pictures are marked, and a plurality of two-dimensional skeleton points of the animal are obtained through the fusion of the markings;

A two-dimensional skeleton extraction module, used to train a deep neural network, complete the tracking of multiple two-dimensional skeleton points of the animal, and then combine a plurality of the two-dimensional skeleton points in each frame to form a two-dimensional skeleton sequence;

A three-dimensional skeleton extraction module, which is used for using a triangulation algorithm to map the two-dimensional skeleton sequence in combination with the calibration result into a three-dimensional space to calculate a three-dimensional skeleton;

The processing module is used to rotate, translate, and invert a plurality of the three-dimensional skeleton coordinates without distortion, and align them to the horizontal line, and use the vertical direction of the main camera as the Z-axis to align the Z-axis to the height of the real world ;

The visualization module is used for reading the data of a plurality of the three-dimensional skeletons into the workspace, connecting the plurality of the three-dimensional skeletons into lines, and displaying them frame by frame on the screen.
The three-dimensional capture system for animal behavior recording according to claim 8, wherein the method of the "picture acquisition module" comprises: using five cameras, dividing the five cameras into four pairs, one of which is The camera is the main camera, and the remaining four cameras are sub-cameras, and 5 of the cameras are controlled to synchronously shoot the active videos of animals at different positions and angles;

The method of the "three-dimensional skeleton extraction module" includes: using a triangulation algorithm to reconstruct the two-dimensional skeleton sequence in pairs, reconstructing four spatial point positions for each of the feature points of the animal, and finally using the least squares method. Optimizing the spatial point positions is performed to obtain the optimal spatial point positions, and the two-dimensional skeleton sequence composed of the optimal spatial point positions is mapped to the three-dimensional space in combination with the calibration results to calculate the three-dimensional skeleton.
The three-dimensional capture system for animal behavior recording according to claim 8, wherein the "calibration module" is further used for:

Start a plurality of the cameras, the cameras are automatically initialized, the calibration board includes a pattern for calibration, the size of the pattern is measured, and the cameras synchronously shoot several calibration boards at different positions and angles. Next, obtain a plurality of pictures of the calibration plate pattern, and obtain a plurality of rotation matrices and translation vectors of the cameras.
The three-dimensional capture system for animal behavior records according to claim 8, wherein the "picture acquisition module" further comprises:

The frame rate is 40-70FPS;

The picture quality is 440*280-740*580;

The shooting time is 10-20 minutes.
Application of the three-dimensional capture system for animal behavior recording according to any one of claims 8 to 11 in non-human primate behavior recording.