WO2023108711A1

WO2023108711A1 - Method and apparatus for synchronously analyzing behavior information and pupil information, and device and medium

Info

Publication number: WO2023108711A1
Application number: PCT/CN2021/140013
Authority: WO
Inventors: 郭丰; 张佳佳; 蔚鹏飞; 王立平
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2021-12-14
Filing date: 2021-12-21
Publication date: 2023-06-22
Also published as: CN114220168A

Abstract

Disclosed herein are a method and apparatus for synchronously analyzing behavior information and pupil information, and a device and a medium. The method for synchronously analyzing behavior information and pupil information comprises: acquiring behavior video data and pupil video data of a target object within the same time period, wherein the behavior video data comprises video data acquired from at least four photographing angles; respectively analyzing the behavior video data and the pupil video data, so as to determine behavior information and pupil information of the target object; and establishing an association relationship between the behavior information and the pupil information according to the collection time of the behavior video data and the pupil video data.

Description

Behavior and pupil information synchronous analysis method, device, equipment and medium

This application claims priority to a Chinese patent application with application number 202111530272.9 filed with the China Patent Office on December 14, 2021, the entire contents of which are incorporated herein by reference.

technical field

The present application relates to the technical field of machine vision, for example, to a method, device, device and medium for synchronous analysis of behavior and pupil information.

Background technique

The information of the pupil is usually associated with the nervous system stimulation of the brain, indicating that the human or animal body produces some specific behaviors or the central specific circuit is activated. For example, when we are afraid, our central system receives the signal of fear. Information and generate feedback information, which will make us tremble with fear, and then have the behavior of wanting to flee, and at the same time, the pupils will dilate to about 3 times their usual size.

The method of recording brain nerve activity, including electrophysiological, optogenetic and other technical means, can determine the relationship between exercise and brain nerve activity, and can be used as a bridge to explore pupil information and brain nerve activity through the relationship between pupil information and behavior.

However, there is no mature scheme for synchronously analyzing behavioral information and pupil information.

Contents of the invention

This application provides a method, device, equipment, and medium for synchronous analysis of behavior and pupil information, so as to achieve synchronous acquisition and analysis of the behavior information and pupil information of the target object, and to match the behavior information and pupil information. Research such as Loop provides research data.

The embodiment of the present application provides a method for synchronously analyzing behavior and pupil information, the method comprising:

Acquiring behavior video data and pupil video data of the target object within the same time period, wherein the behavior video data includes video data obtained from at least four shooting angles;

Analyzing the behavior video data and the pupil video data respectively to determine the behavior information and pupil information of the target object;

According to the collection time of the behavior video data and the pupil video data, an association relationship between the behavior information and the pupil information is established.

The present application also provides a device for synchronous analysis of behavior and pupil information, which device includes:

The data acquisition module is configured to obtain behavioral video data and pupil video data of the target object within the same time period, wherein the behavioral video data includes video data acquired by at least four shooting angles;

The data analysis module is configured to analyze the behavior video data and the pupil video data respectively, and determine the behavior information and pupil information of the target object;

The data matching module is configured to establish an association relationship between the behavior information and the pupil information according to the collection time of the behavior video data and the pupil video data.

The present application also provides a kind of computer equipment, and described computer equipment comprises:

one or more processors;

memory configured to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors are made to implement the above method for synchronously analyzing behavior and pupil information.

The present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above method for synchronous analysis of behavior and pupil information is realized.

Description of drawings

Fig. 1 is a flow chart of a method for synchronous analysis of behavior and pupil information provided in Embodiment 1 of the present application;

FIG. 2 is a schematic diagram of key points of a skeleton of a target object provided in Embodiment 1 of the present application;

Fig. 3 is a schematic diagram of pupil key points of a target object provided in Embodiment 1 of the present application;

FIG. 4 is a schematic diagram of key points around the eyeball of a target object provided in Embodiment 1 of the present application;

Fig. 5 is a schematic structural diagram of a behavior and pupil information synchronous analysis device provided in Embodiment 2 of the present application;

FIG. 6 is a schematic structural diagram of a computer device provided in Embodiment 3 of the present application.

Detailed ways

The application will be described below in conjunction with the accompanying drawings and embodiments. The specific embodiments described herein are for illustration of the application only. For ease of description, only parts relevant to the present application are shown in the drawings.

Embodiment one

Figure 1 is a flow chart of a method for synchronous analysis of behavior and pupil information provided in Embodiment 1 of the present application. This embodiment can be applied to the study of behavior and brain nerve activity, for example, it is suitable for synchronous analysis of behavior with mice as research objects The scene of information and pupil information. The method can be executed by a behavior and pupil information synchronous analysis device, which can be realized by software and/or hardware, and integrated in a computer device with application development function.

As shown in Figure 1, the behavior and pupil information synchronous analysis method includes the following steps:

S110. Acquire behavior video data and pupil video data of the target object within the same time period, wherein the behavior video data includes video data obtained from at least four shooting angles.

The target object refers to the object of the experimental research. Usually, the experimental objects in the experimental research of brain science are experimental animals such as mice, monkeys, and dogs. When collecting the behavior video data of the target object, 4 or more cameras are used to collect video data of different azimuth angles to observe the behavior of the target object from different angles. During the shooting process, the behavior of the target object is free Unrestricted. The reason why the behavior video data of the target object is collected from multiple angles is that when the behavior of the target object is recognized, the target object needs to be reconstructed in three dimensions, and the target object can be reconstructed in three dimensions based on the two-dimensional data from multiple angles. Multiple cameras that simultaneously collect behavioral video data are calibrated with internal and external parameters. The correction process may be to use multiple cameras to take pictures of the preset checkerboard, and perform correction based on the pictured results. For example, use a checkerboard (12*9) as the calibration board, use multiple cameras with fixed shooting positions to shoot the calibration board at different angles, each camera takes 60 pictures, and then use the StereoCameraCalibrator GUI toolbox in MATLAB to perform calibration Operation to obtain the internal and external parameters of multiple cameras, save them, and use them in the subsequent process of recognizing the behavior of the target object.

The pupil video data is the pupil video data of the target object recorded synchronously by the infrared camera. A camera fixing device can be worn on the head of the target object, and the infrared camera can be fixed through the camera fixing device, so that the shooting field of view of the infrared camera at its fixed position can be aligned with the eyes of the target object, and the change of the pupil can be captured information. One or two infrared cameras can be installed. In some embodiments, if the target object itself is small in size and the head has limited load-bearing capacity, in order to prevent the infrared camera worn on the head from being too heavy and affecting the activities of the target object, only one infrared camera can be set to collect the pupil changes of one eye information. For larger target objects, two infrared cameras can also be used to simultaneously record the pupil change information of the two eyes.

S120. Analyze the behavior video data and the pupil video data respectively, and determine behavior information and pupil information of the target object.

When analyzing the behavior video data, firstly, the behavior video data is analyzed using a preset pose estimation model to determine the preset skeleton key points of the target object in each frame of video image in the behavior video data. The preset pose estimation model in this step can be a pre-trained DeepLabCut deep learning model, which can be matched to a suitable DeepLabCut deep learning model for different target objects, and the video images in the behavioral video data can be input into the pre-trained In the DeepLabCut deep learning model, the skeleton key points corresponding to the target object can be extracted. For example, when the target object is a mouse, the skeleton key points can be limbs, nose, ears, head, torso, tail, etc. Multiple key points that can be used to display the posture of the target object, such as the multiple key points of the mouse (small square points with different gray values) shown in Figure 2 . Based on 2D pose estimates extracted from video images from different angles at the same time point, that is, multiple skeletal keypoints, the target object can be reconstructed in 3D by a triangulation algorithm. Then, the 3D reconstruction result of the target object may be input into a preset behavior analysis model to classify the behavior of the target object and determine the behavior information of the target object. Among them, the preset behavior classification model can be a pre-trained Behavior Atlas behavior analysis model, which can automatically analyze the three-dimensional behavior of animals without supervision, and determine the behavior category of the target object. In the behavior analysis process of the Behavior Atlas behavior analysis model, the time series of the 3D reconstruction results of the input target object will be segmented, and the segmented sequence will be subjected to Uniform Manifold Approximation and Projection (Uniform Manifold Approximation and Projection) , UMAP) dimensionality reduction, followed by hierarchical clustering to divide behaviors into multiple behavior categories.

When analyzing the pupil video data, first, based on the preset marker point extraction model, extract the preset pupil peripheral marker points in each frame of video image in the pupil video data (8 pupil peripheral marker points as shown in Figure 3 -dots with different grayscales), pupil center point and preset eyeball peripheral marking points (the eyeball peripheral marking points-dots with different grayscales as shown in Figure 4). Among them, the preset marker point extraction model can also be a pre-trained DeepLabCut deep learning model, and the DeepLabCut deep learning model can be trained by using the pupil images of the target object that have pre-marked the key points around the pupil and the key points around the eyeball, and then the The trained model is used to identify the key points of the pupil and eyeball in the pupil video data. The pupil diameter of the target object is determined according to the preset pupil peripheral marker points and the pupil center point. The eyeball center position of the target object is determined based on the preset eyeball peripheral marker points, and then the relative position between the pupil and the eyeball of the target object is determined based on the eyeball center position and the pupil center point. The pupil diameter of the target object and the relative position between the pupil of the target object and the eyeball are used as the pupil information of the target object. In this step, the center position of the eyeball and the lengths of the two symmetry axes corresponding to the fitted ellipsoid can be determined by means of ellipse fitting. The center position of the eyeball remains fixed, and the center position of the pupil changes according to the behavior of the target object. Changes in the relative positional relationship between the center position of the pupil and the center position of the eyeball can be analyzed to determine changes in the pupil during a series of behavioral reactions of the target object.

Before determining the relative position of the pupil of the target object and the eyeball, it is also necessary to filter the position of the center point of the pupil corresponding to each frame of the video image in the pupil video data. This is because the target object does not keep their eyes open all the time, and sometimes they close their eyes. Then, in some pupil video images, the exact position of the pupil center point cannot be extracted when the eyes of the target object are closed, and the position information will be an abnormal value. Then, the median filtering method can be used to filter out abnormal values.

In addition, when collecting the pupil video data of the target object, the data may be collected in batches, and there may be situations where the fixed relative positions of the infrared camera and the target object are different. Then, for the same target object, the ellipse fitting The resulting eyeballs are not the same and need to be corrected. For the pupil video data collected in different data collection batches of the target object, the preset eyeball peripheral marker points can be input into the preset ellipse fitting algorithm to determine the eyeball center position of the target object; the eyeball center determined by the ellipse fitting algorithm The corresponding ellipsoidal axis of symmetry is corrected. For example, select a group of data with a larger value of the symmetry axis in the ellipse fitting result as the standard group, modify one or more groups of data with a smaller value of the symmetry axis in the ellipse fitting result, and multiply it by a sparse value to make different batches The collected data remains consistent.

S130. Establish an association relationship between the behavior information and the pupil information according to the behavior video data and the collection time of the pupil video data.

When matching behavior information and pupil information, first, check whether the total number of frames, video duration, and sampling rate of pupil video data and behavior video data are consistent, that is, the time sequence of pupil video data and behavior video data in video image frames whether it can be aligned. A linear interpolation method may be used to supplement frames (supplement video image frames) for video data in which frames are lost. For example, in seconds, linearly interpolate the position where the time interval between adjacent video image frames is the largest within 1 second. Alternatively, for non-drop frame video data, a corresponding number of video images may be deleted at the position where the time interval of image frames is the smallest. In addition, for the case of different sampling rates, the interp1 function can be used to interpolate frames evenly.

After complementing the frame of the video, the behavior information is matched with the pupil information to establish the relationship between the two. According to the video data after the frame complement operation, the association relationship between the behavior information of the preset time window and the pupil information can be established. Wherein, the preset time window is a specified period of time, which may be a period of time during which a behavior is exhibited and lasts, such as being determined according to the time point when external stimulation is performed on the target object. In the process of collecting the behavior video data and pupil video data of the target object, the target object may be stimulated by light, or external stimuli such as intimidation stimulation, so as to observe the stress response behavior of the target object and the state of the pupil when the stress response occurs. Variety. When external stimulation is given to the target subject, the time of start and end of stimulation will be recorded. Then, the preset time window may be a time period from the start of the external stimulus to the target object to the end of the external stimulus. Matching the behavior information and pupil information in the same time period completes the data analysis process.

In the technical solution of this embodiment, the behavior video data of the target object at multiple angles and the pupil video data of the target object in the same time period are acquired on the premise that the target object moves freely, and the behavior video data and the pupil video data are separately Analyze and determine the behavior information and pupil information of the target object; finally, the association between behavior information and pupil information can be established according to the collection time of behavior video data and pupil video data. The technical solution of the embodiment of the present application solves the problem of synchronously analyzing the behavior information and pupil information technology of experimental mice in the related art, and can realize synchronous collection of the behavior video and pupil video of the target object, and analyze the corresponding behavior information and pupil information , to match behavioral information and pupil information, and provide research data for brain science and neural mechanism circuits.

Embodiment two

Fig. 5 is a schematic structural diagram of the behavior and pupil information synchronous analysis device provided by the second embodiment of the present application. This embodiment and the behavior and pupil information synchronization method in the above embodiment belong to the same idea, and are suitable for studying behavior and brain nerve activity For example, it is suitable for synchronous analysis of behavioral information and pupil information with mice as research objects. The device can be realized by software and/or hardware, and integrated into a server device with application development functions.

As shown in FIG. 5 , the behavior and pupil information synchronous analysis device includes: a data collection module 210 , a data analysis module 220 and a data matching module 230 .

The data acquisition module 210 is configured to acquire behavior video data and pupil video data of the target object within the same time period, wherein the behavior video data includes video data obtained from at least four shooting angles; the data analysis module 220 is configured to The behavior video data and the pupil video data are analyzed separately to determine the behavior information and pupil information of the target object; the data matching module 230 is configured to, according to the acquisition time of the behavior video data and the pupil video data, An association relationship between the behavior information and the pupil information is established.

The technical solution of this embodiment determines the behavior information of the target object by acquiring the behavior video data of the target object at multiple angles and the pupil video data of the target object in the same time period, and analyzing the behavior video data and the pupil video data respectively and pupil information; finally, an association relationship between behavior information and pupil information can be established according to the collection time of the behavior video data and the pupil video data. The technical solution of the embodiment of the present application solves the problem of synchronously analyzing the behavior information and pupil information technology of experimental mice in the related art, and can realize synchronous collection of the behavior video and pupil video of the target object, and analyze the corresponding behavior information and pupil information , to match behavioral information and pupil information, and provide research data for brain science and neural mechanism circuits.

In an optional implementation manner, the data analysis module 220 includes a behavior information analysis submodule, which is configured to:

Using a preset attitude estimation model to analyze the behavior video data, determine the preset skeleton key points of the target object in each frame of video image in the behavior video data; according to the same time point in the behavior video data performing 3D reconstruction on the target object based on preset skeleton key points in video images from different angles; inputting the 3D reconstruction result of the target object into a preset behavior analysis model to classify the behavior of the target object, Determine the behavior information of the target object.

In an optional implementation manner, the data analysis module 220 includes a pupil information analysis submodule, which is configured to:

Based on the preset marker point extraction model, extract the preset pupil peripheral marker points, pupil center points and preset eyeball peripheral marker points in each frame of video image in the pupil video data; according to the preset pupil peripheral marker points and The pupil center point determines the pupil diameter of the target object; determines the eyeball center position of the target object based on the preset eyeball peripheral marker points, and determines the eyeball center position based on the eyeball center position and the pupil center point position The relative position of the pupil of the target object and the eyeball; the pupil diameter of the target object and the relative position of the pupil of the target object and the eyeball are used as the pupil information of the target object.

In an optional implementation manner, the pupil information analysis submodule is also set to:

Before determining the relative position of the pupil of the target object and the eyeball, filtering is performed on the position of the center point of the pupil corresponding to each frame of video image in the pupil video data.

For the pupil video data collected in different data collection batches of the target object, input the preset eyeball peripheral marker points into a preset ellipse fitting algorithm to determine the eyeball center position of the target object;

The ellipsoid symmetry axis corresponding to the eyeball center position determined by the ellipse fitting algorithm is corrected.

In an optional implementation manner, the data matching module 230 is set to:

Comparing the number of image frames, the duration and the video sampling rate of the behavior video data and the pupil video data are the same, and determining the comparison result; according to the comparison result, the video data with a relatively small number of image frames is subjected to a supplementary frame operation; according to the supplementary frame The manipulated video data establishes a relationship between behavior information and pupil information in a preset time window, wherein the preset time window is determined according to the time point when external stimulation is performed on the target object.

In an optional implementation manner, the data matching module 230 is also configured to:

For video data with small number of image frames, duration or video sampling rate, the linear interpolation method is used to supplement video image frames.

The behavior and pupil information synchronous analysis device provided in the embodiment of the present application can execute the behavior and pupil information synchronous analysis method provided in any embodiment of the present application, and has corresponding functional modules and effects for executing the method.

Embodiment Three

FIG. 6 is a schematic structural diagram of a computer device provided in Embodiment 3 of the present application. FIG. 6 shows a block diagram of an exemplary computer device 12 suitable for implementing embodiments of the present application. The computer device 12 shown in FIG. 6 is only an example, and should not limit the functions and scope of use of the embodiment of the present application. The computer device 12 can be any terminal device with computing capability, such as an intelligent controller, a server, a mobile phone and other terminal devices.

As shown in FIG. 6, computer device 12 takes the form of a general-purpose computing device. Components of computer device 12 may include, but are not limited to: one or more processors or processing units 16 , system memory 28 , bus 18 connecting various system components including system memory 28 and processing unit 16 .

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (Extended Industry Standard Architecture, ISA) bus, Micro Channel Architecture (Micro Channel Architecture, MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

Computer device 12 includes a variety of computer system readable media. These media can be any available media that can be accessed by computer device 12 and include both volatile and nonvolatile media, removable and non-removable media.

System memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32 . Computer device 12 may include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (not shown in FIG. 6, commonly referred to as a "hard drive"). Although not shown in FIG. 6, a disk drive configured to read and write to a removable non-volatile disk (such as a "floppy disk") may be provided, as well as a removable non-volatile disk (such as a Compact Disc Read-Only Memory, CD-ROM), Digital Video Disc Read-Only Memory (DVD-ROM) or other optical media) CD-ROM drive. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The system memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of the embodiments of the present application.

Program/utility 40 may be stored, for example, in system memory 28 as a set (at least one) of program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or a combination of these examples may include implementations of the network environment. The program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the computer device 12, and/or with Any device (eg, network card, modem, etc.) that enables the computing device 12 to communicate with one or more other computing devices. This communication can be performed through an input/output (Input/Output, I/O) interface 22 . Moreover, the computer device 12 can also communicate with one or more networks (such as a local area network (Local Area Network, LAN), a wide area network (Wide Area Network, WAN) and/or a public network, such as the Internet) through the network adapter 20. As shown, network adapter 20 communicates with other modules of computer device 12 via bus 18 . It should be appreciated that although not shown in FIG. 6 , other hardware and/or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

The processing unit 16 executes a variety of functional applications and data processing by running the program stored in the system memory 28, such as implementing the behavior and pupil information synchronization analysis method provided by the embodiment of the present invention, including:

Obtaining behavior video data and pupil video data of the target object within the same time period, wherein the behavior video data includes video data obtained from at least four shooting angles; analyzing the behavior video data and the pupil video data respectively , determining the behavior information and pupil information of the target object; establishing an association relationship between the behavior information and the pupil information according to the behavior video data and the collection time of the pupil video data.

Embodiment four

Embodiment 4 of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for synchronously analyzing behavior and pupil information as provided in any embodiment of the present application is implemented, including:

The computer storage medium in the embodiments of the present application may use any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer-readable storage medium may be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more conductors, portable computer disks, hard disks, RAM, ROM, Erasable Programmable Read-Only Memory, EPROM or flash memory), optical fiber, CD-ROM, optical storage device, magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a data signal carrying computer readable program code in baseband or as part of a carrier wave. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. .

The program code contained on the computer readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, electric wire, optical cable, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.

Computer program codes for performing the operations of the present application may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional A procedural programming language, such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. Where a remote computer is involved, the remote computer can be connected to the user computer through any kind of network, including a LAN or WAN, or it can be connected to an external computer (eg, via the Internet using an Internet service provider).

Those of ordinary skill in the art should understand that the above-mentioned multiple modules or multiple steps of the present application can be realized by general-purpose computing devices, and they can be concentrated on a single computing device, or distributed on a network formed by multiple computing devices , alternatively, they can be implemented with computer device executable program codes, so that they can be stored in a storage device and executed by a computing device, or they can be made into a plurality of integrated circuit modules respectively, or one of them Multiple modules or steps are implemented as a single integrated circuit module. As such, the application is not limited to any specific combination of hardware and software.

Claims

A method for synchronous analysis of behavior and pupil information, comprising:

Acquiring behavior video data and pupil video data of the target object within the same time period, wherein the behavior video data includes video data obtained from at least four shooting angles;

Analyzing the behavior video data and the pupil video data respectively to determine the behavior information and pupil information of the target object;

According to the collection time of the behavior video data and the pupil video data, an association relationship between the behavior information and the pupil information is established.
The method according to claim 1, wherein analyzing the behavior video data to determine the behavior information of the target object comprises:

Analyzing the behavioral video data using a preset pose estimation model to determine the preset skeleton key points of the target object in each frame of video image in the behavioral video data;

Carrying out three-dimensional reconstruction of the target object according to preset skeleton key points in video images of different angles at the same time point in the behavioral video data;

Input the 3D reconstruction result of the target object into a preset behavior analysis model, classify the behavior of the target object, and determine the behavior information of the target object.
The method according to claim 1, wherein analyzing the pupil video data to determine pupil information of the target object comprises:

Based on the preset marker point extraction model, extracting preset pupil peripheral marker points, pupil center points and preset eyeball peripheral marker points in each frame of video image in the pupil video data;

determining the pupil diameter of the target object according to the preset pupil peripheral marker points and the pupil center point;

Determine the eyeball center position of the target object based on the preset eyeball peripheral marker points, and determine the relative position between the pupil and the eyeball of the target object based on the eyeball center position and the pupil center point;

The pupil diameter of the target object and the relative position between the pupil of the target object and the eyeball are used as the pupil information of the target object.
The method according to claim 3, before said determining the relative position of the pupil and the eyeball of the target object, further comprising:

Filtering is performed on the position of the center point of the pupil in each frame of video image in the pupil video data.
The method according to claim 3, wherein, for the pupil video data collected by the target object in different data collection batches, the determination of the eyeball center position of the target object based on the preset eyeball peripheral marker points includes :

Input the preset eyeball peripheral marker points into a preset ellipse fitting algorithm to determine the eyeball center position of the target object;

The ellipsoid symmetry axis corresponding to the eyeball center position determined by the ellipse fitting algorithm is corrected.
The method according to claim 1, wherein said establishing an association between the behavior information and the pupil information according to the collection time of the behavior video data and the pupil video data includes:

Comparing whether the number of image frames, duration and video sampling rate of the behavior video data and the pupil video data are the same, and determine the comparison result;

According to the comparison result, the frame complement operation is performed on the video data with a relatively small number of image frames;

According to the video data after the supplementary frame operation, the association relationship between the behavior information of the preset time window and the pupil information is established, wherein the preset time window is determined according to the time point when external stimulation is performed on the target object of.
The method according to claim 6, wherein, performing frame complementing operation on video data with a relatively small number of image frames according to the comparison result comprises:

For video data with a small number of image frames, duration or video sampling rate, the linear interpolation method is used to supplement video image frames.
A device for synchronous analysis of behavior and pupil information, comprising:

The data acquisition module is configured to acquire behavioral video data and pupil video data of the target object within the same time period, wherein the behavioral video data includes video data obtained from at least four shooting angles;

The data analysis module is configured to analyze the behavior video data and the pupil video data respectively, and determine the behavior information and pupil information of the target object;

The data matching module is configured to establish an association relationship between the behavior information and the pupil information according to the collection time of the behavior video data and the pupil video data.
A computer device comprising:

at least one processor;

a memory configured to store at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the behavior and pupil information synchronization analysis method according to any one of claims 1-7.
A computer-readable storage medium storing a computer program, wherein when the program is executed by a processor, the method for synchronous analysis of behavior and pupil information according to any one of claims 1-7 is realized.