WO2023097773A1 - 步态分析方法、装置、设备及存储介质 - Google Patents

步态分析方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2023097773A1
WO2023097773A1 PCT/CN2021/137903 CN2021137903W WO2023097773A1 WO 2023097773 A1 WO2023097773 A1 WO 2023097773A1 CN 2021137903 W CN2021137903 W CN 2021137903W WO 2023097773 A1 WO2023097773 A1 WO 2023097773A1
Authority
WO
WIPO (PCT)
Prior art keywords
gait
analyzed
skeleton
limb
moment
Prior art date
Application number
PCT/CN2021/137903
Other languages
English (en)
French (fr)
Inventor
陈可
蔚鹏飞
王立平
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Publication of WO2023097773A1 publication Critical patent/WO2023097773A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/34—Smoothing or thinning of the pattern; Morphological operations; Skeletonisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition

Definitions

  • the present application relates to the technical field of image processing, for example, to a gait analysis method, device, equipment and storage medium.
  • Gait analysis provides automated, fast, objective and complete analysis of gait-related changes following a medical condition or acute pain.
  • the premise of gait analysis is that animals need to perform continuous movement on a specific track. For animals, such forced movement will cause them to deviate from the original law of motion, and the gait parameters obtained from this are divorced from the real situation, and their application value is limited.
  • the present application provides a gait analysis method, device, equipment and storage medium to realize the effect of gait analysis on an object to be analyzed in a freely moving state.
  • the present application provides a method of gait analysis, which may include:
  • Gait parameters of the subject to be analyzed are determined from multiple target skeletons.
  • the present application also provides a gait analysis device, which may include:
  • the limb image acquisition module is configured to acquire the limb images of the object to be analyzed located on the static plane at multiple moments;
  • the three-dimensional skeleton determination module is configured to determine the three-dimensional skeleton of the object to be analyzed at each moment according to the limb image at each moment for the limb image at each moment;
  • the target skeleton obtaining module is configured to determine multiple gait occurrence moments of the gait of the object to be analyzed from multiple moments according to the three-dimensional skeleton at multiple moments, and use the three-dimensional skeleton at each gait occurrence moment as the target skeleton ;
  • the gait parameter determination module is configured to determine the gait parameters of the object to be analyzed according to multiple target skeletons.
  • the application also provides a gait analysis device, which may include:
  • processors one or more processors
  • memory configured to store one or more programs
  • the one or more processors realize the above-mentioned gait analysis method.
  • the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned gait analysis method is realized.
  • Fig. 1 is the flowchart of a kind of gait analysis method that the application embodiment 1 provides;
  • FIG. 2 is a schematic diagram of a two-dimensional posture in a gait analysis method provided in Embodiment 1 of the present application;
  • Fig. 3 is a flow chart of a gait analysis method provided in Embodiment 2 of the present application.
  • Fig. 4 is a flowchart of an optional example in a gait analysis method provided in Embodiment 2 of the present application;
  • FIG. 5 is a flow chart of a gait analysis method provided in Embodiment 3 of the present application.
  • FIG. 6 is a schematic diagram of an image capture device in a gait analysis method provided in Embodiment 3 of the present application.
  • FIG. 7 is a structural block diagram of a gait analysis device provided in Embodiment 4 of the present application.
  • Fig. 8 is a schematic structural diagram of a gait analysis device provided in Embodiment 5 of the present application.
  • FIG. 1 is a flowchart of a gait analysis method provided in Embodiment 1 of the present application. This embodiment is applicable to the situation of performing gait analysis on an object to be analyzed in a freely moving state.
  • the method can be executed by the gait analysis device provided in the embodiment of the present application, the device can be realized by software and/or hardware, and the device can be integrated in a gait analysis device, which can be a user terminal or a server.
  • the method of the embodiment of the present application includes the following steps:
  • the object to be analyzed can be the object whose gait parameters are to be analyzed, which can be animals and/or people, and is not limited here;
  • the static plane can be a stationary plane located on the ground or suspended in the air, so it is located in the stationary plane
  • the object to be analyzed on can move freely on the stationary plane.
  • the limb image can be an image of a gait limb related to the gait of the object to be analyzed that is captured at a moment, and the gait limb can be the limbs of an animal, two legs of a person, etc., and is not limited here;
  • the number of body images at a moment may be one, two or more, and it is not limited here.
  • the gait of the object to be analyzed occurs in three-dimensional space, in order to accurately determine its gait parameters, for the limb image at each moment, it can be determined according to the limb image at each moment The following three-dimensional skeleton. Since the three-dimensional skeleton of the object to be analyzed at a moment is unique, when the number of limb images at a moment includes at least two, the at least two limb images can be used to jointly determine its body at a moment. 3D skeleton.
  • the 3D skeleton obtained in this step can be filtered to remove the noise points in it, so as to obtain a stable gait, and the filtering method can be median Filtering, mean filtering, etc. are not limited here.
  • Gait is the state when you actually walk, and it is a kind of movement, such as sniffing, turning, climbing, landing, one step forward and two steps back, etc. can only be called movement, not gait.
  • the object to be analyzed does not necessarily have a gait at multiple moments. Therefore, in order to accurately determine the gait parameters of the object to be analyzed, it is possible to first determine when the gait occurs.
  • Gait parameters In practical applications, whether a moment is a gait occurrence moment can be determined according to the three-dimensional skeleton at the moment, the associated moment associated with the moment, and/or the three-dimensional skeleton at multiple moments, which is not limited here.
  • the target skeleton is the three-dimensional skeleton at the time when the gait occurs, for example, it is the three-dimensional skeleton of the object to be analyzed in a freely moving state when the gait occurs, so the gait parameters determined based on multiple target skeletons are consistent with the object to be analyzed.
  • the gait parameters of the original movement rules can be used as the research basis for drug screening, behavior analysis, animal disease modeling, neural circuit mechanism, etc.
  • the technical solution of the embodiment of the present application by acquiring body images of the subject to be analyzed at multiple moments in a state of free movement on a stationary plane; since the gait of the subject to be analyzed occurs in three-dimensional space, for each According to the limb image at each moment, the three-dimensional skeleton of the object to be analyzed at each moment is determined according to the limb image at each moment, thus ensuring the accuracy of the subsequent determined gait parameters of the object to be analyzed; because the object to be analyzed The gait does not always occur at multiple moments, so the multiple gait occurrence moments of the object to be analyzed can be determined from multiple moments according to the three-dimensional skeleton at multiple moments; since the target skeleton is the gait occurrence moment The lower 3D skeleton, so the gait parameters can be determined from multiple target skeletons.
  • the above technical solution can accurately obtain the gait parameters of the subject to be analyzed in a freely moving state.
  • An optional technical solution is to determine multiple gait occurrence moments of the gait of the object to be analyzed from multiple moments according to the three-dimensional skeleton at multiple moments, which may include: obtaining the skeleton time corresponding to the three-dimensional skeleton at multiple moments Sequence, and take the object behavior of the object to be analyzed as the segmentation target to segment the skeleton time series; combined with the target moving speed of the object to be analyzed under the multiple segmented skeleton time series, the multiple segmented time series The skeleton time series is clustered on the object behavior; according to the clustering results, multiple gait occurrence moments of the gait occurrence of the object to be analyzed are determined at multiple moments.
  • the skeleton time series can be a time series corresponding to three-dimensional skeletons at multiple times, and the skeleton time series is segmented based on the object behavior of the object to be analyzed, so that the multiple skeletons of the object to be analyzed
  • the object behaviors under the time series are not exactly the same.
  • the object behaviors can be gait, sniffing, turning, climbing, landing, one step forward, two steps back, static, etc., which are not limited here; the above skeleton time
  • the segmentation process of the sequence can be realized based on the machine learning model, or can be realized based on other methods, which is not limited here.
  • the target moving speed of the object to be analyzed under each skeleton time series after segmentation (such as the average, maximum or minimum moving speed determined according to the three-dimensional skeleton, etc. ), cluster the multiple segmented skeleton time series on the object behavior, and obtain the clustering result corresponding to each object behavior (that is, the clustered skeleton time series), and from each clustered Find the gait occurrence time series corresponding to the gait occurrence in the skeleton time series, and use multiple moments under the gait occurrence time series as the gait occurrence time.
  • Uniform Manifold Approximation and Projection (UMAP) dimensionality reduction can be performed on the segmented skeleton time series first, and then multiple dimensionality-reduced skeleton time series can be performed Hierarchical clustering, thus ensuring the accuracy of clustering.
  • clustering can be performed on the basis of each segmented skeleton time series, or can be performed on the basis of each segmented skeleton time series
  • Each three-dimensional skeleton below is clustered as a unit, which is not limited here. The above technical solution achieves the effect of accurately determining the moment when the gait occurs by combining the three-dimensional skeleton and the moving speed of the target.
  • the three-dimensional skeleton is a three-dimensional skeleton that includes multiple limb bone points on the gait limb of the object to be analyzed; according to the three-dimensional skeleton at multiple moments, the occurrence step of the object to be analyzed is determined from multiple moments.
  • the multiple gait occurrence moments of the state may include: for each three-dimensional skeleton, rotating the three-dimensional skeleton so that the skeleton plane where the rotated three-dimensional skeleton is located is parallel to the static plane; judging the gait according to the rotated three-dimensional skeleton Whether the limb touches the static plane; according to the multiple judgment results corresponding to the multiple times, determine the multiple gait occurrence times of the object to be analyzed from the multiple times.
  • the limb skeleton points may include skeleton points on the limbs of gait, and the three-dimensional skeleton may be a skeleton in three-dimensional space formed by these limb skeleton points. Since the reconstruction process of the 3D skeleton is realized based on the device parameters of the image capture device that is set to capture body images, the device parameters are obtained by calibrating the image capture device. Affected by the calibration accuracy, the skeleton plane where the 3D skeleton is located is different. must be parallel to the rest plane.
  • the 3D skeleton can be rotated (for example, by the principle of cross product of normal vectors, etc.), so that the rotated 3D skeleton is in the
  • the skeleton plane is parallel to the rest plane, and the rest plane is mostly a horizontal plane in three-dimensional space; then the judgment result of whether the gait limb touches the rest plane is obtained, and the moment when the gait limb leaves the rest plane is taken as the gait occurrence time, by This achieves the effect of accurately determining the moment when the gait occurs.
  • the above-mentioned image capture device may be a camera, a camera, etc.; the above-mentioned device parameters may be device internal parameters and/or device external parameters, etc.
  • determining the three-dimensional skeleton of the object to be analyzed at each moment according to the limb image at each moment may include: for each limb image at each moment, according to the Each limb image determines the two-dimensional posture of a plurality of preset skeleton points on the object to be analyzed at each moment; according to at least one two-dimensional posture corresponding to at least one limb image at each moment and respectively set as shooting The calibrated device parameters of at least one image capture device of at least one limb image at each moment are reconstructed to obtain a three-dimensional skeleton of multiple preset skeleton points on the object to be analyzed at each moment.
  • the preset skeleton point may be a preset skeleton point on the image to be analyzed, which may or may not be a limb skeleton point, which is not limited here.
  • each body image may correspond to its own two-dimensional pose.
  • the above-mentioned two-dimensional attitude determination process can be realized based on a deep learning network model such as DeepLabCut, of course, it can also be realized by other methods, which are not limited here.
  • a deep learning network model such as DeepLabCut
  • the three-dimensional skeleton of multiple preset skeleton points on the object to be analyzed is unique at one moment, it can be set according to the multiple two-dimensional poses at this moment and the images respectively set to capture the limb images corresponding to these two-dimensional poses After the camera is calibrated, the device parameters are reconstructed to obtain a three-dimensional skeleton.
  • a triangulation algorithm is used to map multiple two-dimensional poses into a three-dimensional space to obtain a three-dimensional skeleton, thereby obtaining the effect of stably tracking the three-dimensional skeleton.
  • the calibration process of the above-mentioned device parameters can be realized through the following steps: multiple limb images at a time are captured by different image capture devices, and one of these image capture devices is selected as the main image capture device, and use the rest of the image capture devices as secondary image capture devices, the main image capture device and multiple secondary image capture devices are respectively composed of an image capture device combination, that is, each image capture device combination includes a main image capture device device and a secondary image capture device, the main image capture device in the combination of multiple image capture devices is the same, and the secondary image capture devices are different from each other.
  • the calibration plate is placed on a static plane, and multiple image capture devices in each image capture device combination are controlled to take N images of the calibration plate from different angles, wherein N is a positive integer.
  • Device parameters of a plurality of image capturing devices are respectively obtained based on these calibration plate images, which may include a rotation matrix R and a translation vector T.
  • FIG. 3 is a flow chart of a gait analysis method provided in Embodiment 2 of the present application. This embodiment is described on the basis of the above technical solutions.
  • this embodiment optionally, for each three-dimensional skeleton, determine the limb angle of the gait limb of the object to be analyzed at the time corresponding to the three-dimensional skeleton according to the three-dimensional skeleton; Multiple limb angles corresponding to the target skeleton determine the gait parameters of the object to be analyzed.
  • explanations of terms that are the same as or corresponding to those in the foregoing embodiments will not be repeated here.
  • the method of this embodiment may include the following steps:
  • the limb angle can represent the angle of the gait limb at a moment, and different gait limbs can have their own limb angles.
  • limb angles can be calculated in various ways. As an example, taking an animal to be analyzed as an example, its gait limbs include four limbs, the left forelimb, the left hindlimb, the right forelimb, and the right hindlimb. Align the four limbs to the origin in the three-dimensional space, and then calculate the limb angles of different limbs at the moment of one step by calculating the normalized size.
  • the limb angle of the left forelimb can first determine the vector formed by the connection line between the root of the left forelimb and the root of the right forelimb, and the vector formed by the connection line between the root of the left forelimb and the root of the rear forelimb, and then combine these two The angle between two vectors is used as the limb angle.
  • Other ways can also be used to calculate the limb angles of different gait limbs, which are not limited here.
  • the gait parameters are the parameters when the gait limb occurs
  • the limb angle corresponding to the target skeleton can be determined from multiple limb angles, that is, the limb angle at the moment when the gait occurs.
  • the limb angle can reflect the orientation angle of the corresponding gait limb, and the orientation angle is related to the gait
  • the gait parameters can be determined according to the multiple limb angles belonging to the moment when the gait occurs
  • the plurality of limb angles may include angles of each gait limb at a plurality of gait occurrence times.
  • the gait limb angle of the object to be analyzed is determined at the moment corresponding to the 3D skeleton through the three-dimensional skeleton, and then the gait parameters are determined according to multiple limb angles belonging to the moment when the gait occurs, This results in an accurate determination of the gait parameters.
  • An optional technical solution is to determine the gait parameters of the object to be analyzed according to multiple limb angles corresponding to multiple target skeletons among the limb angles at multiple moments, which may include: obtaining multiple time-continuous The gait occurrence time series formed at the time of gait occurrence; for each gait occurrence time series, according to the multiple limb angles under each gait occurrence time series, determine the position of the object to be analyzed under each gait occurrence time series The stance phase information of the stance phase and the swing phase information of the swing phase; according to the stance phase information and the swing phase information under each gait occurrence time series, the gait parameters of the object to be analyzed are determined.
  • the multiple moments under the gait occurrence time series are multiple gait occurrence moments that are continuous in time. Since the object to be analyzed may have gait occurrence for a period of time, gait stop for a period of time, gait occurrence and gait stop Therefore, the sequence number of the gait occurrence time series may be one, two or more, which is not limited here.
  • the stance phase information of the stance phase and the swing phase information of the swing phase under which the object to be analyzed can be determined according to the angles of multiple limbs under it, where the swing phase can be understood as the The state of the lower limbs swinging forward in the air during the gait process, while the stance phase can be understood as the state of the lower limbs in the air support plane during the gait process.
  • the two can be distinguished by the change direction of the limb angle. If the limb angle becomes larger, the It shows that it is a swing phase, and if the angle of the limbs becomes smaller, it shows that it is a support phase.
  • the swing phase information may be represented by a duration of a swing phase, swing amplitude, etc.
  • the support phase information may be represented by a duration of a support phase, etc., which are not limited herein.
  • the gait parameters of the object to be analyzed can be determined according to the support phase information and swing phase information under each gait occurrence time series. and so on, thus achieving the effect of accurately determining the gait parameters.
  • the calibration board is used to calibrate the internal and external parameters of multiple cameras, and the multiple cameras are used to simultaneously photograph animals in a freely moving state;
  • the two-dimensional posture is obtained based on the deep learning network model, and then Three-dimensional reconstruction of multiple two-dimensional poses based on the triangulation algorithm to obtain a three-dimensional skeleton; automatic segmentation of object behavior based on machine learning models, and extraction of gait occurrence moments from multiple moments; detection of the horizontal plane of the three-dimensional space, that is, the static plane , and carry out the rotation of the three-dimensional skeleton; carry out the filtering processing of the three-dimensional skeleton; carry out the calculation of the limb angle of the three-dimensional skeleton; finally determine the gait parameters of the animal according to the multiple limb angles at the time when the gait occurs.
  • FIG. 5 is a flow chart of a gait analysis method provided in Embodiment 3 of the present application.
  • acquiring the body images of the object to be analyzed on the stationary plane at multiple moments may include: acquiring the image of the object to be analyzed on the stationary plane at multiple times based on the image capturing device in the image capturing device Limb image at a moment, wherein the image capturing device includes at least four image capturing devices and a suspended and fixed planar device configured to provide a stationary plane, the device color of the planar device includes the environment ground in the living environment of the object to be analyzed The color related to the ground color, at least three image capture devices in at least four image capture devices are fixed on the same side of the plane device as the object to be analyzed, and at least one image capture device is fixed on the other side of the plane device on the side.
  • explanations of terms that are the same as or corresponding to those in the foregoing embodiments will not be repeated here.
  • the method of this embodiment may include the following steps:
  • the image capturing device Based on the image capturing device in the image capturing device, acquire limb images of the subject to be analyzed on a static plane at multiple moments, wherein the image capturing device includes at least four image capturing devices, and a suspension set to provide a static plane
  • the device color of the planar device includes a color related to the ground color of the environmental ground under the living environment of the object to be analyzed, and at least three of the at least four image capture devices are fixed on the plane device and are to be analyzed.
  • On one side of the same side as the analysis object and at least one imaging device is fixed on the other side of the planar device.
  • the planar device may be a device configured to provide a static plane in the image capture device, and its device color may be a color related to the ground color, and the ground color may be the color of the environmental ground in the living environment of the object to be analyzed.
  • the number of image capturing devices can be at least four, and at least three of the image capturing devices are fixed on the same side of the plane device as the object to be analyzed.
  • the calibration plate images taken by at least three image capture devices have overlapped parts, and multiple image capture devices cooperate with each other, so that even if the gait limbs in the field of view of one image capture device are blocked, the rest of the image capture devices will also be blocked.
  • the occluded gait limbs can be photographed, thereby ensuring the effective reconstruction of the subsequent three-dimensional skeleton; at least one of the image capturing devices is fixed on the other side of the plane device, which is to clearly capture the footsteps of the subject to be analyzed and body movement. In this way, limb images can be captured completely and synchronously at multiple times based on multiple image capture devices.
  • At least three of the at least four image capturing devices are fixed on the side of the planar device on the same side as the object to be analyzed, and at least one image capturing device is fixed on the side of the planar device.
  • the color of the device includes colors related to the ground color of the environmental ground in the living environment of the object to be analyzed, thereby avoiding the situation that the object to be analyzed affects its original law of motion due to the fear of heights, so that the subsequent obtained gait parameters in line with the real situation.
  • the image capture device may include a camera holder 10 configured to fix a first camera 11 (which is an image capture device), a second camera 13 (which is another image capture device) And planar device 12, wherein camera mount 10 can be fixed on the top of arbitrary ceiling, plate etc., convenient dismounting;
  • the number of cameras of the first camera 11 is 4 (Camera1-Camera4), and 4 first cameras 11 are respectively installed on camera On the bottom of the four pillars of the fixed frame 10, its orientation is horizontal and slightly downward, thereby ensuring the integrity of the object to be analyzed in the field of view;
  • the plane device 12 is a translucent frosted plate, gray black, so that the The subject to be analyzed feels that he is on the ground;
  • the number of cameras of the second camera 13 is 1 (Camera5), and its orientation is vertically upward, thereby clearly photographing the footsteps and body
  • FIG. 7 is a structural block diagram of a gait analysis device provided in Embodiment 4 of the present application, and the device is configured to execute the gait analysis method provided in any of the above embodiments.
  • the device and the gait analysis method of the above-mentioned embodiment belong to the same idea.
  • the device may include: a limb image acquisition module 410 , a three-dimensional skeleton determination module 420 , a target skeleton acquisition module 430 and a gait parameter determination module 440 .
  • the limb image acquisition module 410 is configured to acquire the limb images of the object to be analyzed located on the stationary plane at multiple moments; the three-dimensional skeleton determination module 420 is configured to, for the limb images at each moment, according to the limb images at each moment The image determines the three-dimensional skeleton of the object to be analyzed at each moment; the target skeleton obtaining module 430 is configured to determine multiple gait occurrence moments of the gait of the object to be analyzed from multiple moments according to the three-dimensional skeleton at multiple moments, The three-dimensional skeleton at each gait occurrence moment is used as the target skeleton; the gait parameter determination module 440 is configured to determine the gait parameters of the object to be analyzed according to multiple target skeletons.
  • the target skeleton obtaining module 430 may include:
  • the skeleton time series segmentation unit is set to obtain the skeleton time series corresponding to the three-dimensional skeleton at multiple times, and takes the object behavior of the object to be analyzed as the segmentation target to segment the skeleton time series;
  • the skeleton time series clustering unit is set to combine the target moving speed of the object to be analyzed under the multiple segmented skeleton time series, and cluster the multiple segmented skeleton time series on the object behavior;
  • the first determination unit of the gait occurrence time It is set to determine multiple gait occurrence moments at which the object to be analyzed takes a gait at multiple moments according to the clustering results.
  • the three-dimensional skeleton is a three-dimensional skeleton including multiple limb bone points on the gait limb of the object to be analyzed;
  • the target skeleton obtaining module 430 may include:
  • the three-dimensional skeleton rotation unit is set to rotate the three-dimensional skeleton for each three-dimensional skeleton, so that the skeleton plane where the three-dimensional skeleton after rotation is located is parallel to the stationary plane; the three-dimensional skeleton judging unit is set to judge the step according to the three-dimensional skeleton after the rotation whether the stance limbs touch the static plane; the second determination unit of the gait occurrence moment is set to determine the multiple gait occurrences of the gait of the object to be analyzed from Dongge time according to the multiple judgment results corresponding to the multiple moments time.
  • the gait parameter determination module 440 may include:
  • the limb angle determination unit is set to determine the limb angle of the gait limb of the object to be analyzed at the moment corresponding to each three-dimensional skeleton according to each three-dimensional skeleton; the gait parameter determination unit is set to be based on Among the limb angles at multiple moments, multiple limb angles corresponding to multiple target skeletons are used to determine the gait parameters of the object to be analyzed.
  • the optional gait parameter determination unit is set to:
  • Phase information For each gait occurrence time series, according to multiple limb angles under each gait occurrence time series, determine the support phase information and swing phase swing of the object to be analyzed under each gait occurrence time series Phase information: According to the support phase information and swing phase information under each gait occurrence time series, the gait parameters of the object to be analyzed are determined.
  • the three-dimensional skeleton determination module 420 may include:
  • a two-dimensional pose determination unit configured to determine the two-dimensional poses of multiple preset skeleton points on the object to be analyzed at each moment according to each limb image at each moment for each limb image at each moment ;
  • the three-dimensional skeleton determination unit is configured to be based on at least one two-dimensional pose corresponding to at least one limb image at each moment and at least one image capture device respectively configured to capture at least one limb image at each moment after calibration
  • the device parameters are reconstructed to obtain the three-dimensional skeleton of multiple preset bone points on the object to be analyzed at each moment.
  • the limb image acquisition module 410 may include:
  • the limb image acquisition unit is configured to acquire limb images of an object to be analyzed on a stationary plane at multiple moments based on the image capturing device in the image capturing device; wherein the image capturing device includes at least four image capturing devices, and is configured to Provide a suspended and fixed planar device of a static plane, the device color of the planar device includes a color related to the ground color of the environmental ground under the living environment of the object to be analyzed, and at least three of the at least four image capture devices are fixed on the One side of the planar device on the same side as the object to be analyzed and at least one image capture device are fixed on the other side of the planar device.
  • the gait analysis device uses the limb image acquisition module 410 to acquire the limb images of the object to be analyzed at multiple moments in a state of free movement on a stationary plane; due to the gait of the object to be analyzed It occurs in three-dimensional space, so the three-dimensional skeleton determination module 420 determines the three-dimensional skeleton of the object to be analyzed at each moment according to the limb image at each moment, thus ensuring the subsequent determination
  • the module 430 obtained through the target skeleton can be determined from multiple moments according to the three-dimensional skeleton at multiple moments
  • the target skeleton is a three-dimensional skeleton at the gait occurrence moment, the gait parameters can be determined according to multiple target skeletons through the gait parameter determination module 440 .
  • the gait analysis device provided in the embodiment of the present application can execute the gait analysis method provided in any embodiment of the present application, and has corresponding functional modules and effects for executing the method.
  • the multiple units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, multiple functional units
  • the titles are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application.
  • FIG. 8 is a schematic structural diagram of a gait analysis device provided in Embodiment 5 of the present application.
  • the device includes a memory 510 , a processor 520 , an input device 530 and an output device 540 .
  • the number of processors 520 in the device can be one or more, and one processor 520 is taken as an example in FIG. , in FIG. 8, the connection through the bus 550 is taken as an example.
  • the memory 510 can be configured to store software programs, computer-executable programs and modules, such as program instructions/modules corresponding to the gait analysis method in the embodiment of the present application (for example, in the gait analysis device Limb image acquisition module 410, three-dimensional skeleton determination module 420, target skeleton acquisition module 430 and gait parameter determination module 440).
  • the processor 520 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 510 , that is, realizes the above-mentioned gait analysis method.
  • the memory 510 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, and the like.
  • the memory 510 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices.
  • memory 510 may include memory located remotely from processor 520, which remote memory may be connected to the device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
  • the input device 530 may be configured to receive input of numeric or character information, and generate key signal input related to user settings and function control of the device.
  • the output device 540 may include a display device such as a display screen.
  • Embodiment 6 of the present application provides a storage medium containing computer-executable instructions, the computer-executable instructions are used to perform a gait analysis method when executed by a computer processor, the method comprising:
  • the three-dimensional skeleton at a moment determines the multiple gait occurrence moments of the gait of the object to be analyzed from multiple moments, and uses the three-dimensional skeleton at each gait occurrence moment as the target skeleton; determines the target skeleton according to the multiple target skeletons
  • the object's gait parameters The object's gait parameters.
  • a storage medium containing computer-executable instructions provided by an embodiment of the present application the computer-executable instructions are not limited to the method operations described above, and can also perform related steps in the gait analysis method provided by any embodiment of the present application. operate.
  • the present application can be implemented by means of software and necessary general-purpose hardware, or can also be implemented by means of hardware.
  • the technical solution of the present application can be embodied in the form of software products in essence, and the computer software products can be stored in computer-readable storage media, such as computer floppy disks, read-only memory (Read-Only Memory, ROM), random access Memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disc, etc., including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the described embodiment of the present application. Methods.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Psychiatry (AREA)
  • Social Psychology (AREA)
  • Human Computer Interaction (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

本文公开了一种步态分析方法、装置、设备及存储介质。该步态分析方法包括：获取位于静止平面上的待分析对象在多个时刻下的肢体图像；针对于每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架；根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架；根据多个目标骨架确定待分析对象的步态参数。

Description

步态分析方法、装置、设备及存储介质
本申请要求在2021年11月30日提交中国专利局、申请号为202111439683.7的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，例如涉及一种步态分析方法、装置、设备及存储介质。
背景技术
步态分析可以对病症或是急性疼痛后与步态相关的变化提供自动化、快速、客观和完整的分析结果。步态分析的前提是需要动物在特定跑道上进行持续性运动。对于动物而言，这样的被迫运动会导致其脱离原本的运动规律，由此得到的步态参数脱离了真实情况，应用价值有限。
发明内容
本申请提供了一种步态分析方法、装置、设备及存储介质，以实现对处于自由移动状态下的待分析对象进行步态分析的效果。
本申请提供了一种步态分析方法，可以包括：
获取位于静止平面上的待分析对象在多个时刻下的肢体图像；
针对每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架；
根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架；
根据多个目标骨架确定待分析对象的步态参数。
本申请还提供了一种步态分析装置，可以包括：
肢体图像获取模块，设置为获取位于静止平面上的待分析对象在多个时刻下的肢体图像；
三维骨架确定模块，设置为针对每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架；
目标骨架得到模块，设置为根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨 架作为目标骨架；
步态参数确定模块，设置为根据多个目标骨架确定待分析对象的步态参数。
本申请还提供了一种步态分析设备，可以包括：
一个或多个处理器；
存储器，设置为存储一个或多个程序；
当一个或多个程序被一个或多个处理器执行，使得一个或多个处理器实现上述的步态分析方法。
本申请还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述的步态分析方法。
附图说明
图1是本申请实施例一提供的一种步态分析方法的流程图；
图2是本申请实施例一提供的一种步态分析方法中二维姿态的示意图；
图3是本申请实施例二提供的一种步态分析方法的流程图；
图4是本申请实施例二提供的一种步态分析方法中可选示例的流程图；
图5是本申请实施例三提供的一种步态分析方法的流程图；
图6是本申请实施例三提供的一种步态分析方法中图像拍摄设备的示意图；
图7是本申请实施例四提供的一种步态分析装置的结构框图；
图8是本申请实施例五提供的一种步态分析设备的结构示意图。
具体实施方式
下面结合附图和实施例对本申请进行说明。此处所描述的具体实施例仅仅用于解释本申请。为了便于描述，附图中仅示出了与本申请相关的部分。
实施例一
图1是本申请实施例一提供的一种步态分析方法的流程图。本实施例可适用于对处于自由移动状态下的待分析对象进行步态分析的情况。该方法可以由本申请实施例提供的步态分析装置来执行，该装置可由软件和/或硬件的方式实现，该装置可以集成在步态分析设备上，该设备可以是用户终端或服务器。
参见图1，本申请实施例的方法包括如下步骤：
S110、获取位于静止平面上的待分析对象在多个时刻下的肢体图像。
待分析对象可以是待分析其步态参数的对象，其可以是动物和/或人，在此未做限定；静止平面可以是静止不动的位于地面或是悬空固定的平面，因此位于静止平面上的待分析对象可以在静止平面上自由移动。肢体图像可以是在一个时刻下被拍摄的包含待分析对象的与步态有关的步态肢体的图像，该步态肢体可以是动物的四肢、人的两条腿等，在此未做限定；该一个时刻下的肢体图像的图像数量可以是一张、两张或是多张，在此亦未做限定。
S120、针对于每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架。
由于待分析对象的步态是发生在三维空间中，因此为了准确确定其的步态参数，针对每个时刻下的肢体图像，可以根据该每个时刻下的肢体图像确定其在该每个时刻下的三维骨架。由于待分析对象在一个时刻下的三维骨架是唯一的，因此当该一个时刻下的肢体图像的图像数量包括至少两张时，可以根据该至少两张肢体图像共同确定其在该一个时刻下的三维骨架。
在此基础上，可选的，为了提高后续的步态参数的确定精度，可以对本步骤得到的三维骨架进行滤波来去除其中的噪声点，以此得到稳定的步态，滤波方式可以是中值滤波、均值滤波等，在此未做限定。
S130、根据多个时刻下的三维骨架确定多个时刻中待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架。
步态是真正走起来时的状态，其是移动中的一种，如嗅探、转向、攀爬、降落、前进一步退后两步等只能称为移动，不能称为步态。换言之，待分析对象在多个时刻下不一定都会发生步态。因此，为了准确确定待分析对象的步态参数，可以先确定其在何时发生步态，发生步态的时刻可以称为步态发生时刻，然后再确定待分析对象在步态发生时刻时的步态参数。在实际应用中，一个时刻是否为步态发生时刻，可以根据该时刻下的、与该时刻关联的关联时刻下的和/或多个时刻下的三维骨架进行确定，在此未做限定。
S140、根据多个目标骨架确定待分析对象的步态参数。
由于目标骨架是步态发生时刻下的三维骨架，例如是处于自由移动状态下的待分析对象在发生步态时的三维骨架，因此根据多个目标骨架确定出的步态参数是符合待分析对象原本的运动规律的步态参数，其可以作为药物筛选、行为分析、动物疾病造模、神经环路机制等多项内容的研究依据。
在上述技术方案中，第一、由于其可以自动确定步态发生时刻，因此无需强迫待分析对象在跑道上进行跑动，这意味着由此得到的步态参数符合其原本的运动规律。第二、由于是通过分析肢体图像来确定步态参数，由此无需剃除 待分析对象脚底的毛发以涂上荧光涂料来获取脚印，对于待分析对象而言是无创的。第三，无需预先训练待分析对象在跑道上进行跑动，人力成本和时间成本较低。第四，无需通过单独配置的压力传感器、荧光涂料等来获取脚印，经济成本较低。
本申请实施例的技术方案，通过获取位于静止平面上即处于自由移动状态下的待分析对象在多个时刻下的肢体图像；由于待分析对象的步态是发生在三维空间中，因此针对每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架，由此保证了后续确定的待分析对象的步态参数的准确性；由于待分析对象在多个时刻下并非一直发生步态，因此可以根据多个时刻下的三维骨架从多个时刻中确定出待分析对象发生步态的多个步态发生时刻；由于目标骨架是步态发生时刻下的三维骨架，因此可以根据多个目标骨架确定步态参数。上述技术方案，可以准确得到处于自由移动状态下的待分析对象的步态参数。
一种可选技术方案，根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，可包括：获取多个时刻下的三维骨架对应的骨架时间序列，并以待分析对象的对象行为为切分目标，对骨架时间序列进行切分；结合待分析对象在多个切分后的骨架时间序列下的目标移动速度，对多个切分后的骨架时间序列在对象行为上进行聚类；根据聚类结果确定多个时刻中待分析对象发生步态的多个步态发生时刻。其中，骨架时间序列可以是对应多个时刻下的三维骨架的时间序列，以待分析对象的对象行为为基准，对骨架时间序列进行切分，以使待分析对象在切分后的多个骨架时间序列下的对象行为是不完全相同的，该对象行为可以是发生步态、嗅探、转向、攀爬、降落、前进一步退后两步、静止等，在此未做限定；上述骨架时间序列的切分过程可以是基于机器学习模型实现的，也可以是基于其余方式实现的，在此亦未做限定。在此基础上，为了提高对象行为分类的准确性，可以结合待分析对象在切分后的每个骨架时间序列下的目标移动速度(如根据三维骨架确定的平均、最大或是最小移动速度等)，对多个切分后的骨架时间序列在对象行为上进行聚类，得到对应于每个对象行为的聚类结果(即聚类后的骨架时间序列)，并从每个聚类后的骨架时间序列中找到与对象行为是发生步态对应的步态发生时间序列，将该步态发生时间序列下的多个时刻作为步态发生时刻。实际应用中，可选的，可以先对切分后的骨架时间序列进行统一流形逼近与投影(Uniform Manifold Approximation and Projection，UMAP)降维，然后再对多个降维后的骨架时间序列进行层次聚类，由此保证了聚类的准确性。再可选的，在对多个切分后的骨架时间序列进行聚类时，可以以每个切分后的骨架时间序列为单位进行聚类，也可以以每个切分后的骨架时间序列下的每个三维骨架为单位进行 聚类，在此未做限定。上述技术方案，通过将三维骨架和目标移动速度相结合，达到了步态发生时刻的准确确定的效果。
一种可选的技术方案，三维骨架是包括待分析对象的步态肢体上的多个肢体骨骼点的三维的骨架；根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，可以包括：针对于每个三维骨架，对三维骨架进行旋转，以使旋转后的三维骨架所在的骨架平面平行于静止平面；根据旋转后的三维骨架判断步态肢体是否接触到静止平面；根据多个时刻分别对应的多个判断结果，从多个时刻中确定待分析对象发生步态的多个步态发生时刻。其中，肢体骨骼点可以包括步态肢体上的骨骼点，三维骨架可以是由这些肢体骨骼点构成的三维空间上的骨架。由于三维骨架的重建过程是基于设置为拍摄肢体图像的图像拍摄装置的装置参数实现的，装置参数是通过对图像拍摄装置进行标定后得到的，受到标定精度的影响，三维骨架所在的骨架平面不一定平行于静止平面。为此，为了通过判断步态肢体是否接触到静止平面来确定待分析对象是否发生步态，可以旋转三维骨架(如通过法向量叉乘原理等方式进行旋转)，以使旋转后的三维骨架所在的骨架平面平行于静止平面，该静止平面多为三维空间中的水平面；进而得到步态肢体是否接触到静止平面的判断结果，并将步态肢体离开静止平面的时刻作为步态发生时刻，由此达到了准确确定步态发生时刻的效果。上述图像拍摄装置可以是摄像头、照相机等；上述装置参数可以装置内参和/或装置外参等。
另一种可选技术方案，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架，可以包括：针对于每个时刻下的每个肢体图像，根据每个时刻下的每个肢体图像确定待分析对象上的多个预设骨骼点在每个时刻下的二维姿态；根据与每个时刻下的至少一个肢体图像分别对应的至少一个二维姿态和分别设置为拍摄每个时刻下的至少一个肢体图像的至少一个图像拍摄装置的标定后的装置参数，重建得到待分析对象上的多个预设骨骼点在每个时刻下的三维骨架。其中，预设骨骼点可以是待分析图像上的预先设置的骨骼点，其可能是肢体骨骼点也可能不是肢体骨骼点，在此未做限定。针对一个时刻下的一个肢体图像，根据该肢体图像确定待分析对象上的多个预设骨骼点在该时刻下的二维姿态，其可以通过多个预设骨骼点间的连线进行表示，示例性的，参见图2。当该时刻下的肢体图像的图像数量是至少两张时，每张肢体图像可以分别对应有各自的二维姿态。上述二维姿态的确定过程可以基于深度学习网络模型如DeepLabCut实现，当然也可通过其余方式实现，在此未做限定。由于待分析对象上的多个预设骨骼点在一个时刻下的三维骨架唯一，因此可以根据该时刻下的多个二维姿态和分别设置为拍摄与这些二维姿态各自对应的肢体图像的图像拍摄装置的标定后的装置参数重建得到三维骨架，如通过三角测量算法将多 个二维姿态映射到三维空间中以得到三维骨架，由此得到了稳定追踪三维骨架的效果。
在此基础上，可选的，可以通过如下步骤实现上述装置参数的标定过程：一个时刻下的多个肢体图像分别是由不同的图像拍摄装置拍摄得到，从这些图像拍摄装置中选择一个作为主图像拍摄装置，并且将其余的图像拍摄装置作为副图像拍摄装置，将主图像拍摄装置和多个副图像拍摄装置分别组成一个图像拍摄装置组合，即每个图像拍摄装置组合中包括一个主图像拍摄装置和一个副图像拍摄装置，多个图像拍摄装置组合中的主图像拍摄装置是相同的，且副图像拍摄装置是互不相同的。将标定板放置于静止平面上，控制每个图像拍摄装置组合中的多个图像拍摄装置分别从不同角度对标定板拍摄N张标定板图像，其中，N是正整数。基于这些标定板图像分别得到多个图像拍摄装置的装置参数，其可以包括旋转矩阵R和平移向量T。
实施例二
图3是本申请实施例二提供的一种步态分析方法的流程图。本实施例以上述技术方案为基础进行说明。本实施例中，可选的，针对于每个三维骨架，根据三维骨架确定待分析对象的步态肢体在三维骨架对应的时刻下的肢体角度；根据多个时刻下的肢体角度中与多个目标骨架对应的多个肢体角度，确定待分析对象的步态参数。其中，与上述实施例相同或相应的术语的解释在此不再赘述。
参见图3，本实施例的方法可以包括如下步骤：
S210、获取位于静止平面上的待分析对象在多个时刻下的肢体图像。
S220、针对于每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架。
S230、根据多个时刻下的三维骨架确定多个时刻中待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架。
S240、针对于每个三维骨架，根据三维骨架确定待分析对象的步态肢体在与三维骨架对应的时刻下的肢体角度。
肢体角度可以表示步态肢体在一个时刻下的角度，不同步态肢体可以对应有各自的肢体角度。在实际应用中，肢体角度可以通过多种方式计算得到，示例性的，以待分析对象是动物为例，其步态肢体包括左前肢、左后肢、右前肢和右后肢这四个肢体，可以将这四个肢体对齐至三维空间中的原点上，然后通过求取归一化大小来计算得到不同肢体在一步态发生时刻下的肢体角度。以左前肢的肢体角度为例，其可以先确定左前肢的根部和右前肢的根部的连线构成 的向量、以及左前肢的根部和后前肢的根部的连线构成的向量，然后将这两个向量的夹角作为肢体角度。当然，还可以通过其余方式计算得到不同步态肢体的肢体角度，在此未做限定。
S250、根据多个时刻下的肢体角度中与多个目标骨架对应的多个肢体角度，确定待分析对象的步态参数。
由于步态参数是步态肢体发生步态时的参数，因此可以先从多个肢体角度中确定与目标骨架对应的肢体角度，即属于步态发生时刻下的肢体角度。在此基础上，由于肢体角度可以反映出相应的步态肢体的朝向角度，其中朝向角度是与步态有关的，因此可以根据属于步态发生时刻下的多个肢体角度确定出步态参数，该多个肢体角度可以包括每个步态肢体在多个步态发生时刻下的角度。
本申请实施例的技术方案，通过三维骨架确定待分析对象的步态肢体在与该三维骨架对应的时刻下的肢体角度，进而根据属于步态发生时刻下的多个肢体角度确定步态参数，由此达到了步态参数的准确确定的效果。
一种可选的技术方案，根据多个时刻下的肢体角度中与多个目标骨架对应的多个肢体角度，确定待分析对象的步态参数，可以包括：获取由在时间上连续的多个步态发生时刻构成的步态发生时间序列；针对每个步态发生时间序列，根据每个步态发生时间序列下的多个肢体角度，确定待分析对象在每个步态发生时间序列下的支撑相的支撑相信息以及摆动相的摆动相信息；根据每个步态发生时间序列下的支撑相信息以及摆动相信息，确定待分析对象的步态参数。其中，步态发生时间序列下的多个时刻是在时间上连续的多个步态发生时刻，由于待分析对象有可能出现发生步态一段时间、停止步态一段时间、发生步态和停止步态交替出现等情况，因此该步态发生时间序列的序列数量有可能是一个、两个或多个，在此未做限定。在一个步态发生时间序列下，可以根据其下的多个肢体角度来确定待分析对象在其下的支撑相的支撑相信息和摆动相的摆动相信息，其中摆动相可以理解为在发生步态过程中下肢在空中向前摆动的状态，而支撑相可以理解为在发生步态过程中下肢在空中支撑平面的状态，二者可通过肢体角度的变化方向进行区分，如肢体角度变大则说明是摆动相，如肢体角度变小则说明是支撑相。摆动相信息可以通过一次摆动相的持续时间、摆动幅度等进行表示，支撑相信息可以通过一次支撑相的持续时间等进行表示，在此未做限定。可以根据每个步态发生时间序列下的支撑相信息和摆动相信息确定待分析对象的步态参数，该步态参数可以通过支撑相时间、摆动相时间、支撑相比例、步频、步幅等进行表示，由此达到了步态参数的准确确定的效果。
为了从整体上理解上述步骤的实现过程，下面结合示例对其进行示例性的说明。示例性的，参见图4，利用标定板进行多个摄像头的内参和外参的标定， 并利用该多个摄像头同步拍摄处于自由移动状态下的动物；基于深度学习网络模型得到二维姿态，然后基于三角测量算法将多个二维姿态进行三维重建，得到三维骨架；基于机器学习模型进行对象行为的自动切分，从多个时刻中提取出步态发生时刻；检测三维空间的水平面即静止平面，并进行三维骨架的旋转；进行三维骨架的滤波处理；进行三维骨架的四肢角度计算；最后根据步态发生时刻下的多个肢体角度确定动物的步态参数。
实施例三
图5是本申请实施例三提供的一种步态分析方法的流程图。本实施例以上述技术方案为基础进行说明。本实施例中，可选的，获取位于静止平面上的待分析对象在多个时刻下的肢体图像，可以包括：基于图像拍摄设备中的图像拍摄装置获取位于静止平面上的待分析对象在多个时刻下的肢体图像，其中，图像拍摄设备包括至少四个图像拍摄装置、及设置为提供静止平面的悬空固定的平面装置，平面装置的装置颜色包括与待分析对象的生活环境下的环境地面的地面颜色相关的颜色，至少四个图像拍摄装置中的至少三个图像拍摄装置固定在平面装置的与待分析对象同侧的一侧上、以及至少一个图像拍摄装置固定在平面装置的另一侧上。其中，与上述实施例相同或相应的术语的解释在此不再赘述。
参见图5，本实施例的方法可以包括如下步骤：
S310、基于图像拍摄设备中的图像拍摄装置获取位于静止平面上的待分析对象在多个时刻下的肢体图像，其中，图像拍摄设备包括至少四个图像拍摄装置、以及设置为提供静止平面的悬空固定的平面装置，平面装置的装置颜色包括与待分析对象的生活环境下的环境地面的地面颜色相关的颜色，至少四个图像拍摄装置中的至少三个图像拍摄装置固定在平面装置的与待分析对象同侧的一侧上、以及至少一个图像拍摄装置固定在平面装置的另一侧上。
平面装置可以是图像拍摄设备中设置为提供静止平面的装置，其装置颜色可以是与地面颜色相关的颜色，该地面颜色可以是待分析对象的生活环境下的环境地面的颜色，这是为了在平面装置悬空固定时，位于静止平面上的待分析对象也会感觉自己在地面上，不会因为高空恐惧而影响其原本的运动规律。图像拍摄装置的装置数量可以是至少四个，其中的至少三个图像拍摄装置固定在平面装置的与待分析对象同侧的一侧，这是为了便于标定每个图像拍摄装置的装置参数，因为至少三个图像拍摄装置拍摄出的标定板图像才会存在重合部分，而且多个图像拍摄装置相互配合，由此即使一个图像拍摄装置视野中的步态肢体被遮挡住，其余的图像拍摄装置也能拍摄到该被遮挡住的步态肢体，由此保证了后续三维骨架的有效重建；其中的至少一个图像拍摄装置固定在平面装置 的另一侧，这是为了清晰拍摄到待分析对象的脚步和肢体运动。由此，可以基于多个图像拍摄装置在多个时刻下完整且同步的拍摄到肢体图像。
S320、针对于每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架。
S330、根据多个时刻下的三维骨架确定多个时刻中待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架。
S340、根据多个目标骨架确定待分析对象的步态参数。
本申请实施例的技术方案，通过至少四个图像拍摄装置中的至少三个图像拍摄装置固定在平面装置的与待分析对象同侧的一侧上、及至少一个图像拍摄装置固定在平面装置的另一侧上，由此可以保证每个图像拍摄装置的装置参数的有效标定、以及多个时刻下步态肢体的完整且同步的拍摄；同时，设置为提供静止平面的悬空固定的平面装置的装置颜色包括与待分析对象的生活环境下的环境地面的地面颜色相关的颜色，由此可以避免出现待分析对象因高空恐惧而影响其原本的运动规律的情况，以使后续得到的步态参数符合真实情况。
为了理解上述图像拍摄设备的结构，下面可结合示例对其进行示例性的说明。示例性的，参见图6，该图像拍摄设备可以包括设置为固定第一摄像头11(其是一种图像拍摄装置)的摄像头固定架10、第二摄像头13(其是另一种图像拍摄装置)和平面装置12，其中摄像头固定架10可以固定在任意天花板、板材等的顶部，方便拆卸；第一摄像头11的摄像头数量是4个(Camera1-Camera4)，4个第一摄像头11分别安装在摄像头固定架10的四个柱子的底端上，其朝向为水平方向，微斜向下，由此保证视野中待分析对象的完整性；平面装置12是半透明磨砂板，灰黑色，由此让待分析对象感觉自己在地面上；第二摄像头13的摄像头数量是1个(Camera5)，其朝向为垂直向上，由此从底部来清晰拍摄到待分析对象的脚步和肢体运动。上述图像拍摄设备的组成较为简单，经济成本较低；而且易于操作，人力成本较低。
实施例四
图7为本申请实施例四提供的一种步态分析装置的结构框图，该装置设置为执行上述任意实施例所提供的步态分析方法。该装置与上述实施例的步态分析方法属于同一个构思，在步态分析装置的实施例中未详尽描述的细节内容，可以参考上述步态分析方法的实施例。参见图7，该装置可以包括：肢体图像获取模块410、三维骨架确定模块420、目标骨架得到模块430和步态参数确定模块440。
肢体图像获取模块410，设置为获取位于静止平面上的待分析对象在多个时 刻下的肢体图像；三维骨架确定模块420，设置为针对每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架；目标骨架得到模块430，设置为根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架；步态参数确定模块440，设置为根据多个目标骨架确定待分析对象的步态参数。
可选的，目标骨架得到模块430，可以包括：
骨架时间序列切分单元，设置为获取多个时刻下的三维骨架对应的骨架时间序列，并以待分析对象的对象行为为切分目标，对骨架时间序列进行切分；骨架时间序列聚类单元，设置为结合待分析对象在多个切分后的骨架时间序列下的目标移动速度，对多个切分后的骨架时间序列在对象行为上进行聚类；步态发生时刻第一确定单元，设置为根据聚类结果确定多个时刻中待分析对象发生步态的多个步态发生时刻。
可选的，三维骨架是包括待分析对象的步态肢体上的多个肢体骨骼点的三维的骨架；目标骨架得到模块430，可以包括：
三维骨架旋转单元，设置为针对每个三维骨架，对三维骨架进行旋转，以使旋转后的三维骨架所在的骨架平面平行于静止平面；三维骨架判断单元，设置为根据旋转后的三维骨架判断步态肢体是否接触到静止平面；步态发生时刻第二确定单元，设置为根据与多个时刻分别对应的多个判断结果，从东哥时刻中确定待分析对象发生步态的多个步态发生时刻。
可选的，步态参数确定模块440，可以包括：
肢体角度确定单元，设置为针对于每个三维骨架，根据每个三维骨架确定待分析对象的步态肢体在与每个三维骨架对应的时刻下的肢体角度；步态参数确定单元，设置为根据多个时刻下的肢体角度中与多个目标骨架对应的多个肢体角度，确定待分析对象的步态参数。
在此基础上，可选的，步态参数确定单元，设置为：
获取由在时间上连续的多个步态发生时刻构成的步态发生时间序列；
针对于每个步态发生时间序列，根据每个步态发生时间序列下的多个肢体角度，确定待分析对象在每个步态发生时间序列下的支撑相的支撑相信息和摆动相的摆动相信息；根据每个步态发生时间序列下的支撑相信息以及摆动相信息，确定待分析对象的步态参数。
可选的，三维骨架确定模块420，可以包括：
二维姿态确定单元，设置为针对每个时刻下的每个肢体图像，根据每个时刻下的每个肢体图像确定待分析对象上的多个预设骨骼点在每个时刻下的二维姿态；三维骨架确定单元，设置为根据与每个时刻下的至少一个肢体图像分别对应的至少一个二维姿态和分别设置为拍摄每个时刻下的至少一个肢体图像的至少一个图像拍摄装置的标定后的装置参数，重建得到待分析对象上的多个预设骨骼点在每个时刻下的三维骨架。
可选的，肢体图像获取模块410，可以包括：
肢体图像获取单元，设置为基于图像拍摄设备中的图像拍摄装置获取位于静止平面上的待分析对象在多个时刻下的肢体图像；其中，图像拍摄设备包括至少四个图像拍摄装置、以及设置为提供静止平面的悬空固定的平面装置，平面装置的装置颜色包括与待分析对象的生活环境下的环境地面的地面颜色相关的颜色，至少四个图像拍摄装置中的至少三个图像拍摄装置固定在平面装置的与待分析对象同侧的一侧上、及至少一个图像拍摄装置固定在平面装置的另一侧上。
本申请实施例四所提供的步态分析装置，通过肢体图像获取模块410获取位于静止平面上即处于自由移动状态下的待分析对象在多个时刻下的肢体图像；由于待分析对象的步态是发生在三维空间中，因此通过三维骨架确定模块420针对每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架，由此保证了后续确定的待分析对象的步态参数的准确性；由于待分析对象在多个时刻下并非一直发生步态，因此通过目标骨架得到模块430可以根据多个时刻下的三维骨架从多个时刻中确定出待分析对象发生步态的多个步态发生时刻；由于目标骨架是步态发生时刻下的三维骨架，因此通过步态参数确定模块440可以根据多个目标骨架确定步态参数。上述装置，可以准确得到处于自由移动状态下的待分析对象的步态参数。
本申请实施例所提供的步态分析装置可执行本申请任意实施例所提供的步态分析方法，具备执行方法相应的功能模块和效果。
上述步态分析装置的实施例中，所包括的多个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，多个功能单元的名称也只是为了便于相互区分，并不用于限制本申请的保护范围。
实施例五
图8为本申请实施例五提供的一种步态分析设备的结构示意图，参见图8，该设备包括存储器510、处理器520、输入装置530和输出装置540。设备中的 处理器520的数量可以是一个或多个，图8中以一个处理器520为例；设备中的存储器510、处理器520、输入装置530和输出装置540可以通过总线或其它方式连接，图8中以通过总线550连接为例。
存储器510作为一种计算机可读存储介质，可设置为存储软件程序、计算机可执行程序以及模块，如本申请实施例中的步态分析方法对应的程序指令/模块(如，步态分析装置中的肢体图像获取模块410、三维骨架确定模块420、目标骨架得到模块430和步态参数确定模块440)。处理器520通过运行存储在存储器510中的软件程序、指令以及模块，从而执行设备的多种功能应用以及数据处理，即实现上述的步态分析方法。
存储器510可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据设备的使用所创建的数据等。此外，存储器510可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储器510可包括相对于处理器520远程设置的存储器，这些远程存储器可以通过网络连接至设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入装置530可设置为接收输入的数字或字符信息，以及产生与装置的用户设置以及功能控制有关的键信号输入。输出装置540可包括显示屏等显示设备。
实施例六
本申请实施例六提供一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行一种步态分析方法，该方法包括：
获取位于静止平面上的待分析对象在多个时刻下的肢体图像；针对每个时刻下的肢体图像，根据每个时刻下的肢体图像确定待分析对象在每个时刻下的三维骨架；根据多个时刻下的三维骨架从多个时刻中确定待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架；根据多个目标骨架确定待分析对象的步态参数。
本申请实施例所提供的一种包含计算机可执行指令的存储介质，其计算机可执行指令不限于如上所述的方法操作，还可以执行本申请任意实施例所提供的步态分析方法中的相关操作。
通过以上关于实施方式的描述，所属领域的技术人员可以了解到，本申请可借助软件及必需的通用硬件来实现，也可以通过硬件实现。本申请的技术方案本质上可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算 机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、闪存(FLASH)、硬盘或光盘等，包括多个指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请实施例所述的方法。

Claims (10)

  1. 一种步态分析方法，包括：
    获取位于静止平面上的待分析对象在多个时刻下的肢体图像；
    针对于每个时刻下的肢体图像，根据所述每个时刻下的肢体图像确定所述待分析对象在所述每个时刻下的三维骨架；
    根据所述多个时刻下的三维骨架从所述多个时刻中确定所述待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的三维骨架作为目标骨架；
    根据多个目标骨架确定所述待分析对象的步态参数。
  2. 根据权利要求1所述的方法，其中，所述根据所述多个时刻下的三维骨架从所述多个时刻中确定所述待分析对象发生步态的多个步态发生时刻，包括：
    获取所述多个时刻下的三维骨架对应的骨架时间序列，并以所述待分析对象的对象行为为切分目标，对所述骨架时间序列进行切分；
    结合所述待分析对象在多个切分后的骨架时间序列下的目标移动速度，对所述多个切分后的骨架时间序列在所述对象行为上进行聚类；
    根据聚类结果确定所述多个时刻中所述待分析对象发生步态的多个步态发生时刻。
  3. 根据权利要求1所述的方法，其中，所述三维骨架是包括所述待分析对象的步态肢体上的多个肢体骨骼点的三维的骨架；
    所述根据所述多个时刻下的三维骨架从所述多个时刻中确定所述待分析对象发生步态的多个步态发生时刻，包括：
    针对于每个三维骨架，对所述三维骨架进行旋转，以使旋转后的所述三维骨架所在的骨架平面平行于所述静止平面；
    根据旋转后的所述三维骨架判断所述步态肢体是否接触到所述静止平面；
    根据与所述多个时刻分别对应的多个判断结果，从所述多个时刻中确定所述待分析对象发生步态的多个步态发生时刻。
  4. 根据权利要求1所述的方法，其中，所述根据多个目标骨架确定所述待分析对象的步态参数，包括：
    针对于每个三维骨架，根据所述每个三维骨架确定所述待分析对象的步态肢体在与所述每个三维骨架对应的时刻下的肢体角度；
    根据所述多个时刻下的肢体角度中与所述多个目标骨架对应的多个肢体角 度，确定所述待分析对象的步态参数。
  5. 根据权利要求4所述的方法，其中，所述根据所述多个时刻下的肢体角度中与所述多个目标骨架对应的多个肢体角度，确定所述待分析对象的步态参数包括：
    获取由在时间上连续的多个步态发生时刻构成的步态发生时间序列；
    针对每个步态发生时间序列，根据所述每个步态发生时间序列下的多个肢体角度，确定所述待分析对象在所述每个步态发生时间序列下的支撑相的支撑相信息和摆动相的摆动相信息；
    根据每个步态发生时间序列下的支撑相信息以及摆动相信息，确定所述待分析对象的步态参数。
  6. 根据权利要求1所述的方法，其中，所述根据所述每个时刻下的肢体图像确定所述待分析对象在所述每个时刻下的三维骨架，包括：
    针对于所述每个时刻下的每个肢体图像，根据所述每个时刻下的每个肢体图像确定所述待分析对象上的多个预设骨骼点在所述每个时刻下的二维姿态；
    根据与所述每个时刻下的至少一个肢体图像分别对应的至少一个二维姿态和分别设置为拍摄所述每个时刻下的至少一个肢体图像的至少一个图像拍摄装置的标定后的装置参数，重建得到所述待分析对象上的多个预设骨骼点在所述每个时刻下的三维骨架。
  7. 根据权利要求1所述的方法，其中，所述获取位于静止平面上的待分析对象在多个时刻下的肢体图像，包括：
    基于图像拍摄设备中的图像拍摄装置获取位于所述静止平面上的所述待分析对象在所述多个时刻下的肢体图像；
    其中，所述图像拍摄设备包括至少四个图像拍摄装置、以及设置为提供所述静止平面的悬空固定的平面装置，所述平面装置的装置颜色包括与所述待分析对象的生活环境下的环境地面的地面颜色相关的颜色，所述至少四个图像拍摄装置中的至少三个图像拍摄装置固定在所述平面装置的与所述待分析对象同侧的一侧上、以及至少一个图像拍摄装置固定在所述平面装置的另一侧上。
  8. 一种步态分析装置，包括：
    肢体图像获取模块，设置为获取位于静止平面上的待分析对象在多个时刻下的肢体图像；
    三维骨架确定模块，设置为针对每个时刻下的肢体图像，根据所述每个时 刻下的肢体图像确定所述待分析对象在所述每个时刻下的三维骨架；
    目标骨架得到模块，设置为根据所述多个时刻下的三维骨架从所述多个时刻中确定所述待分析对象发生步态的多个步态发生时刻，并将每个步态发生时刻下的所述三维骨架作为目标骨架；
    步态参数确定模块，设置为根据多个目标骨架确定所述待分析对象的步态参数。
  9. 一种步态分析设备，包括：
    至少一个处理器；
    存储器，设置为存储至少一个程序；
    当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-7中任一项所述的步态分析方法。
  10. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1-7中任一项所述的步态分析方法。
PCT/CN2021/137903 2021-11-30 2021-12-14 步态分析方法、装置、设备及存储介质 WO2023097773A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111439683.7 2021-11-30
CN202111439683.7A CN114093034A (zh) 2021-11-30 2021-11-30 一种步态分析方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2023097773A1 true WO2023097773A1 (zh) 2023-06-08

Family

ID=80305661

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/137903 WO2023097773A1 (zh) 2021-11-30 2021-12-14 步态分析方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN114093034A (zh)
WO (1) WO2023097773A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114869270B (zh) * 2022-03-07 2023-07-18 电子科技大学 一种基于小动物图像和足底压力数据融合的步态分析方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200050840A1 (en) * 2018-08-07 2020-02-13 Georgetown University Multidimensional Analysis of Gait in Rodent
CN111209882A (zh) * 2020-01-13 2020-05-29 孝感峰创智能科技有限公司 一种全向运动装置的步态信息获取方法、系统及可读存储介质
CN113016715A (zh) * 2021-03-23 2021-06-25 广东省科学院动物研究所 一种灵长类动物精细动作行为分析方法和系统
CN113255462A (zh) * 2021-04-29 2021-08-13 深圳大学 步态评分方法、系统、计算机程序产品及可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200050840A1 (en) * 2018-08-07 2020-02-13 Georgetown University Multidimensional Analysis of Gait in Rodent
CN111209882A (zh) * 2020-01-13 2020-05-29 孝感峰创智能科技有限公司 一种全向运动装置的步态信息获取方法、系统及可读存储介质
CN113016715A (zh) * 2021-03-23 2021-06-25 广东省科学院动物研究所 一种灵长类动物精细动作行为分析方法和系统
CN113255462A (zh) * 2021-04-29 2021-08-13 深圳大学 步态评分方法、系统、计算机程序产品及可读存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
HAN, YANING ET AL.: "MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice", NEUROSCIENCE BULLETIN, 12 October 2021 (2021-10-12), pages 303 - 317, XP037787494, DOI: 10.1007/s12264-021-00778-6 *
HUANG KANG, HAN YANING, CHEN KE, PAN HONGLI, ZHAO GAOYANG, YI WENLING, LI XIAOXI, LIU SIYUAN, WEI PENGFEI, WANG LIPING: "A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping", NATURE COMMUNICATIONS, vol. 12, no. 1, XP093070349, DOI: 10.1038/s41467-021-22970-y *

Also Published As

Publication number Publication date
CN114093034A (zh) 2022-02-25

Similar Documents

Publication Publication Date Title
US11928800B2 (en) Image coordinate system transformation method and apparatus, device, and storage medium
KR102394593B1 (ko) 신체 이미징
CN110807364B (zh) 三维人脸与眼球运动的建模与捕获方法及系统
CN108921907B (zh) 一种运动测试评分的方法、装置、设备及存储介质
CN113658211B (zh) 一种用户姿态的评估方法、装置以及处理设备
CN106372629A (zh) 一种活体检测方法和装置
CN113850865A (zh) 一种基于双目视觉的人体姿态定位方法、系统和存储介质
Gärtner et al. Deep reinforcement learning for active human pose estimation
CN107368790B (zh) 行人检测方法、系统、计算机可读存储介质及电子设备
CN111079578A (zh) 行为检测方法及装置
CN108985263B (zh) 一种数据采集方法、装置、电子设备和计算机可读介质
CN110443884B (zh) 手部运动重建方法和装置
CN109584321A (zh) 用于基于深度学习的图像重建的系统和方法
Elhayek et al. Fully automatic multi-person human motion capture for vr applications
WO2023097773A1 (zh) 步态分析方法、装置、设备及存储介质
Wu et al. [Retracted] 3D Film Animation Image Acquisition and Feature Processing Based on the Latest Virtual Reconstruction Technology
US11250592B2 (en) Information processing apparatus
CN109389367A (zh) 人员考勤方法、装置及存储介质
CN116868912A (zh) 检测动物社交障碍行为的装置及其方法、电子设备、介质
CN109460077B (zh) 一种自动跟踪方法、自动跟踪设备及自动跟踪系统
CN116109974A (zh) 体积视频展示方法以及相关设备
CN115546408A (zh) 模型简化方法、装置、存储介质、电子设备及产品
CN114067422A (zh) 一种用于辅助驾驶的视线检测方法、装置及存储介质
Tang et al. Motion human parsing: A new benchmark for 3d human parsing
CN116310245A (zh) 体积视频上挂载道具的方法、装置、介质、设备及产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21966197

Country of ref document: EP

Kind code of ref document: A1