CN114999646A

CN114999646A - Newborn exercise development assessment system, method, device and storage medium

Info

Publication number: CN114999646A
Application number: CN202210622070.5A
Authority: CN
Inventors: 伊鸣; 黄新瑞; 韩彤妍; 黄春玲; 商潇腾
Original assignee: Peking University
Current assignee: Peking University
Priority date: 2022-06-01
Filing date: 2022-06-01
Publication date: 2022-09-02
Anticipated expiration: 2042-06-01
Also published as: CN114999646B

Abstract

The invention relates to a system, a method, a device and a storage medium for evaluating the newborn exercise development, wherein the system comprises: the video acquisition module is used for remotely acquiring a whole body movement video of the newborn; the video processing module is used for extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the neonate; the data analysis module is used for obtaining a curve of the space coordinate and the preset joint angle along with time change according to the space coordinate position data, extracting corresponding time domain characteristics and frequency domain characteristics according to the curve, and evaluating the whole body movement of the neonate according to the time domain characteristics, the frequency domain characteristics and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result; and the result storage and output module is used for storing and outputting the whole body movement video, the time domain characteristics, the frequency domain characteristics and the whole body movement quality evaluation result. Through this technical scheme, carry out automatic classification to neonate GMs, improve the rate of accuracy that neonate's abnormal behavior detected.

Description

Newborn exercise development assessment system, method, device and storage medium

Technical Field

The invention relates to the technical field of medical information, in particular to a system, a method, a device and a storage medium for evaluating the motion development of a newborn.

Background

A newborn infant may suffer from neurodevelopmental abnormalities or disorders if it is compromised by high risk factors such as premature birth (typically born at 37 weeks gestational age), low body weight (mostly below 2500 g), asphyxia, hypoxic-ischemic encephalopathy, intracranial hemorrhage, etc. during fetal, childbirth, and neonatal period. Along with the improvement of medical treatment level, the survival rate of high-risk neonates is continuously improved, how to reduce the incidence of neurodevelopmental dysfunction of the high-risk neonates and alleviate the degree of disability is more and more paid more attention by parents and pediatric clinicians. At present, domestic pediatric clinicians generally evaluate the brain injury and clinical rehabilitation efficacy of newborns according to the examination of craniocerebral ultrasonography, MRI, CT and other imaging, Brainstem Auditory Evoked Potential (BAEP), biochemical indexes and the like. Clinically, the diagnosis of cerebral palsy, mental retardation and other diseases can be confirmed only after the child ages one year or even two years. A neurological disorder caused by brain damage occurring before, during or shortly after birth is a disorder that mainly affects movement, posture and coordination. Whole body movements (GMs) are spontaneous movements that occur from early fetal to about 20 weeks after term and they can last from a few seconds to a few minutes, and are divided into at least two normal movements and five abnormal movements. GMs involve the entire body in a range of arm, leg, neck and torso movements, the intensity of which may vary over time. The early neurology development expert Prechtl indicated that GMs could be generated by a central pattern generator embedded from the brainstem to the spinal cord, and that the characteristics of these movements may change as the cerebral cortex develops. The method of assessing neuronal development by observation GMs has equal or better efficacy in prognostic prediction than neurological tests and is therefore useful for the diagnosis of future diseases. Providing a reliable method for early diagnosis, assessing the quality, complexity and spontaneity of the infant GMs at a specific time window of infant development (typically 0 to 20 weeks post-partum), identifying infants at risk of neuromotor disorders, finding ill infants with poor prognosis early, and giving rehabilitation intervention early enough to improve their prognosis, reversing the outcome of poor neurodevelopmental, are of great clinical and social importance.

Prechtl, "qualitative assessment of whole body movement in preterm, term and young infants" is the basis of GMs, GMs in a typical developing infant shows complexity, variability, and for infants with impaired nervous system, GMs loses its complex and variable character and becomes less fluid. These patterns of abnormalities GMs are powerful predictors of the prognosis of the infant's progression to cerebral palsy, mental retardation, etc. GMs can only be visually assessed by a trained clinician who has the license to assess. These clinicians require extensive training and years of actual evaluation experience to achieve suitable accuracy. Since this method requires a long time of observation GMs by the clinician, it is susceptible to fatigue of the observer and difficult to make an objective and quantitative assessment. The manual, time consuming and scarce availability of evaluators currently assessed at GMs, makes the current GMs test generally only useful in situations where there is a medical problem, such as preterm birth, stroke, hypoxia or congenital heart disease, and not used as a physical examination screening tool for normal infants, while mass screening can identify more high-risk infants. Automated quantitative evaluation of GMs is required for screening of large numbers of infants for physical examination. GMs the development of an automated quantitative assessment early diagnosis system helps to reduce the time and costs associated with current manual diagnostic practices, potentially helping healthcare professionals more reliably communicate information to patient families. Furthermore, the development of a suitably reliable automated tool means that all infants can be analysed, helping the healthcare professional to determine any additional care requirements.

In order to be able to objectively assess and measure the infant's movement, there are studies to assess the spontaneous movement of the limb during GMs using a position sensor and an acceleration sensor attached to the limb, analyze the periodicity of velocity and acceleration in the spontaneous movement, and diagnose dyskinesia with the extracted features. However, the above studies attach sensors or markers to the infant, which can interfere with the spontaneous occurrence of motion. As an alternative, there is research and development of a markerless measurement method for baby motion-video analysis method. These methods typically use video-based optical flow methods, frequency analysis, and background subtraction. However, each of these approaches lacks robustness in handling unnecessary information, illumination variations, body part size, and external influences. Due to inherent limitations of such conventional optical flow-based methods, there are researchers who have recently begun to validate the validity of pose-based assessments. Human gesture automatic recognition has been an active area of research in recent years. Accurately tracking the motion of various parts of the body during the occurrence of motion behaviors is an important content of motion science. With the rapid development of computer software and hardware technology, when a camera continuously shoots at a high enough speed, the motion trail of key points of each part of a body can be obtained from an image sequence. One of the most widely known methods for automatically estimating body pose from RGB video/images is openpos, which produces an output that provides joint positions and orientations of the body limbs based on a set of predetermined key points. In 2020, Google's artificial intelligence work pipeline framework MediaPipe provides a single person body pose estimation algorithm BlazePose for running on edge devices, which is not only more than 16 more predicted body key points than OpenPose, but also has performance 25-75 times faster on a middle-end cell phone CPU than OpenPose on a 20-core desktop CPU, and is a Machine Learning (ML) solution for high fidelity body pose tracking. However, existing human posture estimation frameworks are almost exclusively trained and tested using adult images, and therefore their application cannot be directly generalized to humans of different shapes and body compositions, such as infants, and especially newborns. At present, no adult posture estimation algorithm is migrated to a mature posture estimation model built by a baby, so that the position information of the motion key point directly extracted by using the adult posture estimation model is not accurate enough, and the ideal prediction effect is difficult to obtain when the method is applied to the automatic evaluation of the baby GMs. Additionally, because GMs assess the sensitivity of desired video data, a significant challenge facing researchers attempting to automate GMs assessments is the availability of publicly accessible data sets. Thus, enabling GMs automated assessment based on modern deep learning techniques is limited by the need for well-annotated large infant data sets, and it is difficult to obtain data sets containing infant images/videos for specific research purposes, especially neonatal data sets.

The organs of the newborn are incompletely developed, the resistance is low, and the hospital infection rate is high. Therefore, the neonatal ward has a strict disinfection and isolation system and strictly limits the entrance and exit of people. Too many people stay in the ward, which can affect the environment and work order of the ward. In addition, premature infants are generally placed in a separate nursery or dedicated incubator in order to subject the infant to an environment in which the temperature, humidity, oxygen concentration, etc. are all controlled within appropriate ranges. Therefore, it is necessary to develop new modes of neonatal data collection.

Disclosure of Invention

In order to overcome the problems in the related art, the invention provides a system, a method, a device and a storage medium for evaluating the motion development of the neonate, which automatically classify the neonate GMs by using an optimized machine learning classification algorithm, thereby improving the accuracy of detecting the abnormal behavior of the neonate.

According to a first aspect of embodiments of the present invention, there is provided a neonatal motor development assessment system for assessing neonatal brain development status, predicting brain neurodevelopmental disorder of premature infants, and early intervention rehabilitation therapy basis, the system comprising:

the video acquisition module is used for remotely acquiring a whole body movement video of the neonate;

the video processing module is used for extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the newborn;

the data analysis module is used for obtaining a curve of a space coordinate and a preset joint included angle along with time change according to space coordinate position data of key points of all parts of a body in the whole body movement video, extracting corresponding time domain characteristics and frequency domain characteristics according to the curve, and evaluating the whole body movement of the neonate according to the time domain characteristics, the frequency domain characteristics and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result;

and the result storage and output module is used for storing the whole body movement video, the time domain characteristics, the frequency domain characteristics and the whole body movement quality evaluation result, and constructing a whole body movement evaluation database of the newborn for output.

In one embodiment, preferably, the whole body movement quality assessment result includes any one of: the writhing phase normal movements, the monotonous movements and the spasm-synchronized movements.

In one embodiment, preferably, the video processing module is specifically configured to:

and extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the neonate through a pre-trained neonate key point estimation model.

In one embodiment, preferably, the training process of the neonatal keypoint estimation model comprises:

carrying out key point identification on the whole body movement training video of the baby in the baby whole body movement video data set through a MediaPipe BlazePose adult posture estimation model to obtain corresponding 33 key point information;

marking and displaying 33 key points in the whole body exercise training video of the infant according to the 33 key point information, and manually determining whether the key point positions are accurate so as to screen out the target whole body exercise training video with the accurate key point positions;

extracting 23 pieces of target key point information from 33 pieces of key point information corresponding to the target whole body movement training video;

performing model iterative training according to the 23 target key point information of the target whole body motion training video to obtain a newborn key point estimation model, wherein the 23 target key points comprise: nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left little finger, right little finger, left forefinger, right forefinger, left thumb, right thumb, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe and right toe.

In an embodiment, preferably, the data analysis module is specifically configured to:

obtaining a curve of the change of the space coordinate and the preset joint included angle along with time according to the space coordinate position data of key points of all parts of the body in the whole body movement video;

according to the space coordinate of the neonate and a curve of the preset joint included angle changing along with time, corresponding time domain characteristics and frequency domain characteristics are calculated, wherein the time domain characteristics comprise: maximum value, minimum value, range, mean value, standard deviation, root mean square value, K-order central moment, K-order origin moment, median, mode, skewness, kurtosis factor, form factor, pulse factor and margin factor, the frequency domain characteristics include: mean, variance, entropy, energy, skewness, kurtosis, waveform mean, waveform standard deviation, waveform skewness, and waveform kurtosis;

and evaluating the whole body movement of the neonate according to the time domain characteristics, the frequency domain characteristics and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result.

In one embodiment, preferably, the training process of the pre-trained neonatal whole body motion quality assessment model comprises:

acquiring a whole body movement training video of the infant from the whole body movement video data set of the infant;

extracting spatial coordinate position training data of key points of all parts of the body in the whole body movement training video through a pre-trained newborn key point estimation model;

obtaining a training space coordinate and a curve of a preset joint included angle changing along with time according to the space coordinate position training data, and extracting corresponding training time domain features and training frequency domain features according to the training space coordinate and the curve of the preset joint included angle changing along with time;

according to the time domain features for training and the frequency domain features for training, respectively training by adopting various classification models to obtain corresponding whole body movement quality assessment models, wherein the various classification models comprise: a decision tree model, a naive Bayes model, a discriminative analysis model, a kernel approximation model, a support vector machine model, a logarithmic probability regression model, a nearest neighbor model and a neural network model;

and selecting an optimal model from the multiple whole body movement quality assessment models as the whole body movement quality assessment model for the newborn.

In one embodiment, preferably, the whole-body motion video comprises a two-dimensional whole-body motion video or a three-dimensional whole-body motion video;

when the whole-body motion video includes a two-dimensional whole-body motion video, the spatial coordinate position data includes two-dimensional spatial coordinate position data;

when the whole-body movement video includes a three-dimensional whole-body movement video (i.e., a plurality of different-angle two-dimensional whole-body movement videos), the spatial coordinate position data includes three-dimensional spatial coordinate position data (i.e., a plurality of different-angle two-dimensional spatial coordinate position data integration results).

According to a second aspect of embodiments of the present invention, there is provided a neonatal motor development assessment method, the method comprising:

remotely acquiring a whole body motion video of a newborn;

extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the newborn;

obtaining a curve of a space coordinate and a preset joint included angle along with time change according to space coordinate position data of key points of all parts of a body in the whole body movement video, extracting corresponding time domain characteristics and frequency domain characteristics according to the curve, and evaluating the whole body movement of the neonate according to the time domain characteristics, the frequency domain characteristics and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result;

and storing the whole body movement video, the time domain characteristics, the frequency domain characteristics and the whole body movement quality evaluation result, and constructing a whole body movement evaluation database of the newborn for output.

According to a third aspect of embodiments of the present invention, there is provided a newborn exercise development assessment apparatus, the apparatus including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

remotely acquiring a whole-body motion video of a neonate;

The technical scheme provided by the embodiment of the invention can have the following beneficial effects:

in the embodiment of the invention, a newborn video without an additional mark is used as input, a newborn key point estimation model is improved through transfer learning based on an adaptive attitude estimation algorithm in the MediaPipe BlazePose adult field without manually selecting key points in advance, quantitative indexes reflecting the motion characteristics of the newborn GMs are extracted, and meanwhile, an optimized machine learning classification algorithm is used for automatically classifying the newborn GMs, so that the accuracy of detecting the abnormal behaviors of the newborn is improved. The unmarked motion measurement and quantitative evaluation system for the neonate GMs, which is described by the invention, can objectively and effectively evaluate the whole body motion state of the neonate, predict the risk of the neonate suffering from the neuromotor diseases, eliminate adverse factors such as doctor subjective factor interference and individual difference, possibly thoroughly change the conventional clinic diagnosis and treatment mode GMs, assist doctors to diagnose and predict the neonatal cranial nerve development in an early stage and establish a personalized intervention and rehabilitation plan, and can better serve the clinic by building a video/characteristic parameter database of the healthy neonate and the sick neonate GMs as a reference population.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.

Fig. 1 is a block diagram illustrating a system for assessing neonatal motor development, according to an exemplary embodiment.

Fig. 2 is a schematic diagram illustrating a neonatal whole body motion video acquisition module performing video capture remote acquisition according to an exemplary embodiment.

Fig. 3 is a schematic diagram illustrating a training process of a neonatal keypoint estimation model, according to an exemplary embodiment.

Fig. 4 is a diagram illustrating selected body key points in a selected neonatal whole body motion video analysis, according to an exemplary embodiment.

Fig. 5 is a schematic diagram illustrating a process of accurately extracting spatial coordinates of key points of a body in whole body movement of a newborn based on video analysis according to an exemplary embodiment.

FIG. 6 is a graphical illustration of extracted spatial coordinates, a particular joint angle, versus time, according to an exemplary embodiment.

Fig. 7 is a flow chart illustrating a method for assessing neonatal motor development, according to an exemplary embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.

Fig. 1 is a block diagram illustrating a neonatal motor development assessment system according to an exemplary embodiment.

As shown in fig. 1, according to a first aspect of the embodiments of the present invention, there is provided a system for assessing neonatal motor development, for assessing neonatal brain development status, predicting premature brain neurodevelopmental disorder, and for early intervention rehabilitation therapy, the system comprising:

the video acquisition module 11 is used for remotely acquiring a whole body movement video of a newborn;

specifically, the AnyDesk software can be used for remote computer/industrial personal computer operation. AnyDesk is a free and practical remote control tool and can provide stable and reliable remote desktop connection service for users. The AnyDesk software is downloaded and installed, the remote computer/industrial personal computer is connected through the exclusive address number, the user can set an autonomous connection password as long as the exclusive AnyDesk address is input, the connection can be easily performed even if the AnyDesk software is not arranged beside the computer, any response operation is not needed, the password can be known, and the AnyDesk software is very convenient and humanized. The exclusive address adopts the latest data transmission protocol and adopts the highest-level encryption mode to ensure the safety of data transmission, and moreover, the connection technology ensures that both sides can realize delay-free operation feedback and cannot cause asynchronous operation and audio due to networks or other reasons. In the setting, the user can set pictures, privacy, audio, file transmission, printers and the like in the transmission process so as to achieve the optimal remote control effect.

In view of GMs assessing the sensitivity of desired video data, a significant challenge facing researchers attempting to automate GMs assessment is the availability of publicly accessible data sets. Since existing body posture estimation frameworks are almost exclusively trained and tested using adult images, it can be appreciated that it is difficult to obtain data sets containing images of infants, especially neonates, for research purposes. Therefore, an evaluation framework using video data captured from neonates is one of the future directions contemplated by the present invention. Furthermore, given the relatively small number of high-risk neonate video sequences, the project would like to extend this work by classifying larger data sets. This project has been working in close cooperation with local hospitals in an effort to produce real world data sets for future evaluation. The video images are taken under the supervision of a physician to minimize the burden on the infant and avoid potential hazards. The skin thermoregulation center of the newborn is not complete in function, subcutaneous fat is thin, four limbs are often stretched, the body surface area is relatively large, heat dissipation capacity is increased, and the warm box is a main heat preservation device for premature babies. The moderate temperature is the environmental temperature which has the lowest oxygen consumption but can maintain the normal body temperature according to different ages in days and body weights of the premature infants, and the metabolism and the body temperature of the infants are influenced when the temperature exceeds the temperature range +/-2 ℃. The moderate temperatures of premature infants are different from those of full-term newborns. The moderate temperature is higher for the younger fetus, and gradually decreases with the increase of the age of the day. Thus, premature infants placed in the incubator need to adjust moderate temperatures daily by weight and day age. The temperature and humidity regulation of the warm box is particularly important. The video acquisition device fully considers the particularity of the neonate, particularly the premature infant, and the automatic temperature and humidity regulation can be realized by placing the neonate on the infant radiation heat-preservation table in the video acquisition process. Video images of 48 subjects, including full-term infants and low-weight newborn infants, were used in this experiment. Recording and analysis followed the GMs-assessed clinical criteria of Prechtl, and crying and sleeping time were excluded from the analysis.

As shown in fig. 2, the video capture module may use the arched door mount S4 to position one (or three) high speed cameras S3 at a fixed location from the warming table S1 and a pan-tilt head at a fixed distance from one of the cameras S3, such that the cameras are fixed directly above and parallel to the surface of the crib S1, without the need for a covering over the surface of the crib. Assuming that the infant lies supine in the center of the crib, the height of the camera is adjusted so that the entire body of the infant can be captured within the video frame. In order to accurately track the movement of the baby, the parameters of the image pickup in the invention are a frame rate of 60FPS, an exposure time of 2000ms and an image size of 800 x 1024 pixels. Under the condition of insufficient ambient light, a portable fluorescent lamp tube S5 is additionally arranged for supplementing light. The video acquisition module can display and monitor the tested placement position and the movement process in real time, can finish automatic start-stop acquisition, and is stored as a png continuous frame image or an avi video format, so that subsequent image processing is facilitated.

The video acquisition module can capture three-dimensional motion information besides acquiring two-dimensional information, is compact and exquisite, is convenient for acquiring high-frequency high-definition videos in a small space and has no background interference. In order to meet the requirements, three cameras are used for synchronous acquisition, the resolution ratio of the cameras is more than 100 ten thousand pixels, the frame rate is more than 100FPS, and the field angle is as large as possible. The synchronization among the multiple cameras is completed by a trigger line and a synchronization generator S8, one end of the trigger line is connected with the cameras, the other end of the trigger line is connected with the synchronization generator S8, and the external trigger synchronization is carried out through TTL signals. The collected data are stored in the industrial personal computer S9 through a U3 line (one end is connected with a camera, and the other end is connected with an acquisition card) and the acquisition card (inserted into a PCIe card slot of the industrial personal computer), and the collected data are preprocessed on the industrial personal computer. Synchronously acquiring software by multiple cameras, and previewing real-time videos of each camera one by one; setting exposure, gain and resolution of a camera; external triggering or Free-run acquisition is possible; the recording time can be set by time or frame number; storage formats raw, bmp, jpg and avi can be set; the number of the collected pictures can be recorded, and the frame loss quantity can be recorded; the long-time frame-loss-free storage can be realized under a computer. The multi-camera synchronous acquisition software should also be capable of subsequent programming function addition based on the camera's SDK.

In the video acquisition process, the neonate is in an extremely strict and comfortable environment with extremely strict temperature, humidity and safety degree, and is accompanied by professional medical care personnel. The facility setting and the monitoring state of the acquisition system are not easy to miss the optimal rescue opportunity when an emergency (such as convulsion) occurs. The camera is used for collecting scene monitoring and communicating with medical care personnel in a ward in real time through the wide-angle high-definition video conference camera integrating the camera, the sound and the microphone. When capturing the motion of a neonate, a high-definition high-speed professional RGB camera is used to perform acquisition from three different angles to minimize body part occlusion.

The present invention therefore provides a contactless way of capturing video of the whole body movement of a neonate in a relatively pleasant and painless safe situation.

The video processing module 12 is used for extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the neonate;

in an embodiment, preferably, the video processing module is specifically configured to:

In one embodiment, preferably, as shown in fig. 3, the training process of the neonatal keypoint estimation model includes:

carrying out key point identification on the whole body movement training video of the baby in the baby whole body movement video data set through a MediaPipe BlazePose adult posture estimation model to obtain corresponding 33 key point information; and saving 33 key point information to generate a csv file.

Marking and displaying 33 key points in the whole body movement training video of the baby according to the 33 key point information, and manually determining whether the key point positions are accurate so as to screen out the target whole body movement training video with the accurate key point positions;

performing model iterative training according to the 23 target key point information of the target whole body motion training video to obtain a neonatal key point estimation model, wherein, as shown in fig. 4, the 23 target key points include: nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left little finger, right little finger, left forefinger, right forefinger, left thumb, right thumb, left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe and right toe.

Extracting 23 pieces of target key point information from 33 pieces of key point information corresponding to the target whole body movement training video; specifically, data in the csv file can be processed, the 2 nd to 11 th key point position information is removed, other 23 target key point information is retained, a video with more accurate key point estimation and result data thereof are preferably input into a constructed Keras neural network, such as an LSTM model, a better model is obtained by freely adjusting the neural network hierarchy, optimizing model parameters and the like to accurately position postures in all videos of a newborn, and finally multi-camera posture estimation calibration is performed to obtain more accurate three-dimensional space coordinates of body key points, as shown in fig. 5 b.

The invention provides a new method for unsupervised newborn posture estimation by improving a Google MediaPipe adult posture estimator BlazePose through transfer learning. The BlazePose is developed by relying on Google Research and is an open-source multimedia machine learning model application framework MediaPipe, the framework provides library files corresponding to languages such as Python, C/C + +, Java and the like, the framework can be applied to Android, iOS, Desktop and Web platforms, a Python API interface is further provided, and the Python API of the BlazePose can not only predict static image postures, but also externally connect a camera to predict human body postures in videos in real time. The person's body posture tracking algorithm can be easily ported to its own program module by calling its API. In addition, MediaPipe also supports the Inference Engine (Inference Engine) of TensorFlow and TF Lite, on which any model of TensorFlow and TF Lite can be used. In addition to running at near real-time speeds on the CPU and even on mobile devices, MediaPipe also supports GPU acceleration of the device itself. The 33 three-dimensional landmarks and the background segmentation mask for the whole body can be inferred from the RGB video frames using blazepos, which also provides support for the ML Kit pose detection API.

The pose estimation framework of the present invention has been applied to extract 2D/3D poses from each of 48 neonates, three camera shot angles, 73 videos per shot angle. Each returned gesture is represented by the 2D/3D coordinates of 23 landmarks of the body keypoint (x, y (and z)). In addition to the 2D/3D coordinates, the confidence score for each keypoint is included in the output. In one embodiment, preferably, the motion video includes a two-dimensional motion video or a three-dimensional motion video; when the motion video includes a two-dimensional motion video, the spatial coordinate position data includes two-dimensional spatial coordinate position data, and when the motion video includes a three-dimensional motion video, the spatial coordinate position data includes three-dimensional spatial coordinate position data. To normalize the data and ensure comparability of direction and displacement between videos, the present invention takes the midpoint between hip joints (fig. 4, key points 13 and 14) as a reference point. As shown in fig. 5a, using MediaPipe BlazePose background segmentation mask can reduce the coordinate prediction inaccuracy caused by the external disturbance of the newborn body; the estimated coordinates of two-dimensional videos with different angles are integrated into three-dimensional space coordinates after being calibrated by the camera direction, so that the inaccuracy of two-dimensional coordinate prediction caused by occlusion can be reduced.

The data analysis module 13 is configured to obtain a curve of a change of a space coordinate and a preset joint angle with time according to space coordinate position data of key points of each part of a body in the whole body movement video, extract corresponding time domain features and frequency domain features according to the curve, and evaluate the whole body movement of the neonate according to the time domain features, the frequency domain features and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result;

data preprocessing is needed after spatial coordinate position data of key points of all parts of the body in the whole body movement video are obtained. First, the spatial coordinate position data of the body key points output by the above-described posture estimation is interpolated and smoothed. That is, the first stage of pre-processing is to remove any abnormal body keypoint locations caused by occlusion during motion or inaccurate prediction by the pose estimation model. For this reason, the confidence score is used as a threshold value for judging the accuracy of the body key point position predicted by the posture estimation model in the invention. The average confidence score for each body keypoint in each video sequence was calculated and then subtracted by 10%. In any frame where the confidence score of a body keypoint is below this threshold, an improved Akima interpolation or linear interpolation is used to interpolate between adjacent frames with confidence scores above the threshold.

The second stage of the preprocessing is to apply a one-dimensional Gaussian filter to smooth the curve of the spatial coordinates of each key point along with the time change, so as to eliminate the influence caused by the noise in the video image.

The third stage of the preprocessing is to calculate the curve of the space coordinate of the movement along with the time; the time series of the preset joint angles is calculated as shown in fig. 6.

For the time series of spatial coordinates, preset joint angles and the like, the invention calculates 17 time domain features and 10 frequency domain features which respectively correspond to the time domain features, and the features can reflect the amplitude and rhythm of the motion from limbs (wrist/ankle) and joint angles (elbow/knee), absolute positions/angles, the change of positions/angles, median velocity, the change of velocity, median absolute velocity, the variability of acceleration, complexity measures (entropy), symmetry measures (left-right cross correlation) and the like.

By establishing the set of strong feature sets based on the whole body movement posture of the newborn, the newborn can be classified by different traditional machine learning classifiers. While sometimes the combined use of features can significantly improve the accuracy of motion recognition, indicating increased risk of neuromotor, it is also possible to compress the data to a lower dimensional range, while retaining the most useful information, providing a complete but manageable impression of the relevant data. Therefore, the dimension reduction is carried out on the feature data set based on two types of filtering methods, the first type is a feature sorting method, and the influence of each feature on the grade classification is explained by a T Test (TT), a maximum correlation minimum redundancy (MRMR) and a chi-square test (CST) respectively, so that an optimized feature number is selected; the second category is feature reduction using PCA (principal Component analysis) and tSNE (t-Distributed stored Neighbor Embedding) algorithms. The optimal features are selected, the complexity, the variability and the symmetry of the whole body movement of the newborn are described by the features with lower dimensionality, the classification ambiguity can be reduced, and redundant information can be deleted from the classification process.

In an example of the present invention, the feasibility of extracting gesture-based features from a video sequence to automatically classify baby body movements into different categories GMs was explored. The classification was based on the GMs evaluation criteria of Prechtl, and was first performed on the video data by a specialist reviewer. The present invention explores the feasibility of using these gesture-based feature sets for automatic classification in a broad machine learning framework by conducting extensive experiments. The experiment of the present invention examined the effectiveness of 8 supervised classification algorithms (decision tree, naive bayes, discriminative analysis, kernel approximation, support vector machines, log-probability regression, nearest neighbor, neural networks) for classifying the extracted pose-based feature set. And the performance of the different classifiers is reported by two cross-validation methods (K-fold and leave-one-out). Based on encouraging results of the present invention in the classification of neonates GMs, in one embodiment, 44 examples of neonate video data were based, of which 19 examples "normal motion in the writhing phase"; in 25 cases of 'monotonous movement', a linear Support Vector Machine (SVM) classifier is used at present, 78% of consistent coincidence rate can be achieved compared with the diagnosis of a doctor, so the method is very suitable for processing data based on postures, particularly on the newborn GMs in the field of medical care.

And the result storage and output module 14 is configured to store the whole body movement video, the time domain feature, the frequency domain feature and the whole body movement quality evaluation result, and construct and output a whole body movement evaluation database of the newborn.

The invention discloses a system for remote data acquisition in a neonatal ward, automatic quantitative accurate evaluation of neonates GMs and prediction of neural development risk, which can change the conventional clinical GMs diagnosis and treatment mode of a traditional pediatrician, improve the working efficiency of the evaluation doctor and save manpower and material resources; the method has the advantages of high universality, accurate detection result and high detection speed, and can be used for long-time motion evaluation detection and large-scale growth and development screening and monitoring of infant groups. Meanwhile, the invention can know the test data, the doctor diagnosis result and the daily updated photo or video data of the premature infant at any time without introducing infection to a neonatal ward, thereby meeting the GMs requirement of automatic quantitative evaluation data collection, improving the hospital service level and improving the communication efficiency between doctors and patients and between clinic and scientific research.

In one embodiment, preferably, the whole body exercise quality assessment result includes any one of: the writhing phase normal movements, the monotonous movements and the spasm-synchronized movements.

according to the space coordinate of the movement of the newborn and a curve of the preset joint included angle changing along with time, corresponding time domain characteristics and frequency domain characteristics are calculated, wherein the time domain characteristics comprise: maximum value, minimum value, range, mean value, standard deviation, root mean square value, K-order central moment, K-order origin moment, median, mode, skewness, kurtosis factor, form factor, pulse factor and margin factor, the frequency domain characteristics include: mean, variance, entropy, energy, skewness, kurtosis, waveform mean, waveform standard deviation, waveform skewness, and waveform kurtosis;

The invention extracts two corresponding characteristic sets aiming at a space coordinate obtained from preprocessed posture data and a time sequence of a preset joint included angle: a time domain feature set and a frequency domain feature set. Deep learning architecture derived feature sets and these manual feature fusions will also be used in the future for further accurate evaluation in subsequent classification practices. In one embodiment, the feature sets are preferably generated to reflect information in three aspects: a) amplitude of motion, including duration and amplitude of motion; b) motion balance, described by analyzing the rate and correlation of motion between left and right regions of the body; c) the rhythm of the movement, such as the periodicity representing the movement, etc. Defining and calculating characteristic parameters, specifically:

1. time domain characterization

Assuming that there are m pieces of data, each piece of data has a length of n, the jth data point of the ith piece of data is represented by z _ij The ith data is represented by z _i To represent. (1) Dimensional time domain features include maximum (maximum), minimum (minimum), range, mean (mean), median (median), mode, standard deviation, root mean square (rms), mean square/ms, and k-th order center/origin moments. (2) Dimensionless time domain features are skewness (skewness), kurtosis (kurtosis), kurtosis factor (kurtosis factor), waveshape factor (waveform factor), pulse factor (pulse factor), and margin factor (margin factor), respectively.

2. Frequency domain features

Let f (k), k 1, 2.., N is the frequency spectrum of the time series, N is half the length of the frequency series, and f (k) is each component in the frequency spectrum. The mean, variance, entropy, energy, skewness, kurtosis of the spectrum, and the mean, standard deviation, skewness, kurtosis, etc. of the spectral envelope waveform may be calculated. The specific calculation formula is shown in table 1.

TABLE 1

In one embodiment, preferably, the training process of the pre-trained neonatal whole body motion quality assessment model includes:

and selecting an optimal model from the multiple whole body movement quality evaluation models as the whole body movement quality evaluation model of the newborn.

when the whole body motion video includes a three-dimensional whole body motion video, the spatial coordinate position data includes three-dimensional spatial coordinate position data.

As shown in fig. 7, according to a second aspect of the embodiments of the present invention, there is provided a newborn exercise development assessment method, including:

step S701, remotely acquiring a whole body movement video of a newborn;

step S702, extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the newborn;

step S703, obtaining a curve of a space coordinate and a preset joint angle changing along with time according to space coordinate position data of key points of all parts of a body in the whole body movement video, extracting corresponding time domain characteristics and frequency domain characteristics according to the curve, and evaluating the whole body movement of the neonate according to the time domain characteristics, the frequency domain characteristics and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result;

step S704, storing the whole body movement video, the time domain feature, the frequency domain feature and the whole body movement quality evaluation result, and constructing a newborn whole body movement evaluation database for outputting.

1. the invention discloses a platform and a system for remote data acquisition in a neonatal ward, automatic quantitative accurate evaluation of neonates GMs and prediction of neural development risk, which can change the conventional clinical GMs diagnosis and treatment mode of the traditional pediatrician, improve the working efficiency of an evaluation doctor and save manpower and material resources; the method has the advantages of high universality, accurate detection result and high detection speed, and can be used for long-time motion evaluation detection and large-scale growth and development screening and monitoring of infant groups.

2. According to the embodiment of the invention, the test data, the doctor diagnosis result and the daily updated photo or video data of the premature infant can be known at any time without introducing infection into a neonatal ward, the requirement of GMs on automatic quantitative evaluation data collection is met, the hospital service level is improved, and the communication efficiency between doctors and patients and between clinic and scientific research is improved.

3. The whole-body movement video acquisition of the newborn is carried out in a suitable environment, the baby is comfortable, the baby does not need to be contacted with the baby, the spontaneous movement of the baby is not influenced, and the operation is convenient and safe.

4. According to the method, the newborn posture estimation model is improved through transfer learning based on a MediaPipe BlazePose posture estimation algorithm adaptive to the adult field without manually selecting key points in advance, so that automatic high-precision marking of 23 key points of a body can be realized, and three-dimensional space coordinates are obtained. Compared with the large data set learning requirement of general deep learning, the efficiency is greatly improved, so that the video action analysis speed is greatly improved.

5. The invention extracts time domain and frequency domain characteristics of a plurality of parameters by combining the motion tracks of the limbs and the whole body of the baby, greatly improves the utilization degree of motion information, and can express the complexity, variability and symmetry of the baby motion. The method is a global analysis of the infant GMs, and improves the specificity and sensitivity of assessment prediction by comparing normalized data of healthy infants and sick infants.

6. The method is used for clinical application, is a comprehensive evaluation of the infant GMs, and can judge the illness state, clinical curative effect and prognosis of the infant more comprehensively and reasonably. The growth and development evaluation can provide a clear thinking direction for the next clinical diagnosis and treatment; and determining whether active intervention is required in clinic according to the evaluation condition and the time and the course of the intervention. The development evaluation method can find the mild brain injury of the baby at the early stage, avoid the insufficiency of single evaluation, avoid missing the best early rehabilitation treatment opportunity, and simultaneously be used for guiding clinical rehabilitation; the development evaluation method can be used for carrying out regular evaluation and linear observation on children patients with brain injury and developmental retardation, and can be used for observing the clinical rehabilitation curative effect and prognosis in a longitudinal view; the development evaluation is purely manual operation and is noninvasive, and children and parents are in good cooperation; the development evaluation can be more comprehensively and reasonably carried out on each high-risk child and each child subjected to physical examination in 42 days; GMs has the advantages of high safety, no adverse side effects, no wound, and no influence on other new treatment methods.

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

remotely acquiring a whole body motion video of a newborn;

extracting the spatial coordinate position data of key points of all parts of the body in the whole body movement video of the neonate;

According to a fourth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method according to any one of the embodiments of the first aspect.

It is further understood that the term "plurality" means two or more, and other terms are analogous. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. The singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

It will be further understood that the terms "first," "second," and the like, are used to describe various information and should not be limited by these terms. These terms are only used to distinguish one type of information from another, and do not indicate a particular order or degree of importance. Indeed, the terms "first," "second," and the like are fully interchangeable. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention.

It is further to be understood that while operations are depicted in the drawings in a particular order, this is not to be understood as requiring that such operations be performed in the particular order shown or in serial order, or that all illustrated operations be performed, to achieve desirable results. In certain environments, multitasking and parallel processing may be advantageous.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It will be understood that the invention is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims

1. A system for assessing neonatal motor development, the system comprising:

2. The system of claim 1, wherein the global motion quality assessment comprises any one of: the writhing phase normal movements, the monotonous movements and the spasm-synchronized movements.

3. The system of claim 1, wherein the video processing module is specifically configured to:

4. The system of claim 1, wherein the training process of the neonatal keypoint estimation model comprises:

5. The system of claim 1, wherein the data analysis module is specifically configured to:

according to the space coordinates of the neonate and a curve of the preset joint included angle changing along with time, corresponding time domain characteristics and frequency domain characteristics are calculated, wherein the time domain characteristics comprise: maximum value, minimum value, range, mean value, standard deviation, root mean square value, K-order central moment, K-order origin moment, median, mode, skewness, kurtosis factor, form factor, pulse factor and margin factor, the frequency domain characteristics include: mean, variance, entropy, energy, skewness, kurtosis, waveform mean, waveform standard deviation, waveform skewness, and waveform kurtosis;

6. The system of claim 1, wherein the training process of the pre-trained neonatal whole-body motion quality assessment model comprises:

7. The system of claim 1, wherein the whole-body motion video comprises a two-dimensional whole-body motion video or a three-dimensional whole-body motion video;

when the whole-body motion video includes a three-dimensional whole-body motion video, the spatial coordinate position data includes three-dimensional spatial coordinate position data.

8. A method for assessing neonatal motor development, the method comprising:

remotely acquiring a whole-body motion video of a neonate;

obtaining a curve of which the space coordinate and the preset joint included angle change along with time according to the space coordinate position data of key points of all parts of the body in the whole body movement video, extracting corresponding time domain characteristics and frequency domain characteristics according to the curve, and evaluating the whole body movement of the neonate according to the time domain characteristics, the frequency domain characteristics and a pre-trained whole body movement quality evaluation model of the neonate to obtain a whole body movement quality evaluation result;

9. An apparatus for assessing the motor development of a newborn, the apparatus comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to:

remotely acquiring a whole-body motion video of a neonate;

10. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, carry out the steps of the method according to any one of claims 1 to 7.